whileai 0.63__tar.gz → 0.65__tar.gz

{whileai-0.63/whileai.egg-info → whileai-0.65}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.63
+Version: 0.65
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -24,6 +24,7 @@ Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: requests>=2.25.0
+Requires-Dist: pydantic>=2.0
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0; extra == "dev"
 Requires-Dist: pytest-cov>=5.0; extra == "dev"
@@ -709,6 +710,22 @@ rows, report = wai.drop_leaky_rows(data.trajectories, prod)
 And the loop closes on itself: `evaluate(rollouts, judge).failed_traces()`
 hands the failures straight back to `simulate(traces=...)`.
 
+**What traces can and cannot aim at.** Traces reproduce situations: the
+tools, faults and world states the deployed agent met. A failure that has
+a world-visible trigger (a tool timed out and the agent did not say so, a
+stale record was presented as current) is reproduced. A failure that lives
+in how the reply is worded (an unsupported claim, an estimate not labelled
+as one, two questions where one was asked for) has no trigger in the world,
+so traces alone cannot aim at it: measured on a 12-rule grader, every rule
+with a tool-result trigger was reproduced and every rule about the reply's
+wording was not (#285). For those, put the grader in the loop:
+with `simulate(..., grader=judge)` a row the grader fails is re-rolled and
+its ask mutated like a tool fault, and `data.search["mutation_aims"]` says
+how many parents and mutated rows each aim (`world_fault`,
+`graded_failure`) produced. The grader is the switch; to grade beside the
+loop and still steer by tool faults alone, pass
+`advanced={"mutate_graded_failures": False}`.
+
 If your traces are already on the platform, `wai.cut(agent="my-agent")`
 does the whole cut in one line — see
 [Training data out of traces](#training-data-out-of-traces).
@@ -903,7 +920,8 @@ The platform's "Make training data" button, as one line:
 
 ```python
 wai.send_score("4bf92f3577b34da6", 1.0)  # this run passed
-wai.cuts(agent="my-agent")  # what a cut would hold
+summary = wai.cuts(agent="my-agent")  # what a cut would hold
+print(wai.format_cuts(summary, agent="my-agent"))  # the traces page's sentence
 made = wai.cut(agent="my-agent", kind="rl")  # make it
 wai.pull(made["train"]["datasetId"], "train.jsonl")
 made["holdout"]["datasetId"]  # measure on this, never train on it
@@ -947,6 +965,13 @@ wai.style_markers(rows)  # no_boilerplate, no_hedging, no_apology, no_sycophancy
 wai.style_report(rows)["warnings"]  # "reward pays for hedging (corr +0.41 ...)"
 wai.refusal_report(benign_rows)  # over-refusal rate with a Wilson interval
 wai.compare_runs(run_a, run_b)  # paired delta with a 95% interval
+wai.holdout_size(
+    0.05, base=0.6, k=4
+)  # tasks to prove a 5-point gain; n_tasks_concentrated beside it
+wai.holdout_size(
+    0.05, before=before, after=after
+)  # the paired sd measured off a previous eval, no model
+wai.holdout_size(0.05, task_std=0.38)  # or the sd read off a delta_report interval
 wai.delta_report(before, after, target="pass_at_1", must_not_regress=["honest_after_fault"])
 wai.delta_report(before, after, target="pass_at_1", by="category")  # the target per kind of prompt
 before = wai.simulate(agent, tools=TOOLS, tasks=base, runs=3)  # the same eval three times
@@ -979,7 +1004,9 @@ rerun = wai.simulate(
 assert base.rollouts_per_request == rerun.rollouts_per_request  # cheap guard
 ```
 
-**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way.
+**Size before you run.** Most evals are too small to see the effects they produce. Per-prompt paired spread is stable for agent rubrics, about 0.38 across five lanes, and at that spread a 50-prompt eval only detects a 10-point gain; a 5-point gain needs about 220 prompts and a 3-point gain about 600. One lane read the same adapter as "barely helps" at 45 and 85 prompts (both straddled zero) and cleared at 131 (+0.13 [+0.07, +0.19]); the effect was real the whole time and a GPU round went to fixing a data problem that was a measurement problem. The effective sample is prompts, not rollouts: raising k sharpens each prompt's estimate but does not narrow a bootstrap over prompts, so spend eval budget on prompts first (k still matters for preference and grouped methods, which need mixed groups). And "straddles zero" means the eval cannot tell, not that the model did not improve; say which. `holdout_size(effect, base=, k=)` says how many paired tasks prove a gain at 80% power, and `detectable_effect(n_tasks, ...)` is the same solved for the gain. Its binomial model assumes the gain is spread evenly across tasks and the two arms are independent draws, and says so in `notes`; when a trait is only exercised by some prompts most tasks are ties, the paired differences spread far wider, and the model under-sizes by several times (a voice lane at 0 -> 0.127 needed 54 tasks where the model said 14), so the model path also returns `n_tasks_concentrated`. On a holdout whose tasks differ in difficulty the model errs the other way, asking for `1 / (1 - Var(p_i) / (p(1-p)))` times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4); `before=` alone reports the spread and that ratio. The honest paths measure: `before=before, after=after` (the same two row lists `delta_report` takes) reads the per-task paired sd off both arms of a previous eval on the same tasks, with the covariance pairing buys in it, and `task_std=` (the per-task sibling of `run_std`) takes the number you read off a `delta_report` (`(hi - lo) * sqrt(n_paired_tasks) / 3.92` from `target_ci95`). `push(purpose="holdout")` warns when the set is too small to prove a 5-point gain.
+
+**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way. Both sides should have the same rollouts per task; when a run lost some (`data.report()["rollouts_lost"]`, with the reasons in `rollouts_lost_by`) and one arm sits at k=4 while the other is at k=2, the report warns next to the sizing line and names both. Unequal k is a precision issue, not a bias: rows lost at random leave the paired delta unbiased and only widen its interval; rows lost for a reason (a timeout on the hard runs) bias it, and only re-running the short arm on its short tasks fixes that. `balance_rollouts=True` (off by default) trims every paired task to the rows both sides have (drawn by `seed=`) so pass^k and pass@k share one k; it costs precision, removes no bias, and `balanced` says how many rows each side gave up.
 
 **Run the eval three times.** One evaluation is a draw, not a number: the same model on the same tasks lands somewhere else next time, and most post-training gains are inside that spread (rlhf-book ch. 16, appendix C). `wai.simulate(agent, tasks=base, runs=3)` replays the task set three times in one call, same tasks, faults and world, and stamps `lineage.eval_run` on every row. Feed both sides to `delta_report` and it works out `run_std` from the repeats itself. The verdict words: `moved` is a change the interval and the re-run band both support; `moved_unreplicated` is a change seen once, which could be noise, and the warning tells you the `runs=3` call that settles it; `within_eval_noise` is a delta smaller than what re-running the eval does on its own, so equivalence, not a win; `no_change_detected` is an interval that covers zero. `ceiling=True` means the before run already passes most of its tasks (0.9 or more, or too few paired tasks left with room), so there is little improvement the eval could show; use harder situations before training again.
 
@@ -1092,6 +1119,42 @@ run.holdout(before=0.42, after=0.58)  # did it work? the run page opens with thi
 
 A run's page opens with one word — **Better**, **Worse**, **About the same** — over the held-out pass rate before and after. The platform's trainer measures it; a run on your own hardware says it with `run.holdout(before, after)`, or `wai.attach_holdout(run_id, before=..., after=...)` once the run has finished. Pass rates are 0 to 1, so 58% is `0.58`; `metric="loss"` sends held-out loss instead (SFT), where lower is better. `run.delta(...)` and `wai.attach_delta(...)` already measure both sides, so they fill the two numbers in themselves, and add `summary["holdout"]` (also `run.holdout_summary`): each side's pass rate with `n_tasks`, `k` and a `ci95`, plus the delta report's verdict word (`moved`, `moved_unreplicated`, `within_eval_noise`, `no_change_detected`). A hosted run read back with `run.refresh()` has the same block with the interval fields `None` and a note that the platform only returned two numbers.
 
+### Report a run so a person can decide
+
+The platform draws one screen per tracked agent at [while.ai/platform/runs](https://while.ai/platform/runs): the held-out score by version with the frontier model as the line to beat, the training curve, what moved on the behaviors you did not train, the judge checks, live traffic on the served version, and cost. A coding agent fills it with `whileai.platform`; the person reads it and presses Promote. Your agent framework stays yours: `track` takes the agent object you already have (OpenAI Agents SDK, Pydantic AI, LangGraph, Claude Agent SDK) and reads the model, the instructions and the tools off it, or you describe it by hand.
+
+```python
+from whileai.platform import Behavior, Frontier, Harness, Judge, track
+
+tracked = track(
+    "refund-bot",  # or track(my_agent): name, model, prompt and tools come from the object
+    model="Qwen/Qwen3-4B",
+    harness=Harness(instructions=SYSTEM_PROMPT, tools=["lookup_order", "issue_refund"]),
+    frontier=Frontier(name="Sonnet 5", score=81, cost_per_1k=18.0),
+)
+tracked.behavior(
+    Behavior(
+        name="refunds",
+        test_version="v2",
+        n=240,
+        judge=Judge(agreement=0.86, human_n=60, length_bias=0.08),
+        noise_floor=2.4,
+        contamination=0,
+        reward_is_judge=False,
+    )
+)
+
+run = tracked.run("v4", method="GRPO", targets=["refunds"], trained_on=["refunds-grpo"])
+run.log(10, reward=0.41, kl=0.01)  # or trainer.add_callback(wai.TrainerCallback(run))
+run.score("refunds", 83, ci=2.7, n=240)  # every behavior, not only the targets
+run.score("length", 76, ci=2.8, n=120)
+run.finish(hours=2.1, gpu="1xH100", cost_usd=31)
+
+str(tracked.verdict())  # "refunds: v4 beats v3 by 5 (interval excludes zero); 1 regression"
+```
+
+Every object is a pydantic model that validates before it leaves the process, and each one's docstring names the rlhfbook.com chapter it comes from. A *harness* is the instructions, tools and model name around the weights; its fingerprint is its version, so a prompt edit shows up as a new version without anyone naming it (Evaluation: a score is only comparable with its setup held constant). A *behavior* has its own frozen held-out test (`test_version`), a `noise_floor` measured by scoring the same model twice, and a judge checked against people (`agreement` over `human_n`) and for `length_bias`. A *run* is scored on every behavior: `targets` are the claim, the rest are the check (Over-Optimization: verbosity, sycophancy and refusals are what moves when the reward is gamed). `ci` is the half-width of the 95% interval; the verdict is whether the candidate's and the served version's intervals overlap. `tracked.live(day, version=, replies=, flagged=)` reports a day of traffic when you serve the model yourself. Logging buffers and never raises into the training loop. Worked example: `recipes/04-train/report-run/`.
+
 ### Is it hacking the reward right now?
 
 ```python
@@ -1191,6 +1254,8 @@ Measured at `avg_turns=4`. The default is now `12`, so a row carries more turns
 | `embedder` | `"hash"` | Prompt selection |
 | `seed` | `0` | Reproducible draws. Bit-for-bit at `concurrency: 1` or with `reproducible=True`, within a process and across processes; otherwise which rows land before the cap depends on thread timing |
 | `avg_turns` | `12` | Target conversation length in turns. The person speaks at most `avg_turns // 2` times; `12` leaves room to verify, look up, confirm, and write. |
+| `mutate_graded_failures` | on with `grader=` | `False` grades beside the loop without steering by the verdict: only tool faults make mutation parents. A row the grader fails (reward under 0.5) is otherwise re-rolled and its ask mutated the way a tool fault's is; `search["mutation_aims"]` counts each aim. `True` without `grader=` is an error |
+| `patience` | `"normal"` | How long the person keeps answering the agent's questions. The first question is always attempted; from the second on the person may walk away (`normal`: 35% then 60%; `short`: 60% then 90%; `endless`: never, the pre-knob behaviour). At any question the person may also leave when it asks for something they could not know. A row the person left ends on the agent's question and carries `ended_by="user_left"`; `search["ended_on_question"]` counts them. |
 
 Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollouts_per_request`; `unique=` → `unique_situations`; `policy=` → `system_prompt`; `risk=` → `fault_rate`.
 
@@ -1198,7 +1263,7 @@ Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollo
 
 Each row, in `data.trajectories` and on disk: `prompt`, `messages`, `steps`, `final_text`, `scenario_id`. Optional `world_state`, `faults`, `reward`, `reason`. `llm_grade=True` adds `llm_reward`. `wai.rank(path)` adds `quality` without changing `reward`. Every row also says how it was sampled: `sampling` is `{"temperature", "max_tokens", "model"}` as the model backend resolved them, or what you passed as `simulate(sampling=...)` for your own callable agent (`None` when you passed nothing, since only you know how it samples). `policy_version` names the model and the system prompt it ran under.
 
-Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
+Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. Every row also names the deploy prompt it was generated under: `lineage.system_prompt_sha` (the hash `policy_version` carries after `@`) , `lineage.system_prompt_head` (its first 120 chars) and `lineage.system_prompt_chars`, with the full text once per run in `data.system_prompts[<sha>]`, so a base rate measured under a full policy is never mistaken for one measured under a bare prompt; `delta_report` warns when its two arms differ on that hash. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
 
 What goes to disk is the whole row, not a summary of it: `data.rows` (the same list `output=` and `save()` write, callable as `data.rows()` too) carries everything the trajectory carries, so a saved run can still prove its own provenance. That includes how the situation was drawn (`scenario_dimensions`, `arm`, `selection_reason`, `behavior_signature`, `seed`), who graded it and how that went (`judge_name`, `judge_status`, `judge_meta`, `lineage`, `label_source`), and what was measured on it (`markers`, read by `marker_summary` and `delta_report`). Two things never ship, at any depth of the row: the teacher-only `privileged` block and its `principle` / `hidden_state` / `reference` / `rubric` fields, which would put the answer key one step from a training file, and `vector`, the raw embedding the diversity search keeps in memory for the length of the run. A privileged block nested inside a carried field or a tool result is dropped the same way, before `messages` is rebuilt from the steps.
 

{whileai-0.63 → whileai-0.65}/README.md

@@ -673,6 +673,22 @@ rows, report = wai.drop_leaky_rows(data.trajectories, prod)
 And the loop closes on itself: `evaluate(rollouts, judge).failed_traces()`
 hands the failures straight back to `simulate(traces=...)`.
 
+**What traces can and cannot aim at.** Traces reproduce situations: the
+tools, faults and world states the deployed agent met. A failure that has
+a world-visible trigger (a tool timed out and the agent did not say so, a
+stale record was presented as current) is reproduced. A failure that lives
+in how the reply is worded (an unsupported claim, an estimate not labelled
+as one, two questions where one was asked for) has no trigger in the world,
+so traces alone cannot aim at it: measured on a 12-rule grader, every rule
+with a tool-result trigger was reproduced and every rule about the reply's
+wording was not (#285). For those, put the grader in the loop:
+with `simulate(..., grader=judge)` a row the grader fails is re-rolled and
+its ask mutated like a tool fault, and `data.search["mutation_aims"]` says
+how many parents and mutated rows each aim (`world_fault`,
+`graded_failure`) produced. The grader is the switch; to grade beside the
+loop and still steer by tool faults alone, pass
+`advanced={"mutate_graded_failures": False}`.
+
 If your traces are already on the platform, `wai.cut(agent="my-agent")`
 does the whole cut in one line — see
 [Training data out of traces](#training-data-out-of-traces).
@@ -867,7 +883,8 @@ The platform's "Make training data" button, as one line:
 
 ```python
 wai.send_score("4bf92f3577b34da6", 1.0)  # this run passed
-wai.cuts(agent="my-agent")  # what a cut would hold
+summary = wai.cuts(agent="my-agent")  # what a cut would hold
+print(wai.format_cuts(summary, agent="my-agent"))  # the traces page's sentence
 made = wai.cut(agent="my-agent", kind="rl")  # make it
 wai.pull(made["train"]["datasetId"], "train.jsonl")
 made["holdout"]["datasetId"]  # measure on this, never train on it
@@ -911,6 +928,13 @@ wai.style_markers(rows)  # no_boilerplate, no_hedging, no_apology, no_sycophancy
 wai.style_report(rows)["warnings"]  # "reward pays for hedging (corr +0.41 ...)"
 wai.refusal_report(benign_rows)  # over-refusal rate with a Wilson interval
 wai.compare_runs(run_a, run_b)  # paired delta with a 95% interval
+wai.holdout_size(
+    0.05, base=0.6, k=4
+)  # tasks to prove a 5-point gain; n_tasks_concentrated beside it
+wai.holdout_size(
+    0.05, before=before, after=after
+)  # the paired sd measured off a previous eval, no model
+wai.holdout_size(0.05, task_std=0.38)  # or the sd read off a delta_report interval
 wai.delta_report(before, after, target="pass_at_1", must_not_regress=["honest_after_fault"])
 wai.delta_report(before, after, target="pass_at_1", by="category")  # the target per kind of prompt
 before = wai.simulate(agent, tools=TOOLS, tasks=base, runs=3)  # the same eval three times
@@ -943,7 +967,9 @@ rerun = wai.simulate(
 assert base.rollouts_per_request == rerun.rollouts_per_request  # cheap guard
 ```
 
-**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way.
+**Size before you run.** Most evals are too small to see the effects they produce. Per-prompt paired spread is stable for agent rubrics, about 0.38 across five lanes, and at that spread a 50-prompt eval only detects a 10-point gain; a 5-point gain needs about 220 prompts and a 3-point gain about 600. One lane read the same adapter as "barely helps" at 45 and 85 prompts (both straddled zero) and cleared at 131 (+0.13 [+0.07, +0.19]); the effect was real the whole time and a GPU round went to fixing a data problem that was a measurement problem. The effective sample is prompts, not rollouts: raising k sharpens each prompt's estimate but does not narrow a bootstrap over prompts, so spend eval budget on prompts first (k still matters for preference and grouped methods, which need mixed groups). And "straddles zero" means the eval cannot tell, not that the model did not improve; say which. `holdout_size(effect, base=, k=)` says how many paired tasks prove a gain at 80% power, and `detectable_effect(n_tasks, ...)` is the same solved for the gain. Its binomial model assumes the gain is spread evenly across tasks and the two arms are independent draws, and says so in `notes`; when a trait is only exercised by some prompts most tasks are ties, the paired differences spread far wider, and the model under-sizes by several times (a voice lane at 0 -> 0.127 needed 54 tasks where the model said 14), so the model path also returns `n_tasks_concentrated`. On a holdout whose tasks differ in difficulty the model errs the other way, asking for `1 / (1 - Var(p_i) / (p(1-p)))` times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4); `before=` alone reports the spread and that ratio. The honest paths measure: `before=before, after=after` (the same two row lists `delta_report` takes) reads the per-task paired sd off both arms of a previous eval on the same tasks, with the covariance pairing buys in it, and `task_std=` (the per-task sibling of `run_std`) takes the number you read off a `delta_report` (`(hi - lo) * sqrt(n_paired_tasks) / 3.92` from `target_ci95`). `push(purpose="holdout")` warns when the set is too small to prove a 5-point gain.
+
+**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way. Both sides should have the same rollouts per task; when a run lost some (`data.report()["rollouts_lost"]`, with the reasons in `rollouts_lost_by`) and one arm sits at k=4 while the other is at k=2, the report warns next to the sizing line and names both. Unequal k is a precision issue, not a bias: rows lost at random leave the paired delta unbiased and only widen its interval; rows lost for a reason (a timeout on the hard runs) bias it, and only re-running the short arm on its short tasks fixes that. `balance_rollouts=True` (off by default) trims every paired task to the rows both sides have (drawn by `seed=`) so pass^k and pass@k share one k; it costs precision, removes no bias, and `balanced` says how many rows each side gave up.
 
 **Run the eval three times.** One evaluation is a draw, not a number: the same model on the same tasks lands somewhere else next time, and most post-training gains are inside that spread (rlhf-book ch. 16, appendix C). `wai.simulate(agent, tasks=base, runs=3)` replays the task set three times in one call, same tasks, faults and world, and stamps `lineage.eval_run` on every row. Feed both sides to `delta_report` and it works out `run_std` from the repeats itself. The verdict words: `moved` is a change the interval and the re-run band both support; `moved_unreplicated` is a change seen once, which could be noise, and the warning tells you the `runs=3` call that settles it; `within_eval_noise` is a delta smaller than what re-running the eval does on its own, so equivalence, not a win; `no_change_detected` is an interval that covers zero. `ceiling=True` means the before run already passes most of its tasks (0.9 or more, or too few paired tasks left with room), so there is little improvement the eval could show; use harder situations before training again.
 
@@ -1056,6 +1082,42 @@ run.holdout(before=0.42, after=0.58)  # did it work? the run page opens with thi
 
 A run's page opens with one word — **Better**, **Worse**, **About the same** — over the held-out pass rate before and after. The platform's trainer measures it; a run on your own hardware says it with `run.holdout(before, after)`, or `wai.attach_holdout(run_id, before=..., after=...)` once the run has finished. Pass rates are 0 to 1, so 58% is `0.58`; `metric="loss"` sends held-out loss instead (SFT), where lower is better. `run.delta(...)` and `wai.attach_delta(...)` already measure both sides, so they fill the two numbers in themselves, and add `summary["holdout"]` (also `run.holdout_summary`): each side's pass rate with `n_tasks`, `k` and a `ci95`, plus the delta report's verdict word (`moved`, `moved_unreplicated`, `within_eval_noise`, `no_change_detected`). A hosted run read back with `run.refresh()` has the same block with the interval fields `None` and a note that the platform only returned two numbers.
 
+### Report a run so a person can decide
+
+The platform draws one screen per tracked agent at [while.ai/platform/runs](https://while.ai/platform/runs): the held-out score by version with the frontier model as the line to beat, the training curve, what moved on the behaviors you did not train, the judge checks, live traffic on the served version, and cost. A coding agent fills it with `whileai.platform`; the person reads it and presses Promote. Your agent framework stays yours: `track` takes the agent object you already have (OpenAI Agents SDK, Pydantic AI, LangGraph, Claude Agent SDK) and reads the model, the instructions and the tools off it, or you describe it by hand.
+
+```python
+from whileai.platform import Behavior, Frontier, Harness, Judge, track
+
+tracked = track(
+    "refund-bot",  # or track(my_agent): name, model, prompt and tools come from the object
+    model="Qwen/Qwen3-4B",
+    harness=Harness(instructions=SYSTEM_PROMPT, tools=["lookup_order", "issue_refund"]),
+    frontier=Frontier(name="Sonnet 5", score=81, cost_per_1k=18.0),
+)
+tracked.behavior(
+    Behavior(
+        name="refunds",
+        test_version="v2",
+        n=240,
+        judge=Judge(agreement=0.86, human_n=60, length_bias=0.08),
+        noise_floor=2.4,
+        contamination=0,
+        reward_is_judge=False,
+    )
+)
+
+run = tracked.run("v4", method="GRPO", targets=["refunds"], trained_on=["refunds-grpo"])
+run.log(10, reward=0.41, kl=0.01)  # or trainer.add_callback(wai.TrainerCallback(run))
+run.score("refunds", 83, ci=2.7, n=240)  # every behavior, not only the targets
+run.score("length", 76, ci=2.8, n=120)
+run.finish(hours=2.1, gpu="1xH100", cost_usd=31)
+
+str(tracked.verdict())  # "refunds: v4 beats v3 by 5 (interval excludes zero); 1 regression"
+```
+
+Every object is a pydantic model that validates before it leaves the process, and each one's docstring names the rlhfbook.com chapter it comes from. A *harness* is the instructions, tools and model name around the weights; its fingerprint is its version, so a prompt edit shows up as a new version without anyone naming it (Evaluation: a score is only comparable with its setup held constant). A *behavior* has its own frozen held-out test (`test_version`), a `noise_floor` measured by scoring the same model twice, and a judge checked against people (`agreement` over `human_n`) and for `length_bias`. A *run* is scored on every behavior: `targets` are the claim, the rest are the check (Over-Optimization: verbosity, sycophancy and refusals are what moves when the reward is gamed). `ci` is the half-width of the 95% interval; the verdict is whether the candidate's and the served version's intervals overlap. `tracked.live(day, version=, replies=, flagged=)` reports a day of traffic when you serve the model yourself. Logging buffers and never raises into the training loop. Worked example: `recipes/04-train/report-run/`.
+
 ### Is it hacking the reward right now?
 
 ```python
@@ -1155,6 +1217,8 @@ Measured at `avg_turns=4`. The default is now `12`, so a row carries more turns
 | `embedder` | `"hash"` | Prompt selection |
 | `seed` | `0` | Reproducible draws. Bit-for-bit at `concurrency: 1` or with `reproducible=True`, within a process and across processes; otherwise which rows land before the cap depends on thread timing |
 | `avg_turns` | `12` | Target conversation length in turns. The person speaks at most `avg_turns // 2` times; `12` leaves room to verify, look up, confirm, and write. |
+| `mutate_graded_failures` | on with `grader=` | `False` grades beside the loop without steering by the verdict: only tool faults make mutation parents. A row the grader fails (reward under 0.5) is otherwise re-rolled and its ask mutated the way a tool fault's is; `search["mutation_aims"]` counts each aim. `True` without `grader=` is an error |
+| `patience` | `"normal"` | How long the person keeps answering the agent's questions. The first question is always attempted; from the second on the person may walk away (`normal`: 35% then 60%; `short`: 60% then 90%; `endless`: never, the pre-knob behaviour). At any question the person may also leave when it asks for something they could not know. A row the person left ends on the agent's question and carries `ended_by="user_left"`; `search["ended_on_question"]` counts them. |
 
 Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollouts_per_request`; `unique=` → `unique_situations`; `policy=` → `system_prompt`; `risk=` → `fault_rate`.
 
@@ -1162,7 +1226,7 @@ Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollo
 
 Each row, in `data.trajectories` and on disk: `prompt`, `messages`, `steps`, `final_text`, `scenario_id`. Optional `world_state`, `faults`, `reward`, `reason`. `llm_grade=True` adds `llm_reward`. `wai.rank(path)` adds `quality` without changing `reward`. Every row also says how it was sampled: `sampling` is `{"temperature", "max_tokens", "model"}` as the model backend resolved them, or what you passed as `simulate(sampling=...)` for your own callable agent (`None` when you passed nothing, since only you know how it samples). `policy_version` names the model and the system prompt it ran under.
 
-Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
+Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. Every row also names the deploy prompt it was generated under: `lineage.system_prompt_sha` (the hash `policy_version` carries after `@`) , `lineage.system_prompt_head` (its first 120 chars) and `lineage.system_prompt_chars`, with the full text once per run in `data.system_prompts[<sha>]`, so a base rate measured under a full policy is never mistaken for one measured under a bare prompt; `delta_report` warns when its two arms differ on that hash. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
 
 What goes to disk is the whole row, not a summary of it: `data.rows` (the same list `output=` and `save()` write, callable as `data.rows()` too) carries everything the trajectory carries, so a saved run can still prove its own provenance. That includes how the situation was drawn (`scenario_dimensions`, `arm`, `selection_reason`, `behavior_signature`, `seed`), who graded it and how that went (`judge_name`, `judge_status`, `judge_meta`, `lineage`, `label_source`), and what was measured on it (`markers`, read by `marker_summary` and `delta_report`). Two things never ship, at any depth of the row: the teacher-only `privileged` block and its `principle` / `hidden_state` / `reference` / `rubric` fields, which would put the answer key one step from a training file, and `vector`, the raw embedding the diversity search keeps in memory for the length of the run. A privileged block nested inside a carried field or a tool result is dropped the same way, before `messages` is rebuilt from the steps.
 

{whileai-0.63 → whileai-0.65}/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "whileai"
-version = "0.63"
+version = "0.65"
 description = "While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt."
 readme = "README.md"
 license = {text = "Apache-2.0"}
 requires-python = ">=3.10"
-dependencies = ["requests>=2.25.0"]
+dependencies = ["requests>=2.25.0", "pydantic>=2.0"]
 authors = [{name = "While"}]
 keywords = [
     "zp", "zeroproof", "while", "evals", "agent-evals",

{whileai-0.63 → whileai-0.65}/whileai/ingest.py

@@ -39,6 +39,13 @@ from whileai._env import getenv
 _DEFAULT_TRACE_URL = "https://api.zeroproofai.com"
 _GZIP_MAGIC = b"\x1f\x8b"
 
+# The resource attribute that names the dataset. The gate reads
+# `zeroproof.dataset` and nothing else, so sending only the whileai spelling
+# lands every batch in a dataset called `traces` whatever you asked for, with
+# a 202 that says so too late to notice. Both are written: the second costs
+# one attribute and means the rename needs no release.
+_DATASET_KEYS = ("zeroproof.dataset", "whileai.dataset")
+
 
 class WhileIngestError(Exception):
     """Raised when the gate rejects a trace batch."""
@@ -67,7 +74,7 @@ def otel_env(api_key: str, dataset: str = "traces", base_url: str | None = None)
         "OTEL_EXPORTER_OTLP_HEADERS": "x-api-key=" + api_key,
         "OTEL_EXPORTER_OTLP_PROTOCOL": "http/json",
         # Resource attribute the gate reads to name the dataset.
-        "OTEL_RESOURCE_ATTRIBUTES": "whileai.dataset=" + dataset,
+        "OTEL_RESOURCE_ATTRIBUTES": ",".join(k + "=" + dataset for k in _DATASET_KEYS),
     }
 
 
@@ -133,9 +140,9 @@ def ingest_traces(
     Push a local OTLP batch file end to end and return ``{datasetId, dataset,
     rows}``.
 
-    ``dataset`` overrides the dataset name by setting the
-    ``whileai.dataset`` resource attribute on every resourceSpan, which
-    requires reading the batch; leave it unset to send the bytes untouched.
+    ``dataset`` overrides the dataset name by setting the dataset resource
+    attribute on every resourceSpan, which requires reading the batch; leave
+    it unset to send the bytes untouched.
     """
     with open(file, "rb") as fh:
         body = fh.read()
@@ -148,9 +155,9 @@ def ingest_traces(
         for resource_span in batch.get("resourceSpans", []):
             resource = resource_span.setdefault("resource", {})
             attributes = [
-                a for a in resource.get("attributes", []) if a.get("key") != "whileai.dataset"
+                a for a in resource.get("attributes", []) if a.get("key") not in _DATASET_KEYS
             ]
-            attributes.append({"key": "whileai.dataset", "value": {"stringValue": dataset}})
+            attributes += [{"key": k, "value": {"stringValue": dataset}} for k in _DATASET_KEYS]
             resource["attributes"] = attributes
         body = json.dumps(batch).encode("utf-8")