whileai 0.63__tar.gz → 0.64__tar.gz

{whileai-0.63/whileai.egg-info → whileai-0.64}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.63
+Version: 0.64
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -709,6 +709,22 @@ rows, report = wai.drop_leaky_rows(data.trajectories, prod)
 And the loop closes on itself: `evaluate(rollouts, judge).failed_traces()`
 hands the failures straight back to `simulate(traces=...)`.
 
+**What traces can and cannot aim at.** Traces reproduce situations: the
+tools, faults and world states the deployed agent met. A failure that has
+a world-visible trigger (a tool timed out and the agent did not say so, a
+stale record was presented as current) is reproduced. A failure that lives
+in how the reply is worded (an unsupported claim, an estimate not labelled
+as one, two questions where one was asked for) has no trigger in the world,
+so traces alone cannot aim at it: measured on a 12-rule grader, every rule
+with a tool-result trigger was reproduced and every rule about the reply's
+wording was not (#285). For those, put the grader in the loop:
+with `simulate(..., grader=judge)` a row the grader fails is re-rolled and
+its ask mutated like a tool fault, and `data.search["mutation_aims"]` says
+how many parents and mutated rows each aim (`world_fault`,
+`graded_failure`) produced. The grader is the switch; to grade beside the
+loop and still steer by tool faults alone, pass
+`advanced={"mutate_graded_failures": False}`.
+
 If your traces are already on the platform, `wai.cut(agent="my-agent")`
 does the whole cut in one line — see
 [Training data out of traces](#training-data-out-of-traces).
@@ -947,6 +963,13 @@ wai.style_markers(rows)  # no_boilerplate, no_hedging, no_apology, no_sycophancy
 wai.style_report(rows)["warnings"]  # "reward pays for hedging (corr +0.41 ...)"
 wai.refusal_report(benign_rows)  # over-refusal rate with a Wilson interval
 wai.compare_runs(run_a, run_b)  # paired delta with a 95% interval
+wai.holdout_size(
+    0.05, base=0.6, k=4
+)  # tasks to prove a 5-point gain; n_tasks_concentrated beside it
+wai.holdout_size(
+    0.05, before=before, after=after
+)  # the paired sd measured off a previous eval, no model
+wai.holdout_size(0.05, task_std=0.38)  # or the sd read off a delta_report interval
 wai.delta_report(before, after, target="pass_at_1", must_not_regress=["honest_after_fault"])
 wai.delta_report(before, after, target="pass_at_1", by="category")  # the target per kind of prompt
 before = wai.simulate(agent, tools=TOOLS, tasks=base, runs=3)  # the same eval three times
@@ -979,7 +1002,9 @@ rerun = wai.simulate(
 assert base.rollouts_per_request == rerun.rollouts_per_request  # cheap guard
 ```
 
-**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way.
+**Size before you run.** Most evals are too small to see the effects they produce. Per-prompt paired spread is stable for agent rubrics, about 0.38 across five lanes, and at that spread a 50-prompt eval only detects a 10-point gain; a 5-point gain needs about 220 prompts and a 3-point gain about 600. One lane read the same adapter as "barely helps" at 45 and 85 prompts (both straddled zero) and cleared at 131 (+0.13 [+0.07, +0.19]); the effect was real the whole time and a GPU round went to fixing a data problem that was a measurement problem. The effective sample is prompts, not rollouts: raising k sharpens each prompt's estimate but does not narrow a bootstrap over prompts, so spend eval budget on prompts first (k still matters for preference and grouped methods, which need mixed groups). And "straddles zero" means the eval cannot tell, not that the model did not improve; say which. `holdout_size(effect, base=, k=)` says how many paired tasks prove a gain at 80% power, and `detectable_effect(n_tasks, ...)` is the same solved for the gain. Its binomial model assumes the gain is spread evenly across tasks and the two arms are independent draws, and says so in `notes`; when a trait is only exercised by some prompts most tasks are ties, the paired differences spread far wider, and the model under-sizes by several times (a voice lane at 0 -> 0.127 needed 54 tasks where the model said 14), so the model path also returns `n_tasks_concentrated`. On a holdout whose tasks differ in difficulty the model errs the other way, asking for `1 / (1 - Var(p_i) / (p(1-p)))` times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4); `before=` alone reports the spread and that ratio. The honest paths measure: `before=before, after=after` (the same two row lists `delta_report` takes) reads the per-task paired sd off both arms of a previous eval on the same tasks, with the covariance pairing buys in it, and `task_std=` (the per-task sibling of `run_std`) takes the number you read off a `delta_report` (`(hi - lo) * sqrt(n_paired_tasks) / 3.92` from `target_ci95`). `push(purpose="holdout")` warns when the set is too small to prove a 5-point gain.
+
+**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way. Both sides should have the same rollouts per task; when a run lost some (`data.report()["rollouts_lost"]`, with the reasons in `rollouts_lost_by`) and one arm sits at k=4 while the other is at k=2, the report warns next to the sizing line and names both. Unequal k is a precision issue, not a bias: rows lost at random leave the paired delta unbiased and only widen its interval; rows lost for a reason (a timeout on the hard runs) bias it, and only re-running the short arm on its short tasks fixes that. `balance_rollouts=True` (off by default) trims every paired task to the rows both sides have (drawn by `seed=`) so pass^k and pass@k share one k; it costs precision, removes no bias, and `balanced` says how many rows each side gave up.
 
 **Run the eval three times.** One evaluation is a draw, not a number: the same model on the same tasks lands somewhere else next time, and most post-training gains are inside that spread (rlhf-book ch. 16, appendix C). `wai.simulate(agent, tasks=base, runs=3)` replays the task set three times in one call, same tasks, faults and world, and stamps `lineage.eval_run` on every row. Feed both sides to `delta_report` and it works out `run_std` from the repeats itself. The verdict words: `moved` is a change the interval and the re-run band both support; `moved_unreplicated` is a change seen once, which could be noise, and the warning tells you the `runs=3` call that settles it; `within_eval_noise` is a delta smaller than what re-running the eval does on its own, so equivalence, not a win; `no_change_detected` is an interval that covers zero. `ceiling=True` means the before run already passes most of its tasks (0.9 or more, or too few paired tasks left with room), so there is little improvement the eval could show; use harder situations before training again.
 
@@ -1191,6 +1216,8 @@ Measured at `avg_turns=4`. The default is now `12`, so a row carries more turns
 | `embedder` | `"hash"` | Prompt selection |
 | `seed` | `0` | Reproducible draws. Bit-for-bit at `concurrency: 1` or with `reproducible=True`, within a process and across processes; otherwise which rows land before the cap depends on thread timing |
 | `avg_turns` | `12` | Target conversation length in turns. The person speaks at most `avg_turns // 2` times; `12` leaves room to verify, look up, confirm, and write. |
+| `mutate_graded_failures` | on with `grader=` | `False` grades beside the loop without steering by the verdict: only tool faults make mutation parents. A row the grader fails (reward under 0.5) is otherwise re-rolled and its ask mutated the way a tool fault's is; `search["mutation_aims"]` counts each aim. `True` without `grader=` is an error |
+| `patience` | `"normal"` | How long the person keeps answering the agent's questions. The first question is always attempted; from the second on the person may walk away (`normal`: 35% then 60%; `short`: 60% then 90%; `endless`: never, the pre-knob behaviour). At any question the person may also leave when it asks for something they could not know. A row the person left ends on the agent's question and carries `ended_by="user_left"`; `search["ended_on_question"]` counts them. |
 
 Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollouts_per_request`; `unique=` → `unique_situations`; `policy=` → `system_prompt`; `risk=` → `fault_rate`.
 
@@ -1198,7 +1225,7 @@ Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollo
 
 Each row, in `data.trajectories` and on disk: `prompt`, `messages`, `steps`, `final_text`, `scenario_id`. Optional `world_state`, `faults`, `reward`, `reason`. `llm_grade=True` adds `llm_reward`. `wai.rank(path)` adds `quality` without changing `reward`. Every row also says how it was sampled: `sampling` is `{"temperature", "max_tokens", "model"}` as the model backend resolved them, or what you passed as `simulate(sampling=...)` for your own callable agent (`None` when you passed nothing, since only you know how it samples). `policy_version` names the model and the system prompt it ran under.
 
-Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
+Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. Every row also names the deploy prompt it was generated under: `lineage.system_prompt_sha` (the hash `policy_version` carries after `@`) , `lineage.system_prompt_head` (its first 120 chars) and `lineage.system_prompt_chars`, with the full text once per run in `data.system_prompts[<sha>]`, so a base rate measured under a full policy is never mistaken for one measured under a bare prompt; `delta_report` warns when its two arms differ on that hash. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
 
 What goes to disk is the whole row, not a summary of it: `data.rows` (the same list `output=` and `save()` write, callable as `data.rows()` too) carries everything the trajectory carries, so a saved run can still prove its own provenance. That includes how the situation was drawn (`scenario_dimensions`, `arm`, `selection_reason`, `behavior_signature`, `seed`), who graded it and how that went (`judge_name`, `judge_status`, `judge_meta`, `lineage`, `label_source`), and what was measured on it (`markers`, read by `marker_summary` and `delta_report`). Two things never ship, at any depth of the row: the teacher-only `privileged` block and its `principle` / `hidden_state` / `reference` / `rubric` fields, which would put the answer key one step from a training file, and `vector`, the raw embedding the diversity search keeps in memory for the length of the run. A privileged block nested inside a carried field or a tool result is dropped the same way, before `messages` is rebuilt from the steps.
 

{whileai-0.63 → whileai-0.64}/README.md

@@ -673,6 +673,22 @@ rows, report = wai.drop_leaky_rows(data.trajectories, prod)
 And the loop closes on itself: `evaluate(rollouts, judge).failed_traces()`
 hands the failures straight back to `simulate(traces=...)`.
 
+**What traces can and cannot aim at.** Traces reproduce situations: the
+tools, faults and world states the deployed agent met. A failure that has
+a world-visible trigger (a tool timed out and the agent did not say so, a
+stale record was presented as current) is reproduced. A failure that lives
+in how the reply is worded (an unsupported claim, an estimate not labelled
+as one, two questions where one was asked for) has no trigger in the world,
+so traces alone cannot aim at it: measured on a 12-rule grader, every rule
+with a tool-result trigger was reproduced and every rule about the reply's
+wording was not (#285). For those, put the grader in the loop:
+with `simulate(..., grader=judge)` a row the grader fails is re-rolled and
+its ask mutated like a tool fault, and `data.search["mutation_aims"]` says
+how many parents and mutated rows each aim (`world_fault`,
+`graded_failure`) produced. The grader is the switch; to grade beside the
+loop and still steer by tool faults alone, pass
+`advanced={"mutate_graded_failures": False}`.
+
 If your traces are already on the platform, `wai.cut(agent="my-agent")`
 does the whole cut in one line — see
 [Training data out of traces](#training-data-out-of-traces).
@@ -911,6 +927,13 @@ wai.style_markers(rows)  # no_boilerplate, no_hedging, no_apology, no_sycophancy
 wai.style_report(rows)["warnings"]  # "reward pays for hedging (corr +0.41 ...)"
 wai.refusal_report(benign_rows)  # over-refusal rate with a Wilson interval
 wai.compare_runs(run_a, run_b)  # paired delta with a 95% interval
+wai.holdout_size(
+    0.05, base=0.6, k=4
+)  # tasks to prove a 5-point gain; n_tasks_concentrated beside it
+wai.holdout_size(
+    0.05, before=before, after=after
+)  # the paired sd measured off a previous eval, no model
+wai.holdout_size(0.05, task_std=0.38)  # or the sd read off a delta_report interval
 wai.delta_report(before, after, target="pass_at_1", must_not_regress=["honest_after_fault"])
 wai.delta_report(before, after, target="pass_at_1", by="category")  # the target per kind of prompt
 before = wai.simulate(agent, tools=TOOLS, tasks=base, runs=3)  # the same eval three times
@@ -943,7 +966,9 @@ rerun = wai.simulate(
 assert base.rollouts_per_request == rerun.rollouts_per_request  # cheap guard
 ```
 
-**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way.
+**Size before you run.** Most evals are too small to see the effects they produce. Per-prompt paired spread is stable for agent rubrics, about 0.38 across five lanes, and at that spread a 50-prompt eval only detects a 10-point gain; a 5-point gain needs about 220 prompts and a 3-point gain about 600. One lane read the same adapter as "barely helps" at 45 and 85 prompts (both straddled zero) and cleared at 131 (+0.13 [+0.07, +0.19]); the effect was real the whole time and a GPU round went to fixing a data problem that was a measurement problem. The effective sample is prompts, not rollouts: raising k sharpens each prompt's estimate but does not narrow a bootstrap over prompts, so spend eval budget on prompts first (k still matters for preference and grouped methods, which need mixed groups). And "straddles zero" means the eval cannot tell, not that the model did not improve; say which. `holdout_size(effect, base=, k=)` says how many paired tasks prove a gain at 80% power, and `detectable_effect(n_tasks, ...)` is the same solved for the gain. Its binomial model assumes the gain is spread evenly across tasks and the two arms are independent draws, and says so in `notes`; when a trait is only exercised by some prompts most tasks are ties, the paired differences spread far wider, and the model under-sizes by several times (a voice lane at 0 -> 0.127 needed 54 tasks where the model said 14), so the model path also returns `n_tasks_concentrated`. On a holdout whose tasks differ in difficulty the model errs the other way, asking for `1 / (1 - Var(p_i) / (p(1-p)))` times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4); `before=` alone reports the spread and that ratio. The honest paths measure: `before=before, after=after` (the same two row lists `delta_report` takes) reads the per-task paired sd off both arms of a previous eval on the same tasks, with the covariance pairing buys in it, and `task_std=` (the per-task sibling of `run_std`) takes the number you read off a `delta_report` (`(hi - lo) * sqrt(n_paired_tasks) / 3.92` from `target_ci95`). `push(purpose="holdout")` warns when the set is too small to prove a 5-point gain.
+
+**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way. Both sides should have the same rollouts per task; when a run lost some (`data.report()["rollouts_lost"]`, with the reasons in `rollouts_lost_by`) and one arm sits at k=4 while the other is at k=2, the report warns next to the sizing line and names both. Unequal k is a precision issue, not a bias: rows lost at random leave the paired delta unbiased and only widen its interval; rows lost for a reason (a timeout on the hard runs) bias it, and only re-running the short arm on its short tasks fixes that. `balance_rollouts=True` (off by default) trims every paired task to the rows both sides have (drawn by `seed=`) so pass^k and pass@k share one k; it costs precision, removes no bias, and `balanced` says how many rows each side gave up.
 
 **Run the eval three times.** One evaluation is a draw, not a number: the same model on the same tasks lands somewhere else next time, and most post-training gains are inside that spread (rlhf-book ch. 16, appendix C). `wai.simulate(agent, tasks=base, runs=3)` replays the task set three times in one call, same tasks, faults and world, and stamps `lineage.eval_run` on every row. Feed both sides to `delta_report` and it works out `run_std` from the repeats itself. The verdict words: `moved` is a change the interval and the re-run band both support; `moved_unreplicated` is a change seen once, which could be noise, and the warning tells you the `runs=3` call that settles it; `within_eval_noise` is a delta smaller than what re-running the eval does on its own, so equivalence, not a win; `no_change_detected` is an interval that covers zero. `ceiling=True` means the before run already passes most of its tasks (0.9 or more, or too few paired tasks left with room), so there is little improvement the eval could show; use harder situations before training again.
 
@@ -1155,6 +1180,8 @@ Measured at `avg_turns=4`. The default is now `12`, so a row carries more turns
 | `embedder` | `"hash"` | Prompt selection |
 | `seed` | `0` | Reproducible draws. Bit-for-bit at `concurrency: 1` or with `reproducible=True`, within a process and across processes; otherwise which rows land before the cap depends on thread timing |
 | `avg_turns` | `12` | Target conversation length in turns. The person speaks at most `avg_turns // 2` times; `12` leaves room to verify, look up, confirm, and write. |
+| `mutate_graded_failures` | on with `grader=` | `False` grades beside the loop without steering by the verdict: only tool faults make mutation parents. A row the grader fails (reward under 0.5) is otherwise re-rolled and its ask mutated the way a tool fault's is; `search["mutation_aims"]` counts each aim. `True` without `grader=` is an error |
+| `patience` | `"normal"` | How long the person keeps answering the agent's questions. The first question is always attempted; from the second on the person may walk away (`normal`: 35% then 60%; `short`: 60% then 90%; `endless`: never, the pre-knob behaviour). At any question the person may also leave when it asks for something they could not know. A row the person left ends on the agent's question and carries `ended_by="user_left"`; `search["ended_on_question"]` counts them. |
 
 Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollouts_per_request`; `unique=` → `unique_situations`; `policy=` → `system_prompt`; `risk=` → `fault_rate`.
 
@@ -1162,7 +1189,7 @@ Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollo
 
 Each row, in `data.trajectories` and on disk: `prompt`, `messages`, `steps`, `final_text`, `scenario_id`. Optional `world_state`, `faults`, `reward`, `reason`. `llm_grade=True` adds `llm_reward`. `wai.rank(path)` adds `quality` without changing `reward`. Every row also says how it was sampled: `sampling` is `{"temperature", "max_tokens", "model"}` as the model backend resolved them, or what you passed as `simulate(sampling=...)` for your own callable agent (`None` when you passed nothing, since only you know how it samples). `policy_version` names the model and the system prompt it ran under.
 
-Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
+Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. Every row also names the deploy prompt it was generated under: `lineage.system_prompt_sha` (the hash `policy_version` carries after `@`) , `lineage.system_prompt_head` (its first 120 chars) and `lineage.system_prompt_chars`, with the full text once per run in `data.system_prompts[<sha>]`, so a base rate measured under a full policy is never mistaken for one measured under a bare prompt; `delta_report` warns when its two arms differ on that hash. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
 
 What goes to disk is the whole row, not a summary of it: `data.rows` (the same list `output=` and `save()` write, callable as `data.rows()` too) carries everything the trajectory carries, so a saved run can still prove its own provenance. That includes how the situation was drawn (`scenario_dimensions`, `arm`, `selection_reason`, `behavior_signature`, `seed`), who graded it and how that went (`judge_name`, `judge_status`, `judge_meta`, `lineage`, `label_source`), and what was measured on it (`markers`, read by `marker_summary` and `delta_report`). Two things never ship, at any depth of the row: the teacher-only `privileged` block and its `principle` / `hidden_state` / `reference` / `rubric` fields, which would put the answer key one step from a training file, and `vector`, the raw embedding the diversity search keeps in memory for the length of the run. A privileged block nested inside a carried field or a tool result is dropped the same way, before `messages` is rebuilt from the steps.
 

{whileai-0.63 → whileai-0.64}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "whileai"
-version = "0.63"
+version = "0.64"
 description = "While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt."
 readme = "README.md"
 license = {text = "Apache-2.0"}

{whileai-0.63 → whileai-0.64}/whileai/simulations/data.py

@@ -366,6 +366,10 @@ class SimulationData:
     # ``model_version`` on every row.
     writer_model: str = ""
     user_model: str | None = None
+    # The deploy prompt the rows were generated under, full text keyed by
+    # the hash every row carries in ``lineage.system_prompt_sha`` (#296).
+    # Those two are the hash's only homes: the row and this text.
+    system_prompts: dict[str, str] = field(default_factory=dict)
 
     @property
     def judge_model(self) -> str | None:
@@ -837,6 +841,10 @@ class SimulationData:
                         # The agent spec: a trainer loading this JSONL later
                         # needs the policy and tool schemas the run knew.
                         "system_prompt": str(getattr(self.profile, "policy", "") or ""),
+                        # the exact text the rows were generated under
+                        # (policy plus scaffold), keyed by the hash each
+                        # row carries in lineage.system_prompt_sha
+                        "system_prompts": dict(self.system_prompts),
                         "tools": list(getattr(self.profile, "tools", None) or []),
                         "stopped_because": self.stopped_because,
                         "coverage": self.coverage,

{whileai-0.63 → whileai-0.64}/whileai/simulations/generate/adapters.py

@@ -343,6 +343,7 @@ def resolve(
     max_turns: int = 5,
     avg_turns: float = 6,
     min_user_turns: int = 1,
+    patience: str = "normal",
     turn_stats: dict | None = None,
     opening_rate: float = 0.0,
     temperature: float | None = None,
@@ -378,6 +379,7 @@ def resolve(
             fault_plans=fault_plans,
             avg_turns=avg_turns,
             min_user_turns=min_user_turns,
+            patience=patience,
             opening_rate=opening_rate,
             execute=execute,
             result_shapes=result_shapes,

{whileai-0.63 → whileai-0.64}/whileai/simulations/generate/agents.py

@@ -911,8 +911,122 @@ def _agent_asked(text: str) -> bool:
     return bool(t) and ("?" in t)
 
 
+#: ``simulate(patience=)``: how long the simulated person keeps answering
+#: the agent's questions. The first question is always attempted; from the
+#: second on the person may walk away at these odds, per level. ``endless``
+#: is what every run did before the knob existed: every question answered
+#: until the depth cap, so whether a thread ended was decided by the turn
+#: budget and never by what the agent said, and no rubric criterion about
+#: asking could fail (#289). The odds are a default, not a measurement: to
+#: ground them, fit a Kaplan-Meier hazard per question index on source
+#: traces (of the threads still answering at question k, the share that
+#: leave at k) and set the levels from it. The 63% of asked source threads
+#: that ended with the person walking away says the shape is common, not
+#: what the per-question hazard is.
+PATIENCE_LEVELS = ("short", "normal", "endless")
+_WALK_AWAY_ODDS = {
+    "short": (0.6, 0.9),
+    "normal": (0.35, 0.6),
+    "endless": (0.0, 0.0),
+}
+#: What ``_user_followup`` returns when the person gives up on the question.
+USER_LEFT = "[leaves]"
+_LEAVES = re.compile(r"\W*leaves\W*", re.I)
+
+
+def _draw(message: str, turn_i: int, salt: str) -> float:
+    """A uniform draw in [0, 1), fixed by the message and turn so a seeded
+    run reproduces exactly."""
+    digest = hashlib.sha256(f"{message}:{turn_i}:{salt}".encode()).hexdigest()
+    return int(digest[:8], 16) / float(1 << 32)
+
+
+def _walk_away_hazard(patience: str, questions: int) -> float:
+    """Chance the person leaves instead of answering, given how many
+    questions the agent already asked on this thread (not counting this
+    one). Zero on the first question: the person always tries once."""
+    second, later = _WALK_AWAY_ODDS.get(
+        str(patience or "normal").lower(), _WALK_AWAY_ODDS["normal"]
+    )
+    if questions <= 0:
+        return 0.0
+    return second if questions == 1 else later
+
+
+def _user_turn_cap(budget: int) -> int:
+    """How many times the person speaks at most on a thread of ``budget``
+    turns. The budget's rule, kept apart from the person's patience: a
+    thread the cap ends is not one the person left."""
+    return max(2, int(budget) // 2)
+
+
+def _user_walks_away(
+    message: str,
+    turn_i: int,
+    *,
+    questions: int = 0,
+    patience: str = "normal",
+) -> bool:
+    """True when the person gives up on the agent's question rather than
+    answer it (#289): the patience hazard at this question index, and
+    nothing about the budget. Deterministic in the message and turn, so a
+    seeded run reproduces."""
+    hazard = _walk_away_hazard(patience, questions)
+    if hazard <= 0:
+        return False
+    return _draw(message, turn_i, "abandon") < hazard
+
+
+def _user_left(text: str) -> bool:
+    """The user-sim's answer was the leave mark and nothing else."""
+    body = re.sub(r"<think>.*?</think>", "", str(text or ""), flags=re.S).strip()
+    return bool(body) and bool(_LEAVES.fullmatch(body))
+
+
+def ended_on_question(rows) -> dict:
+    """``{"share", "n", "user_left"}``: of ``n`` rows, the share that ended
+    on the agent's question, and how many of those the person left
+    (``ended_by="user_left"``; the rest hit the turn budget).
+
+    Report this next to any criterion about how the agent asks. Before #289
+    it was structurally zero, so a lane could not tell "the agent never asked
+    badly" from "the loop cannot produce that ending". The engine writes it
+    to ``search["ended_on_question"]`` on every run.
+    """
+    total = 0
+    ended = 0
+    left = 0
+    for row in rows or []:
+        if not isinstance(row, dict):
+            continue
+        total += 1
+        turns = row.get("messages") or row.get("trajectory") or []
+        last_agent = ""
+        for turn in turns:
+            if isinstance(turn, dict) and turn.get("role") == "assistant":
+                last_agent = str(turn.get("content") or "")
+        if not turns:
+            last_agent = str(row.get("final_text") or "")
+        if last_agent and _agent_asked(last_agent):
+            ended += 1
+        if row.get("ended_by") == "user_left":
+            left += 1
+    return {
+        "share": (ended / total if total else 0.0),
+        "n": total,
+        "user_left": left,
+    }
+
+
 def _want_followup(
-    message: str, turn_i: int, *, user_turns: int = 1, budget: int = 6, agent_text: str = ""
+    message: str,
+    turn_i: int,
+    *,
+    user_turns: int = 1,
+    budget: int = 6,
+    agent_text: str = "",
+    questions: int = 0,
+    patience: str = "normal",
 ) -> bool:
     """True when the human would naturally speak again.
 
@@ -933,13 +1047,28 @@ def _want_followup(
     and asks, the user says yes, the agent acts. At 1.5 user turns most
     rollouts never reach the write, so the rule is never exercised and the
     training set cannot demonstrate it.
+
+    A question no longer earns an answer unconditionally. ``questions`` is
+    how many the agent already asked on this thread; from the second one
+    the person may walk away at the odds ``patience`` sets (#289). Before
+    this, whether a thread ended was decided by the turn budget and never
+    by what the agent said, so no rubric criterion about asking could
+    fail: in source traces 63% of asked threads ended with the person
+    walking away, in generated data 19-22%, all of those depth-cap cuts.
     """
-    cap = max(2, int(budget) // 2)
+    cap = _user_turn_cap(budget)
     if int(user_turns) >= cap:
         return False
     text = str(agent_text or "")
     if _agent_asked(text):
-        return True
+        # Asking is free in simulation and costly in reality. This used to be
+        # an unconditional True (#289); now the person always tries the first
+        # question and from the second on may walk away at the odds
+        # ``patience`` sets. Short threads (budget under 4) never reach a
+        # second question, so the old contract holds there: with room for
+        # one exchange, abandoning would leave the question as the whole
+        # rollout and fill the set with stubs.
+        return not _user_walks_away(message, turn_i, questions=questions, patience=patience)
     if _AGENT_REFUSAL.search(text):
         return True
     if int(budget) < 4:
@@ -948,9 +1077,7 @@ def _want_followup(
         return False
     # Geometric with p = 1 - 1/cap: a thread of cap turns in expectation,
     # deterministic in the message and turn so a seeded run reproduces.
-    digest = hashlib.sha256(f"{message}:{turn_i}:react".encode()).hexdigest()
-    draw = int(digest[:8], 16) / float(1 << 32)
-    return draw < (1.0 - 1.0 / float(cap))
+    return _draw(message, turn_i, "react") < (1.0 - 1.0 / float(cap))
 
 
 # Follow-up user turns only. Opener temperature lives on the writer (0.45–1.05).
@@ -962,6 +1089,7 @@ _USER_SIM_SYSTEM = (
     "Stay in the same world as the opening line and the tools on this thread. "
     "If the agent asked a question, answer it with a concrete detail a person "
     "here would know ({hints}whatever this thread is actually about). "
+    "{leave_note}"
     "{code_note}"
     "If they already acted, react: push back, correct them, or ask for the next thing. "
     "Do not acknowledge. Do not repeat their question. Do not describe a persona."
@@ -988,8 +1116,17 @@ def _detail_hints(tools: list | None) -> list[str]:
     return out[:8]
 
 
-def user_sim_system(tools: list | None = None) -> str:
-    """The user simulator's instructions for this agent's world."""
+_LEAVE_NOTE = (
+    "Only when their question asks for something this person could not know, "
+    "or asks again what was already answered on this thread, the person may "
+    "give up instead: then write exactly [leaves] and nothing else. "
+)
+
+
+def user_sim_system(tools: list | None = None, may_leave: bool = False) -> str:
+    """The user simulator's instructions for this agent's world.
+    ``may_leave`` adds the one way out: a question this person cannot or
+    would not answer may be met with the leave mark (#289)."""
     hints = _detail_hints(tools)
     names = " ".join(_tool_world(tools).split(",")) + " " + " ".join(hints)
     code_note = (
@@ -999,7 +1136,9 @@ def user_sim_system(tools: list | None = None) -> str:
         else ""
     )
     return _USER_SIM_SYSTEM.format(
-        hints=(", ".join(hints) + ", ") if hints else "", code_note=code_note
+        hints=(", ".join(hints) + ", ") if hints else "",
+        leave_note=_LEAVE_NOTE if may_leave else "",
+        code_note=code_note,
     )
 
 
@@ -1255,6 +1394,7 @@ def _user_followup(
     persona_tags: dict | None = None,
     extra: Mapping[str, Any] | None = None,
     turn_stats: dict | None = None,
+    may_leave: bool = False,
 ) -> str:
     """The simulated user's next line, or ``""`` when the writer produced
     none worth keeping. ``extra`` carries the agent's request fields
@@ -1330,7 +1470,7 @@ def _user_followup(
                 base_url,
                 model,
                 [
-                    {"role": "system", "content": user_sim_system(tools)},
+                    {"role": "system", "content": user_sim_system(tools, may_leave=may_leave)},
                     {"role": "user", "content": content},
                 ],
                 tools=None,
@@ -1344,6 +1484,11 @@ def _user_followup(
             continue
         spoken, closed, unclosed = split_reasoning(reply.get("content") or "")
         _note_user_turn(turn_stats, closed, unclosed)
+        # A leave mark is an answer of its own kind: read it before the
+        # reasoning-only retry rule, or a person who thinks then leaves is
+        # retried instead of gone (#289).
+        if may_leave and _user_left(reply.get("content") or ""):
+            return USER_LEFT
         if (closed or unclosed) and len(spoken.strip()) < _MIN_SPOKEN:
             # Reasoning and no spoken line. One more try with the short
             # prompt; never a fragment, never empty user speech.
@@ -1528,6 +1673,7 @@ def local_model(
     max_tokens: int | None = None,
     user_model: str | None = None,
     thinking: bool | None = None,
+    patience: str = "normal",
 ) -> Callable:
     """An agent that talks to an OpenAI-compatible endpoint (a served
     adapter, a local vLLM, any chat server) for ``simulate(agent=...)``.
@@ -1641,11 +1787,13 @@ def local_model(
                         len(messages) - 1, {"role": "assistant", "content": opener_text}
                     )
 
-        def _done(done_steps: list, final: str) -> dict:
+        def _done(done_steps: list, final: str, ended_by: str = "") -> dict:
             out = _finish_on_agent(done_steps, final)
             if opener_text:
                 out["opener"] = opener_text
                 out["opening"] = "agent"
+            if ended_by:
+                out["ended_by"] = ended_by
             return out
 
         steps: list[dict] = []
@@ -1783,10 +1931,21 @@ def local_model(
                 continue
             need_first = n_user < 2
             force_followup = n_user < min_users
+            # Questions the agent already asked on this thread, not counting
+            # this one: the person's patience runs out with them (#289).
+            prior_questions = sum(
+                1 for s in steps if isinstance(s, dict) and _agent_asked(str(s.get("text") or ""))
+            ) - (1 if _agent_asked(spoken) else 0)
             if (room or need_first) and (
                 force_followup
                 or _want_followup(
-                    message, turn_i, user_turns=n_user, budget=budget, agent_text=spoken
+                    message,
+                    turn_i,
+                    user_turns=n_user,
+                    budget=budget,
+                    agent_text=spoken,
+                    questions=prior_questions,
+                    patience=patience,
                 )
             ):
                 follow = _user_followup(
@@ -1804,7 +1963,11 @@ def local_model(
                     force=force_followup,
                     extra=user_extras,
                     turn_stats=turn_stats,
+                    may_leave=patience != "endless",
                 )
+                if follow == USER_LEFT:
+                    # The person could not or would not answer this one.
+                    return _done(steps, spoken or final_text, ended_by="user_left")
                 if follow:
                     last_user = follow
                     n_user += 1
@@ -1818,7 +1981,18 @@ def local_model(
                     # these means the dataset is going single-turn.
                     with turn_stats["lock"]:
                         turn_stats["followup_misses"] = turn_stats.get("followup_misses", 0) + 1
-            return _done(steps, spoken or final_text)
+                return _done(steps, spoken or final_text)
+            # No follow-up. Under the cap and asked, that was the person's
+            # patience; at the cap it was the budget, and the row says so
+            # by carrying no ended_by.
+            left = (
+                (room or need_first)
+                and not force_followup
+                and n_user < _user_turn_cap(budget)
+                and _agent_asked(spoken)
+                and _user_walks_away(message, turn_i, questions=prior_questions, patience=patience)
+            )
+            return _done(steps, spoken or final_text, ended_by="user_left" if left else "")
         return _done(steps, final_text)
 
     agent.__name__ = f"local_model[{model}]"

{whileai-0.63 → whileai-0.64}/whileai/simulations/generate/generator.py

@@ -1603,10 +1603,16 @@ class ModelSimulator:
         if isinstance(parent, str):
             return {"request": parent}
         if isinstance(parent, dict):
-            return {
+            detail = {
                 "id": parent.get("scenario_id") or parent.get("parent_failure_id"),
                 "request": str(parent.get("prompt") or parent.get("request") or "")[:1000],
             }
+            # What the grader said went wrong, so the retry card can name
+            # the broken rule instead of only that something broke (#285).
+            reason = parent.get("grader_reason") or parent.get("reason")
+            if reason and parent.get("reward") is not None:
+                detail["failed"] = str(reason)[:200]
+            return detail
         return {"request": str(parent)[:1000]}
 
     def _system_prompt(self) -> str:
@@ -1761,6 +1767,8 @@ region_id exactly and placing the human's words in message."""
             )
             if prior:
                 retry += f" Note: (earlier you said something like: {prior})"
+            if parent.get("failed"):
+                retry += f" What went wrong last time: {parent['failed']}"
             fail_card["instruction"] = retry
             targets.append(fail_card)
         parents = []