whileai 0.62__tar.gz → 0.64__tar.gz

{whileai-0.62/whileai.egg-info → whileai-0.64}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.62
+Version: 0.64
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -219,6 +219,18 @@ print(wai.pass_at(scored.rows), *scored.warnings)  # a hollow run says so here
 `scored.warnings` is new: no rollout called a tool, a declared tool no
 rollout touched, a marker that fired on no row. A 1.00 on a run like that is not a result; the note names the fix.
 
+A declared tool the world cannot answer is the quiet version of the same
+failure: with `execute=`, a tool that is in the schema but has no branch in
+your function fails exactly like a world fault, the agent reports the miss
+honestly, and a candour rubric rewards the row. Every run now records calls
+and successes per tool in `data.coverage["tools"]` (`n`, `ok`, `fault_n`, and
+`injected` for faults the run scheduled itself), lists the tools that never
+work in `data.coverage["dead_tools"]` (the Wilson 95% upper bound on the
+success rate is under 0.30, so 0 of 9 or 4 of 612 is dead and 0 of 3 or 2 of
+5 is not), adds `dead_tools` to `data.degraded`, and puts the names and the
+one fix that applies to your world in `data.warnings` and `data.report()`.
+Steps with no recorded result are not evidence and never accuse a tool.
+
 ### Bring a model
 
 Bring your own model. Any OpenAI-compatible chat endpoint that returns tool
@@ -227,7 +239,31 @@ your key. To put a number on a model you serve (`wai.serve`, or your own
 vLLM), make it the agent: `wai.simulate(tasks=pinned,
 agent=wai.local_model(endpoint, name, tools=TOOLS, system=POLICY,
 thinking=False))`, and run both arms of a before/after through that same
-call so the only difference is the weights.
+call so the only difference is the weights. `thinking=False` reaches the
+simulated user as well when the agent's own model plays it (the default),
+and whatever a user model still emits as `<think>` is stripped before it
+becomes a user turn; the run reports those under
+`data.search["user_think"]` (counts and shares of the user turns, zeros when
+none) and says so in `data.warnings`. `delta_report` fails a before/after
+whose arms differ in how often they answered at all (`"answered"` in
+`not_comparable`, a two-proportion test at p < 0.01 and a gap over the
+re-run band or 10 points), which is what a reasoning base against an
+adapter trained on think-free targets does under one shared token budget:
+set `thinking=` the same on both arms.
+
+Two `local_model` knobs the situation writer cannot guess for you.
+`result_shapes={tool_name: example_result}` pins what a tool returns, so a
+policy branch that only exists for some results ("credits over $200 go to
+`escalate_to_human`") is reached on purpose instead of by luck. Ids, dates
+and people are re-drawn per call and a number moves by up to about a third
+of itself (`900.0` lands in roughly 600 to 1200), so pick a template value
+whose whole range sits on one side of the threshold and run the same pinned
+tasks under one shape per side. `fault_plans={message: {tool: {"mode":
+"timeout", "rate": 1.0}}}` replays a known fault schedule; `simulate()`
+writes these from `fault_rate=`, so pass your own only to replay one.
+`timeout=` is 300 s by default, enough for a served model that scaled to
+zero to answer its first request; when a call still times out the run says
+so in `data.warnings` with the fix.
 
 ```bash
 export OPENAI_API_KEY=...
@@ -527,6 +563,7 @@ Pass `spec=` if you have a local tools-and-system-prompt folder of your own: a d
 | `user_model` | `None` | Who plays the simulated user in follow-up turns. `None` is the agent's own model; an `openai:`/`vllm:` spec moves that job to another model |
 | `traces` | `None` | Graded traces of the deployed agent — a list of plain row dicts or a JSONL path. Aims the coverage grid at the behaviors those traces show and keeps the sources out of the generated rows. See [Close the loop](#close-the-loop-aim-the-budget-with-traces) |
 | `tasks` | `None` | Re-run a previous run's task set instead of drawing a new one: that run, its rows, or its JSONL path. k is **not** inherited — see [Same tasks, new prompt](#trust-the-numbers) |
+| `timeout` | `300` | Seconds per agent completion, for `local_model` and every model spec. A served model that scaled to zero takes two to three minutes to answer its first request, so a shorter value drops the first pass; a timed-out call is named in `data.warnings` with the fix |
 | `logprobs` | `False` | Ask the rollout model for the log-probability of every token it generates. Each agent turn's step gets `logprob` and `n_tokens`, the row gets the totals. `"tokens"` keeps the per-token list. Model backends only |
 | `sampling` | `None` | How your own callable agent samples, `{"temperature": 0.7, "max_tokens": 1024, "model": "my-model"}`, recorded on every row as given. A model backend records its own and ignores this |
 | `reproducible` | `False` | Same seed, same concurrency, same agent: same rows. Runs batch by batch, so uneven latency costs throughput. Needs the clock off. `concurrency: 1` always runs this way |
@@ -672,6 +709,22 @@ rows, report = wai.drop_leaky_rows(data.trajectories, prod)
 And the loop closes on itself: `evaluate(rollouts, judge).failed_traces()`
 hands the failures straight back to `simulate(traces=...)`.
 
+**What traces can and cannot aim at.** Traces reproduce situations: the
+tools, faults and world states the deployed agent met. A failure that has
+a world-visible trigger (a tool timed out and the agent did not say so, a
+stale record was presented as current) is reproduced. A failure that lives
+in how the reply is worded (an unsupported claim, an estimate not labelled
+as one, two questions where one was asked for) has no trigger in the world,
+so traces alone cannot aim at it: measured on a 12-rule grader, every rule
+with a tool-result trigger was reproduced and every rule about the reply's
+wording was not (#285). For those, put the grader in the loop:
+with `simulate(..., grader=judge)` a row the grader fails is re-rolled and
+its ask mutated like a tool fault, and `data.search["mutation_aims"]` says
+how many parents and mutated rows each aim (`world_fault`,
+`graded_failure`) produced. The grader is the switch; to grade beside the
+loop and still steer by tool faults alone, pass
+`advanced={"mutate_graded_failures": False}`.
+
 If your traces are already on the platform, `wai.cut(agent="my-agent")`
 does the whole cut in one line — see
 [Training data out of traces](#training-data-out-of-traces).
@@ -902,11 +955,21 @@ rows, report = wai.write_rubrics(rows, domain="refunds")  # per-prompt criteria
 scored = wai.run_judge(rows, wai.rubric_judge())  # a verdict per criterion; markers rubric:<item>
 clean, report = wai.decontaminate(
     train_rows, against=[eval_rows]
-)  # 8-gram overlap with the eval set
+)  # same task id, verbatim, or 8-gram overlap with the eval set
+clean, report = wai.decontaminate(
+    train_rows, against=[eval_rows], embedder=embed, similarity=0.85
+)  # plus a semantic pass: n_semantic counted apart, a flag means "reads alike"
 wai.style_markers(rows)  # no_boilerplate, no_hedging, no_apology, no_sycophancy, answered
 wai.style_report(rows)["warnings"]  # "reward pays for hedging (corr +0.41 ...)"
 wai.refusal_report(benign_rows)  # over-refusal rate with a Wilson interval
 wai.compare_runs(run_a, run_b)  # paired delta with a 95% interval
+wai.holdout_size(
+    0.05, base=0.6, k=4
+)  # tasks to prove a 5-point gain; n_tasks_concentrated beside it
+wai.holdout_size(
+    0.05, before=before, after=after
+)  # the paired sd measured off a previous eval, no model
+wai.holdout_size(0.05, task_std=0.38)  # or the sd read off a delta_report interval
 wai.delta_report(before, after, target="pass_at_1", must_not_regress=["honest_after_fault"])
 wai.delta_report(before, after, target="pass_at_1", by="category")  # the target per kind of prompt
 before = wai.simulate(agent, tools=TOOLS, tasks=base, runs=3)  # the same eval three times
@@ -923,7 +986,7 @@ wai.grounding_report(rows)  # grounded rate, and the invented values by tool and
 
 **Judge trust.** Label 50 to 100 rows by hand with `attach_labels` (0/1) -- `report["ok"]` means measured and clean, so with no labels it is `False` and the report says the judge is unmeasured rather than untrustworthy (`format_judge_trust` prints `NOT MEASURED`). The report gives agreement with a Wilson interval and Cohen's kappa, agreement on two task halves (tune the rubric on one, read the other), judge pass rate on short versus long replies within the same human label (length bias the humans rule out), and, with the judge callable, a re-judge of a sample as-is (consistency) and with neutral filler appended (a flip means the judge reads length). Disagreements come back as a review queue. `format_judge_trust(report)` prints it. `probes="all"` (or a list) tries the reward hacks a policy finds first on the judge on purpose: filler, the rubric's own words stuffed in, a claim of success with no evidence, the ask echoed back, a well-formed tool call with empty arguments, a sycophantic opener, a polite refusal. An additive probe is exploitable when failing replies start passing; a replacement probe when a reply with no content passes. `report["exploitable_by"]` names the holes at or over 10%, and a policy trained on this judge will find those same holes. Standalone: `wai.judge_probes(rows, judge, rubric=...)`. The gold set needs both passes and failures; with one class only the report says so and skips the kappa and length flags. With the hosted judge, call `wai.grade` once first (or `whileai.simulations.score.grade_llm.warm_judge`; it is not re-exported) so the cold start, two to three minutes, is not counted as timeouts.
 
-**Decontamination.** Word 8-gram overlap between a dataset's prompts and any evaluation source: row lists, JSONL paths, or platform dataset ids. A row is contaminated when it is an eval prompt verbatim or when one eval text covers at least 80% of its words (`overlap=`, the Llama 2 rule); one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question. Short prompts match verbatim only. `fields=("prompt", "final_text")` also checks replies against eval answers and references. The report separates verbatim hits from near copies and counts hits per field, and returns the clean rows with the first offenders.
+**Decontamination.** Four rules between a dataset's rows and any evaluation source (row lists, JSONL paths, or platform dataset ids), each counted on its own and a row counted once. A row is contaminated when it shares a `scenario_id` or `task_id` with an eval row (`n_same_task`: a task is a situation, not a string, so a rephrasing of an eval situation is the eval situation), when it is an eval prompt verbatim (`n_exact`), or when one eval text covers at least 80% of its words in shared 8-grams (`n_near`; `overlap=`, the Llama 2 rule; one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question, and short prompts match verbatim only). `fields=("prompt", "final_text")` also checks replies against eval answers and references. Word overlap does not see a paraphrase: a holdout written by re-running the generator was 70% within 0.85 cosine of the training batch, and the 8-gram rule flagged 4 of its 101 prompts where a semantic pass flagged 16. Pass `embedder=` (any callable from a list of texts to one vector per text, so nothing is imported; with sentence-transformers, `embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist()`) and rows whose prompt is within `similarity=` (0.85 cosine) of an eval prompt are flagged as `n_semantic`. That flag means the two prompts read alike, not that they are the same task: "cancel one reservation" and "cancel three reservations" for different customers score 0.93 with no shared answer. So the task-id rule decides first, the semantic pass only looks across different task ids, and `report["notes"]` says the flag is a question to check. The default stays lexical; the threshold was read off BGE (unrelated prompts score about 0.55 there) and needs picking for another model, so when the eval rows carry task ids the pass measures how alike distinct tasks read to your embedder (the 99th percentile of similarity over eval-prompt pairs with different task ids) and `notes` says it, and says when `similarity=` sits below it, since a threshold there flags tasks that merely share a domain. The report returns the clean rows with the first offenders, their coverage or similarity, and hits per field.
 
 **Intervals and comparison.** Every pass@1 carries a 95% interval from a bootstrap over tasks (`pass_at(rows).ci95`), and `metric_summary` / `marker_summary` do the same for markers. pass^k and pass@k carry their own (`pass_pow_k_ci95`, `pass_at_k_ci95`), a bootstrap over the k-eligible groups. Markers come from the judge: return `{"reward": ..., "markers": {"name": value}}` from a `grader=` or `run_judge` callable and they land on `row["markers"]`, which is what `marker_summary`, `delta_report` and `from_row` read. `compare_runs` pairs the tasks two runs share, bootstraps the paired difference, and adds a sign-flip permutation p-value; fewer than five shared tasks falls back to an unpaired test and says so. Tasks on one side only are dropped from a paired comparison; `note` says how many and `paired_share` is the fraction that paired, so a verdict over a quarter of the eval reads as one. The verdict `no_difference_detected` means the interval covers zero, not that the runs are equal. A task is a situation, not a string: every report (`pass_at`, `compare_runs`, `delta_report`, `eval_variance`, `curriculum`, `group_signal`, the exporters) groups rows by `wai.task_key(row)`, the engine's `scenario_id` when the row has one, so repeats and rephrasings of one situation count as one task and the same rows give the same task count everywhere. `pass_at(rows).config` and `delta_report(...)["config"]` say what the rows were produced with (temperature, reply budget, policy and judge versions), and `delta_report` warns when the two sides differ.
 
@@ -939,7 +1002,9 @@ rerun = wai.simulate(
 assert base.rollouts_per_request == rerun.rollouts_per_request  # cheap guard
 ```
 
-**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way.
+**Size before you run.** Most evals are too small to see the effects they produce. Per-prompt paired spread is stable for agent rubrics, about 0.38 across five lanes, and at that spread a 50-prompt eval only detects a 10-point gain; a 5-point gain needs about 220 prompts and a 3-point gain about 600. One lane read the same adapter as "barely helps" at 45 and 85 prompts (both straddled zero) and cleared at 131 (+0.13 [+0.07, +0.19]); the effect was real the whole time and a GPU round went to fixing a data problem that was a measurement problem. The effective sample is prompts, not rollouts: raising k sharpens each prompt's estimate but does not narrow a bootstrap over prompts, so spend eval budget on prompts first (k still matters for preference and grouped methods, which need mixed groups). And "straddles zero" means the eval cannot tell, not that the model did not improve; say which. `holdout_size(effect, base=, k=)` says how many paired tasks prove a gain at 80% power, and `detectable_effect(n_tasks, ...)` is the same solved for the gain. Its binomial model assumes the gain is spread evenly across tasks and the two arms are independent draws, and says so in `notes`; when a trait is only exercised by some prompts most tasks are ties, the paired differences spread far wider, and the model under-sizes by several times (a voice lane at 0 -> 0.127 needed 54 tasks where the model said 14), so the model path also returns `n_tasks_concentrated`. On a holdout whose tasks differ in difficulty the model errs the other way, asking for `1 / (1 - Var(p_i) / (p(1-p)))` times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4); `before=` alone reports the spread and that ratio. The honest paths measure: `before=before, after=after` (the same two row lists `delta_report` takes) reads the per-task paired sd off both arms of a previous eval on the same tasks, with the covariance pairing buys in it, and `task_std=` (the per-task sibling of `run_std`) takes the number you read off a `delta_report` (`(hi - lo) * sqrt(n_paired_tasks) / 3.92` from `target_ci95`). `push(purpose="holdout")` warns when the set is too small to prove a 5-point gain.
+
+**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way. Both sides should have the same rollouts per task; when a run lost some (`data.report()["rollouts_lost"]`, with the reasons in `rollouts_lost_by`) and one arm sits at k=4 while the other is at k=2, the report warns next to the sizing line and names both. Unequal k is a precision issue, not a bias: rows lost at random leave the paired delta unbiased and only widen its interval; rows lost for a reason (a timeout on the hard runs) bias it, and only re-running the short arm on its short tasks fixes that. `balance_rollouts=True` (off by default) trims every paired task to the rows both sides have (drawn by `seed=`) so pass^k and pass@k share one k; it costs precision, removes no bias, and `balanced` says how many rows each side gave up.
 
 **Run the eval three times.** One evaluation is a draw, not a number: the same model on the same tasks lands somewhere else next time, and most post-training gains are inside that spread (rlhf-book ch. 16, appendix C). `wai.simulate(agent, tasks=base, runs=3)` replays the task set three times in one call, same tasks, faults and world, and stamps `lineage.eval_run` on every row. Feed both sides to `delta_report` and it works out `run_std` from the repeats itself. The verdict words: `moved` is a change the interval and the re-run band both support; `moved_unreplicated` is a change seen once, which could be noise, and the warning tells you the `runs=3` call that settles it; `within_eval_noise` is a delta smaller than what re-running the eval does on its own, so equivalence, not a win; `no_change_detected` is an interval that covers zero. `ceiling=True` means the before run already passes most of its tasks (0.9 or more, or too few paired tasks left with room), so there is little improvement the eval could show; use harder situations before training again.
 
@@ -1151,6 +1216,8 @@ Measured at `avg_turns=4`. The default is now `12`, so a row carries more turns
 | `embedder` | `"hash"` | Prompt selection |
 | `seed` | `0` | Reproducible draws. Bit-for-bit at `concurrency: 1` or with `reproducible=True`, within a process and across processes; otherwise which rows land before the cap depends on thread timing |
 | `avg_turns` | `12` | Target conversation length in turns. The person speaks at most `avg_turns // 2` times; `12` leaves room to verify, look up, confirm, and write. |
+| `mutate_graded_failures` | on with `grader=` | `False` grades beside the loop without steering by the verdict: only tool faults make mutation parents. A row the grader fails (reward under 0.5) is otherwise re-rolled and its ask mutated the way a tool fault's is; `search["mutation_aims"]` counts each aim. `True` without `grader=` is an error |
+| `patience` | `"normal"` | How long the person keeps answering the agent's questions. The first question is always attempted; from the second on the person may walk away (`normal`: 35% then 60%; `short`: 60% then 90%; `endless`: never, the pre-knob behaviour). At any question the person may also leave when it asks for something they could not know. A row the person left ends on the agent's question and carries `ended_by="user_left"`; `search["ended_on_question"]` counts them. |
 
 Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollouts_per_request`; `unique=` → `unique_situations`; `policy=` → `system_prompt`; `risk=` → `fault_rate`.
 
@@ -1158,7 +1225,7 @@ Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollo
 
 Each row, in `data.trajectories` and on disk: `prompt`, `messages`, `steps`, `final_text`, `scenario_id`. Optional `world_state`, `faults`, `reward`, `reason`. `llm_grade=True` adds `llm_reward`. `wai.rank(path)` adds `quality` without changing `reward`. Every row also says how it was sampled: `sampling` is `{"temperature", "max_tokens", "model"}` as the model backend resolved them, or what you passed as `simulate(sampling=...)` for your own callable agent (`None` when you passed nothing, since only you know how it samples). `policy_version` names the model and the system prompt it ran under.
 
-Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
+Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. Every row also names the deploy prompt it was generated under: `lineage.system_prompt_sha` (the hash `policy_version` carries after `@`) , `lineage.system_prompt_head` (its first 120 chars) and `lineage.system_prompt_chars`, with the full text once per run in `data.system_prompts[<sha>]`, so a base rate measured under a full policy is never mistaken for one measured under a bare prompt; `delta_report` warns when its two arms differ on that hash. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
 
 What goes to disk is the whole row, not a summary of it: `data.rows` (the same list `output=` and `save()` write, callable as `data.rows()` too) carries everything the trajectory carries, so a saved run can still prove its own provenance. That includes how the situation was drawn (`scenario_dimensions`, `arm`, `selection_reason`, `behavior_signature`, `seed`), who graded it and how that went (`judge_name`, `judge_status`, `judge_meta`, `lineage`, `label_source`), and what was measured on it (`markers`, read by `marker_summary` and `delta_report`). Two things never ship, at any depth of the row: the teacher-only `privileged` block and its `principle` / `hidden_state` / `reference` / `rubric` fields, which would put the answer key one step from a training file, and `vector`, the raw embedding the diversity search keeps in memory for the length of the run. A privileged block nested inside a carried field or a tool result is dropped the same way, before `messages` is rebuilt from the steps.
 

{whileai-0.62 → whileai-0.64}/README.md

@@ -183,6 +183,18 @@ print(wai.pass_at(scored.rows), *scored.warnings)  # a hollow run says so here
 `scored.warnings` is new: no rollout called a tool, a declared tool no
 rollout touched, a marker that fired on no row. A 1.00 on a run like that is not a result; the note names the fix.
 
+A declared tool the world cannot answer is the quiet version of the same
+failure: with `execute=`, a tool that is in the schema but has no branch in
+your function fails exactly like a world fault, the agent reports the miss
+honestly, and a candour rubric rewards the row. Every run now records calls
+and successes per tool in `data.coverage["tools"]` (`n`, `ok`, `fault_n`, and
+`injected` for faults the run scheduled itself), lists the tools that never
+work in `data.coverage["dead_tools"]` (the Wilson 95% upper bound on the
+success rate is under 0.30, so 0 of 9 or 4 of 612 is dead and 0 of 3 or 2 of
+5 is not), adds `dead_tools` to `data.degraded`, and puts the names and the
+one fix that applies to your world in `data.warnings` and `data.report()`.
+Steps with no recorded result are not evidence and never accuse a tool.
+
 ### Bring a model
 
 Bring your own model. Any OpenAI-compatible chat endpoint that returns tool
@@ -191,7 +203,31 @@ your key. To put a number on a model you serve (`wai.serve`, or your own
 vLLM), make it the agent: `wai.simulate(tasks=pinned,
 agent=wai.local_model(endpoint, name, tools=TOOLS, system=POLICY,
 thinking=False))`, and run both arms of a before/after through that same
-call so the only difference is the weights.
+call so the only difference is the weights. `thinking=False` reaches the
+simulated user as well when the agent's own model plays it (the default),
+and whatever a user model still emits as `<think>` is stripped before it
+becomes a user turn; the run reports those under
+`data.search["user_think"]` (counts and shares of the user turns, zeros when
+none) and says so in `data.warnings`. `delta_report` fails a before/after
+whose arms differ in how often they answered at all (`"answered"` in
+`not_comparable`, a two-proportion test at p < 0.01 and a gap over the
+re-run band or 10 points), which is what a reasoning base against an
+adapter trained on think-free targets does under one shared token budget:
+set `thinking=` the same on both arms.
+
+Two `local_model` knobs the situation writer cannot guess for you.
+`result_shapes={tool_name: example_result}` pins what a tool returns, so a
+policy branch that only exists for some results ("credits over $200 go to
+`escalate_to_human`") is reached on purpose instead of by luck. Ids, dates
+and people are re-drawn per call and a number moves by up to about a third
+of itself (`900.0` lands in roughly 600 to 1200), so pick a template value
+whose whole range sits on one side of the threshold and run the same pinned
+tasks under one shape per side. `fault_plans={message: {tool: {"mode":
+"timeout", "rate": 1.0}}}` replays a known fault schedule; `simulate()`
+writes these from `fault_rate=`, so pass your own only to replay one.
+`timeout=` is 300 s by default, enough for a served model that scaled to
+zero to answer its first request; when a call still times out the run says
+so in `data.warnings` with the fix.
 
 ```bash
 export OPENAI_API_KEY=...
@@ -491,6 +527,7 @@ Pass `spec=` if you have a local tools-and-system-prompt folder of your own: a d
 | `user_model` | `None` | Who plays the simulated user in follow-up turns. `None` is the agent's own model; an `openai:`/`vllm:` spec moves that job to another model |
 | `traces` | `None` | Graded traces of the deployed agent — a list of plain row dicts or a JSONL path. Aims the coverage grid at the behaviors those traces show and keeps the sources out of the generated rows. See [Close the loop](#close-the-loop-aim-the-budget-with-traces) |
 | `tasks` | `None` | Re-run a previous run's task set instead of drawing a new one: that run, its rows, or its JSONL path. k is **not** inherited — see [Same tasks, new prompt](#trust-the-numbers) |
+| `timeout` | `300` | Seconds per agent completion, for `local_model` and every model spec. A served model that scaled to zero takes two to three minutes to answer its first request, so a shorter value drops the first pass; a timed-out call is named in `data.warnings` with the fix |
 | `logprobs` | `False` | Ask the rollout model for the log-probability of every token it generates. Each agent turn's step gets `logprob` and `n_tokens`, the row gets the totals. `"tokens"` keeps the per-token list. Model backends only |
 | `sampling` | `None` | How your own callable agent samples, `{"temperature": 0.7, "max_tokens": 1024, "model": "my-model"}`, recorded on every row as given. A model backend records its own and ignores this |
 | `reproducible` | `False` | Same seed, same concurrency, same agent: same rows. Runs batch by batch, so uneven latency costs throughput. Needs the clock off. `concurrency: 1` always runs this way |
@@ -636,6 +673,22 @@ rows, report = wai.drop_leaky_rows(data.trajectories, prod)
 And the loop closes on itself: `evaluate(rollouts, judge).failed_traces()`
 hands the failures straight back to `simulate(traces=...)`.
 
+**What traces can and cannot aim at.** Traces reproduce situations: the
+tools, faults and world states the deployed agent met. A failure that has
+a world-visible trigger (a tool timed out and the agent did not say so, a
+stale record was presented as current) is reproduced. A failure that lives
+in how the reply is worded (an unsupported claim, an estimate not labelled
+as one, two questions where one was asked for) has no trigger in the world,
+so traces alone cannot aim at it: measured on a 12-rule grader, every rule
+with a tool-result trigger was reproduced and every rule about the reply's
+wording was not (#285). For those, put the grader in the loop:
+with `simulate(..., grader=judge)` a row the grader fails is re-rolled and
+its ask mutated like a tool fault, and `data.search["mutation_aims"]` says
+how many parents and mutated rows each aim (`world_fault`,
+`graded_failure`) produced. The grader is the switch; to grade beside the
+loop and still steer by tool faults alone, pass
+`advanced={"mutate_graded_failures": False}`.
+
 If your traces are already on the platform, `wai.cut(agent="my-agent")`
 does the whole cut in one line — see
 [Training data out of traces](#training-data-out-of-traces).
@@ -866,11 +919,21 @@ rows, report = wai.write_rubrics(rows, domain="refunds")  # per-prompt criteria
 scored = wai.run_judge(rows, wai.rubric_judge())  # a verdict per criterion; markers rubric:<item>
 clean, report = wai.decontaminate(
     train_rows, against=[eval_rows]
-)  # 8-gram overlap with the eval set
+)  # same task id, verbatim, or 8-gram overlap with the eval set
+clean, report = wai.decontaminate(
+    train_rows, against=[eval_rows], embedder=embed, similarity=0.85
+)  # plus a semantic pass: n_semantic counted apart, a flag means "reads alike"
 wai.style_markers(rows)  # no_boilerplate, no_hedging, no_apology, no_sycophancy, answered
 wai.style_report(rows)["warnings"]  # "reward pays for hedging (corr +0.41 ...)"
 wai.refusal_report(benign_rows)  # over-refusal rate with a Wilson interval
 wai.compare_runs(run_a, run_b)  # paired delta with a 95% interval
+wai.holdout_size(
+    0.05, base=0.6, k=4
+)  # tasks to prove a 5-point gain; n_tasks_concentrated beside it
+wai.holdout_size(
+    0.05, before=before, after=after
+)  # the paired sd measured off a previous eval, no model
+wai.holdout_size(0.05, task_std=0.38)  # or the sd read off a delta_report interval
 wai.delta_report(before, after, target="pass_at_1", must_not_regress=["honest_after_fault"])
 wai.delta_report(before, after, target="pass_at_1", by="category")  # the target per kind of prompt
 before = wai.simulate(agent, tools=TOOLS, tasks=base, runs=3)  # the same eval three times
@@ -887,7 +950,7 @@ wai.grounding_report(rows)  # grounded rate, and the invented values by tool and
 
 **Judge trust.** Label 50 to 100 rows by hand with `attach_labels` (0/1) -- `report["ok"]` means measured and clean, so with no labels it is `False` and the report says the judge is unmeasured rather than untrustworthy (`format_judge_trust` prints `NOT MEASURED`). The report gives agreement with a Wilson interval and Cohen's kappa, agreement on two task halves (tune the rubric on one, read the other), judge pass rate on short versus long replies within the same human label (length bias the humans rule out), and, with the judge callable, a re-judge of a sample as-is (consistency) and with neutral filler appended (a flip means the judge reads length). Disagreements come back as a review queue. `format_judge_trust(report)` prints it. `probes="all"` (or a list) tries the reward hacks a policy finds first on the judge on purpose: filler, the rubric's own words stuffed in, a claim of success with no evidence, the ask echoed back, a well-formed tool call with empty arguments, a sycophantic opener, a polite refusal. An additive probe is exploitable when failing replies start passing; a replacement probe when a reply with no content passes. `report["exploitable_by"]` names the holes at or over 10%, and a policy trained on this judge will find those same holes. Standalone: `wai.judge_probes(rows, judge, rubric=...)`. The gold set needs both passes and failures; with one class only the report says so and skips the kappa and length flags. With the hosted judge, call `wai.grade` once first (or `whileai.simulations.score.grade_llm.warm_judge`; it is not re-exported) so the cold start, two to three minutes, is not counted as timeouts.
 
-**Decontamination.** Word 8-gram overlap between a dataset's prompts and any evaluation source: row lists, JSONL paths, or platform dataset ids. A row is contaminated when it is an eval prompt verbatim or when one eval text covers at least 80% of its words (`overlap=`, the Llama 2 rule); one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question. Short prompts match verbatim only. `fields=("prompt", "final_text")` also checks replies against eval answers and references. The report separates verbatim hits from near copies and counts hits per field, and returns the clean rows with the first offenders.
+**Decontamination.** Four rules between a dataset's rows and any evaluation source (row lists, JSONL paths, or platform dataset ids), each counted on its own and a row counted once. A row is contaminated when it shares a `scenario_id` or `task_id` with an eval row (`n_same_task`: a task is a situation, not a string, so a rephrasing of an eval situation is the eval situation), when it is an eval prompt verbatim (`n_exact`), or when one eval text covers at least 80% of its words in shared 8-grams (`n_near`; `overlap=`, the Llama 2 rule; one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question, and short prompts match verbatim only). `fields=("prompt", "final_text")` also checks replies against eval answers and references. Word overlap does not see a paraphrase: a holdout written by re-running the generator was 70% within 0.85 cosine of the training batch, and the 8-gram rule flagged 4 of its 101 prompts where a semantic pass flagged 16. Pass `embedder=` (any callable from a list of texts to one vector per text, so nothing is imported; with sentence-transformers, `embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist()`) and rows whose prompt is within `similarity=` (0.85 cosine) of an eval prompt are flagged as `n_semantic`. That flag means the two prompts read alike, not that they are the same task: "cancel one reservation" and "cancel three reservations" for different customers score 0.93 with no shared answer. So the task-id rule decides first, the semantic pass only looks across different task ids, and `report["notes"]` says the flag is a question to check. The default stays lexical; the threshold was read off BGE (unrelated prompts score about 0.55 there) and needs picking for another model, so when the eval rows carry task ids the pass measures how alike distinct tasks read to your embedder (the 99th percentile of similarity over eval-prompt pairs with different task ids) and `notes` says it, and says when `similarity=` sits below it, since a threshold there flags tasks that merely share a domain. The report returns the clean rows with the first offenders, their coverage or similarity, and hits per field.
 
 **Intervals and comparison.** Every pass@1 carries a 95% interval from a bootstrap over tasks (`pass_at(rows).ci95`), and `metric_summary` / `marker_summary` do the same for markers. pass^k and pass@k carry their own (`pass_pow_k_ci95`, `pass_at_k_ci95`), a bootstrap over the k-eligible groups. Markers come from the judge: return `{"reward": ..., "markers": {"name": value}}` from a `grader=` or `run_judge` callable and they land on `row["markers"]`, which is what `marker_summary`, `delta_report` and `from_row` read. `compare_runs` pairs the tasks two runs share, bootstraps the paired difference, and adds a sign-flip permutation p-value; fewer than five shared tasks falls back to an unpaired test and says so. Tasks on one side only are dropped from a paired comparison; `note` says how many and `paired_share` is the fraction that paired, so a verdict over a quarter of the eval reads as one. The verdict `no_difference_detected` means the interval covers zero, not that the runs are equal. A task is a situation, not a string: every report (`pass_at`, `compare_runs`, `delta_report`, `eval_variance`, `curriculum`, `group_signal`, the exporters) groups rows by `wai.task_key(row)`, the engine's `scenario_id` when the row has one, so repeats and rephrasings of one situation count as one task and the same rows give the same task count everywhere. `pass_at(rows).config` and `delta_report(...)["config"]` say what the rows were produced with (temperature, reply budget, policy and judge versions), and `delta_report` warns when the two sides differ.
 
@@ -903,7 +966,9 @@ rerun = wai.simulate(
 assert base.rollouts_per_request == rerun.rollouts_per_request  # cheap guard
 ```
 
-**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way.
+**Size before you run.** Most evals are too small to see the effects they produce. Per-prompt paired spread is stable for agent rubrics, about 0.38 across five lanes, and at that spread a 50-prompt eval only detects a 10-point gain; a 5-point gain needs about 220 prompts and a 3-point gain about 600. One lane read the same adapter as "barely helps" at 45 and 85 prompts (both straddled zero) and cleared at 131 (+0.13 [+0.07, +0.19]); the effect was real the whole time and a GPU round went to fixing a data problem that was a measurement problem. The effective sample is prompts, not rollouts: raising k sharpens each prompt's estimate but does not narrow a bootstrap over prompts, so spend eval budget on prompts first (k still matters for preference and grouped methods, which need mixed groups). And "straddles zero" means the eval cannot tell, not that the model did not improve; say which. `holdout_size(effect, base=, k=)` says how many paired tasks prove a gain at 80% power, and `detectable_effect(n_tasks, ...)` is the same solved for the gain. Its binomial model assumes the gain is spread evenly across tasks and the two arms are independent draws, and says so in `notes`; when a trait is only exercised by some prompts most tasks are ties, the paired differences spread far wider, and the model under-sizes by several times (a voice lane at 0 -> 0.127 needed 54 tasks where the model said 14), so the model path also returns `n_tasks_concentrated`. On a holdout whose tasks differ in difficulty the model errs the other way, asking for `1 / (1 - Var(p_i) / (p(1-p)))` times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4); `before=` alone reports the spread and that ratio. The honest paths measure: `before=before, after=after` (the same two row lists `delta_report` takes) reads the per-task paired sd off both arms of a previous eval on the same tasks, with the covariance pairing buys in it, and `task_std=` (the per-task sibling of `run_std`) takes the number you read off a `delta_report` (`(hi - lo) * sqrt(n_paired_tasks) / 3.92` from `target_ci95`). `push(purpose="holdout")` warns when the set is too small to prove a 5-point gain.
+
+**Before and after.** `delta_report` runs `compare_runs` on pass@1 and every marker both row sets share. `target=` names the metric the training was meant to move and gives the headline; `must_not_regress=` names the behaviors whose significant drop fails the report; any other significant drop is a warning. `format_delta_report(report)` prints one line per metric. `eval_variance(run_1, run_2, run_3)` is the eval's own re-run standard deviation (three or more evaluations of the same model); passing it as `run_std=` makes any delta inside twice that band `within_noise`, and a target there reads `within_eval_noise` rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). `by=` names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries `groups`, the target compared within each group, and `groups_down` for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way. Both sides should have the same rollouts per task; when a run lost some (`data.report()["rollouts_lost"]`, with the reasons in `rollouts_lost_by`) and one arm sits at k=4 while the other is at k=2, the report warns next to the sizing line and names both. Unequal k is a precision issue, not a bias: rows lost at random leave the paired delta unbiased and only widen its interval; rows lost for a reason (a timeout on the hard runs) bias it, and only re-running the short arm on its short tasks fixes that. `balance_rollouts=True` (off by default) trims every paired task to the rows both sides have (drawn by `seed=`) so pass^k and pass@k share one k; it costs precision, removes no bias, and `balanced` says how many rows each side gave up.
 
 **Run the eval three times.** One evaluation is a draw, not a number: the same model on the same tasks lands somewhere else next time, and most post-training gains are inside that spread (rlhf-book ch. 16, appendix C). `wai.simulate(agent, tasks=base, runs=3)` replays the task set three times in one call, same tasks, faults and world, and stamps `lineage.eval_run` on every row. Feed both sides to `delta_report` and it works out `run_std` from the repeats itself. The verdict words: `moved` is a change the interval and the re-run band both support; `moved_unreplicated` is a change seen once, which could be noise, and the warning tells you the `runs=3` call that settles it; `within_eval_noise` is a delta smaller than what re-running the eval does on its own, so equivalence, not a win; `no_change_detected` is an interval that covers zero. `ceiling=True` means the before run already passes most of its tasks (0.9 or more, or too few paired tasks left with room), so there is little improvement the eval could show; use harder situations before training again.
 
@@ -1115,6 +1180,8 @@ Measured at `avg_turns=4`. The default is now `12`, so a row carries more turns
 | `embedder` | `"hash"` | Prompt selection |
 | `seed` | `0` | Reproducible draws. Bit-for-bit at `concurrency: 1` or with `reproducible=True`, within a process and across processes; otherwise which rows land before the cap depends on thread timing |
 | `avg_turns` | `12` | Target conversation length in turns. The person speaks at most `avg_turns // 2` times; `12` leaves room to verify, look up, confirm, and write. |
+| `mutate_graded_failures` | on with `grader=` | `False` grades beside the loop without steering by the verdict: only tool faults make mutation parents. A row the grader fails (reward under 0.5) is otherwise re-rolled and its ask mutated the way a tool fault's is; `search["mutation_aims"]` counts each aim. `True` without `grader=` is an error |
+| `patience` | `"normal"` | How long the person keeps answering the agent's questions. The first question is always attempted; from the second on the person may walk away (`normal`: 35% then 60%; `short`: 60% then 90%; `endless`: never, the pre-knob behaviour). At any question the person may also leave when it asks for something they could not know. A row the person left ends on the agent's question and carries `ended_by="user_left"`; `search["ended_on_question"]` counts them. |
 
 Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollouts_per_request`; `unique=` → `unique_situations`; `policy=` → `system_prompt`; `risk=` → `fault_rate`.
 
@@ -1122,7 +1189,7 @@ Aliases: `phrasings=` / `n=` → `requests_per_situation`; `repeats=` → `rollo
 
 Each row, in `data.trajectories` and on disk: `prompt`, `messages`, `steps`, `final_text`, `scenario_id`. Optional `world_state`, `faults`, `reward`, `reason`. `llm_grade=True` adds `llm_reward`. `wai.rank(path)` adds `quality` without changing `reward`. Every row also says how it was sampled: `sampling` is `{"temperature", "max_tokens", "model"}` as the model backend resolved them, or what you passed as `simulate(sampling=...)` for your own callable agent (`None` when you passed nothing, since only you know how it samples). `policy_version` names the model and the system prompt it ran under.
 
-Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
+Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row now says who did which job, next to `model_version` for the agent: `writer_model` (the writer's model, or `template`, `seed`, `pinned` when no model wrote the prompt), `user_model` (absent when the agent took a single message), and `judge_meta.model` once graded; `data.metadata` and the `.meta.json` sidecar carry the same three. Every row also names the deploy prompt it was generated under: `lineage.system_prompt_sha` (the hash `policy_version` carries after `@`) , `lineage.system_prompt_head` (its first 120 chars) and `lineage.system_prompt_chars`, with the full text once per run in `data.system_prompts[<sha>]`, so a base rate measured under a full policy is never mistaken for one measured under a bare prompt; `delta_report` warns when its two arms differ on that hash. When the agent model also wrote the situations or played the user, `data.degraded` holds `same_model` and `data.warnings` says so in one sentence, with the fix: pass `simulator=` for the writer and `user_model=` for the user to put those jobs on a different model.
 
 What goes to disk is the whole row, not a summary of it: `data.rows` (the same list `output=` and `save()` write, callable as `data.rows()` too) carries everything the trajectory carries, so a saved run can still prove its own provenance. That includes how the situation was drawn (`scenario_dimensions`, `arm`, `selection_reason`, `behavior_signature`, `seed`), who graded it and how that went (`judge_name`, `judge_status`, `judge_meta`, `lineage`, `label_source`), and what was measured on it (`markers`, read by `marker_summary` and `delta_report`). Two things never ship, at any depth of the row: the teacher-only `privileged` block and its `principle` / `hidden_state` / `reference` / `rubric` fields, which would put the answer key one step from a training file, and `vector`, the raw embedding the diversity search keeps in memory for the length of the run. A privileged block nested inside a carried field or a tool result is dropped the same way, before `messages` is rebuilt from the steps.
 

{whileai-0.62 → whileai-0.64}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "whileai"
-version = "0.62"
+version = "0.64"
 description = "While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt."
 readme = "README.md"
 license = {text = "Apache-2.0"}

{whileai-0.62 → whileai-0.64}/whileai/simulations/__init__.py

@@ -80,11 +80,14 @@ from .ingest.traces import (
     drop_leaky_rows,
     flaw_rows,
     format_trace_report,
+    infer_harness,
     leakage_report,
     load_traces,
     mine_traces,
+    opening_share,
     simulate_from_traces,
     split_pseudo_production,
+    tools_from_traces,
     trace_report,
 )
 from .monitor import HackMonitor, format_hack_monitor
@@ -175,6 +178,7 @@ from .score.stats import (
     detectable_effect,
     eval_variance,
     holdout_size,
+    marker_names,
     marker_summary,
     metric_summary,
     task_key,
@@ -310,6 +314,7 @@ __all__ = [
     "holdout_size",
     "hosted_model",
     "import_hf",
+    "infer_harness",
     "inspect",
     "issue_delegated_credential",
     "judge_agreement",
@@ -329,6 +334,7 @@ __all__ = [
     "loss_mask",
     "mark_grounding",
     "mark_rows",
+    "marker_names",
     "marker_summary",
     "mean_kl",
     "metric_summary",
@@ -339,6 +345,7 @@ __all__ = [
     "normalize_judge_result",
     "novelty",
     "open_ended_probes",
+    "opening_share",
     "optimize",
     "outcome_check",
     "pairwise_judge",
@@ -393,6 +400,7 @@ __all__ = [
     "to_row",
     "to_trl",
     "tool_call_roundtrip",
+    "tools_from_traces",
     "trace_flag_report",
     "trace_flags",
     "trace_markers",

{whileai-0.62 → whileai-0.64}/whileai/simulations/data.py

@@ -366,6 +366,10 @@ class SimulationData:
     # ``model_version`` on every row.
     writer_model: str = ""
     user_model: str | None = None
+    # The deploy prompt the rows were generated under, full text keyed by
+    # the hash every row carries in ``lineage.system_prompt_sha`` (#296).
+    # Those two are the hash's only homes: the row and this text.
+    system_prompts: dict[str, str] = field(default_factory=dict)
 
     @property
     def judge_model(self) -> str | None:
@@ -837,6 +841,10 @@ class SimulationData:
                         # The agent spec: a trainer loading this JSONL later
                         # needs the policy and tool schemas the run knew.
                         "system_prompt": str(getattr(self.profile, "policy", "") or ""),
+                        # the exact text the rows were generated under
+                        # (policy plus scaffold), keyed by the hash each
+                        # row carries in lineage.system_prompt_sha
+                        "system_prompts": dict(self.system_prompts),
                         "tools": list(getattr(self.profile, "tools", None) or []),
                         "stopped_because": self.stopped_because,
                         "coverage": self.coverage,

{whileai-0.62 → whileai-0.64}/whileai/simulations/export.py

@@ -13,6 +13,16 @@ target model emits thinking tokens. ``training_rows`` closes that gap:
 * strips ``<think>`` blocks from assistant turns, so a thinking rollout
   model never teaches a non-thinking student to emit them.
 
+The strip cuts the other way too. A reasoning base (Qwen3) trained on
+think-free targets learns to print an empty ``<think></think>`` and
+answer at once. That is fine until the eval, where that adapter meets
+the untrained base under one shared ``max_tokens``: the base reasons and
+runs out of budget, the adapter answers, and the delta is a win over
+replies the base never produced (#297). ``delta_report`` fails that
+comparison; the fix is ``thinking=`` set the same on both arms, or
+``strip_think=False`` when the student is a reasoning model and should
+keep reasoning.
+
 Two wire shapes come out of here, and they are not the same shape:
 
 ``format="openai"`` (the default)
@@ -411,6 +421,14 @@ def training_rows(
     its policy trains an agent that never saw its rules. ``mask_mode``
     picks which assistant turns carry loss (see ``loss_mask``).
 
+    ``strip_think=True`` (the default) removes ``<think>`` blocks from the
+    assistant turns. On a reasoning base such as Qwen3 that teaches the
+    adapter to emit an empty ``<think></think>`` and answer at once, so
+    at eval it answers while the untrained base is still reasoning under
+    the same ``max_tokens`` (#297). Pass ``strip_think=False`` when the
+    student should keep reasoning, and set ``thinking=`` the same on both
+    arms of the eval either way.
+
     ``unroll=True`` turns an N-turn conversation into N samples, the
     k-th ending at the k-th assistant turn with loss on that turn only
     (rlhf-book ch. 4, multi-turn masking). Every earlier agent turn then

{whileai-0.62 → whileai-0.64}/whileai/simulations/generate/adapters.py

@@ -343,6 +343,7 @@ def resolve(
     max_turns: int = 5,
     avg_turns: float = 6,
     min_user_turns: int = 1,
+    patience: str = "normal",
     turn_stats: dict | None = None,
     opening_rate: float = 0.0,
     temperature: float | None = None,
@@ -378,6 +379,7 @@ def resolve(
             fault_plans=fault_plans,
             avg_turns=avg_turns,
             min_user_turns=min_user_turns,
+            patience=patience,
             opening_rate=opening_rate,
             execute=execute,
             result_shapes=result_shapes,