whileai 0.62__tar.gz → 0.63__tar.gz

{whileai-0.62/whileai.egg-info → whileai-0.63}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.62
+Version: 0.63
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -219,6 +219,18 @@ print(wai.pass_at(scored.rows), *scored.warnings)  # a hollow run says so here
 `scored.warnings` is new: no rollout called a tool, a declared tool no
 rollout touched, a marker that fired on no row. A 1.00 on a run like that is not a result; the note names the fix.
 
+A declared tool the world cannot answer is the quiet version of the same
+failure: with `execute=`, a tool that is in the schema but has no branch in
+your function fails exactly like a world fault, the agent reports the miss
+honestly, and a candour rubric rewards the row. Every run now records calls
+and successes per tool in `data.coverage["tools"]` (`n`, `ok`, `fault_n`, and
+`injected` for faults the run scheduled itself), lists the tools that never
+work in `data.coverage["dead_tools"]` (the Wilson 95% upper bound on the
+success rate is under 0.30, so 0 of 9 or 4 of 612 is dead and 0 of 3 or 2 of
+5 is not), adds `dead_tools` to `data.degraded`, and puts the names and the
+one fix that applies to your world in `data.warnings` and `data.report()`.
+Steps with no recorded result are not evidence and never accuse a tool.
+
 ### Bring a model
 
 Bring your own model. Any OpenAI-compatible chat endpoint that returns tool
@@ -227,7 +239,31 @@ your key. To put a number on a model you serve (`wai.serve`, or your own
 vLLM), make it the agent: `wai.simulate(tasks=pinned,
 agent=wai.local_model(endpoint, name, tools=TOOLS, system=POLICY,
 thinking=False))`, and run both arms of a before/after through that same
-call so the only difference is the weights.
+call so the only difference is the weights. `thinking=False` reaches the
+simulated user as well when the agent's own model plays it (the default),
+and whatever a user model still emits as `<think>` is stripped before it
+becomes a user turn; the run reports those under
+`data.search["user_think"]` (counts and shares of the user turns, zeros when
+none) and says so in `data.warnings`. `delta_report` fails a before/after
+whose arms differ in how often they answered at all (`"answered"` in
+`not_comparable`, a two-proportion test at p < 0.01 and a gap over the
+re-run band or 10 points), which is what a reasoning base against an
+adapter trained on think-free targets does under one shared token budget:
+set `thinking=` the same on both arms.
+
+Two `local_model` knobs the situation writer cannot guess for you.
+`result_shapes={tool_name: example_result}` pins what a tool returns, so a
+policy branch that only exists for some results ("credits over $200 go to
+`escalate_to_human`") is reached on purpose instead of by luck. Ids, dates
+and people are re-drawn per call and a number moves by up to about a third
+of itself (`900.0` lands in roughly 600 to 1200), so pick a template value
+whose whole range sits on one side of the threshold and run the same pinned
+tasks under one shape per side. `fault_plans={message: {tool: {"mode":
+"timeout", "rate": 1.0}}}` replays a known fault schedule; `simulate()`
+writes these from `fault_rate=`, so pass your own only to replay one.
+`timeout=` is 300 s by default, enough for a served model that scaled to
+zero to answer its first request; when a call still times out the run says
+so in `data.warnings` with the fix.
 
 ```bash
 export OPENAI_API_KEY=...
@@ -527,6 +563,7 @@ Pass `spec=` if you have a local tools-and-system-prompt folder of your own: a d
 | `user_model` | `None` | Who plays the simulated user in follow-up turns. `None` is the agent's own model; an `openai:`/`vllm:` spec moves that job to another model |
 | `traces` | `None` | Graded traces of the deployed agent — a list of plain row dicts or a JSONL path. Aims the coverage grid at the behaviors those traces show and keeps the sources out of the generated rows. See [Close the loop](#close-the-loop-aim-the-budget-with-traces) |
 | `tasks` | `None` | Re-run a previous run's task set instead of drawing a new one: that run, its rows, or its JSONL path. k is **not** inherited — see [Same tasks, new prompt](#trust-the-numbers) |
+| `timeout` | `300` | Seconds per agent completion, for `local_model` and every model spec. A served model that scaled to zero takes two to three minutes to answer its first request, so a shorter value drops the first pass; a timed-out call is named in `data.warnings` with the fix |
 | `logprobs` | `False` | Ask the rollout model for the log-probability of every token it generates. Each agent turn's step gets `logprob` and `n_tokens`, the row gets the totals. `"tokens"` keeps the per-token list. Model backends only |
 | `sampling` | `None` | How your own callable agent samples, `{"temperature": 0.7, "max_tokens": 1024, "model": "my-model"}`, recorded on every row as given. A model backend records its own and ignores this |
 | `reproducible` | `False` | Same seed, same concurrency, same agent: same rows. Runs batch by batch, so uneven latency costs throughput. Needs the clock off. `concurrency: 1` always runs this way |
@@ -902,7 +939,10 @@ rows, report = wai.write_rubrics(rows, domain="refunds")  # per-prompt criteria
 scored = wai.run_judge(rows, wai.rubric_judge())  # a verdict per criterion; markers rubric:<item>
 clean, report = wai.decontaminate(
     train_rows, against=[eval_rows]
-)  # 8-gram overlap with the eval set
+)  # same task id, verbatim, or 8-gram overlap with the eval set
+clean, report = wai.decontaminate(
+    train_rows, against=[eval_rows], embedder=embed, similarity=0.85
+)  # plus a semantic pass: n_semantic counted apart, a flag means "reads alike"
 wai.style_markers(rows)  # no_boilerplate, no_hedging, no_apology, no_sycophancy, answered
 wai.style_report(rows)["warnings"]  # "reward pays for hedging (corr +0.41 ...)"
 wai.refusal_report(benign_rows)  # over-refusal rate with a Wilson interval
@@ -923,7 +963,7 @@ wai.grounding_report(rows)  # grounded rate, and the invented values by tool and
 
 **Judge trust.** Label 50 to 100 rows by hand with `attach_labels` (0/1) -- `report["ok"]` means measured and clean, so with no labels it is `False` and the report says the judge is unmeasured rather than untrustworthy (`format_judge_trust` prints `NOT MEASURED`). The report gives agreement with a Wilson interval and Cohen's kappa, agreement on two task halves (tune the rubric on one, read the other), judge pass rate on short versus long replies within the same human label (length bias the humans rule out), and, with the judge callable, a re-judge of a sample as-is (consistency) and with neutral filler appended (a flip means the judge reads length). Disagreements come back as a review queue. `format_judge_trust(report)` prints it. `probes="all"` (or a list) tries the reward hacks a policy finds first on the judge on purpose: filler, the rubric's own words stuffed in, a claim of success with no evidence, the ask echoed back, a well-formed tool call with empty arguments, a sycophantic opener, a polite refusal. An additive probe is exploitable when failing replies start passing; a replacement probe when a reply with no content passes. `report["exploitable_by"]` names the holes at or over 10%, and a policy trained on this judge will find those same holes. Standalone: `wai.judge_probes(rows, judge, rubric=...)`. The gold set needs both passes and failures; with one class only the report says so and skips the kappa and length flags. With the hosted judge, call `wai.grade` once first (or `whileai.simulations.score.grade_llm.warm_judge`; it is not re-exported) so the cold start, two to three minutes, is not counted as timeouts.
 
-**Decontamination.** Word 8-gram overlap between a dataset's prompts and any evaluation source: row lists, JSONL paths, or platform dataset ids. A row is contaminated when it is an eval prompt verbatim or when one eval text covers at least 80% of its words (`overlap=`, the Llama 2 rule); one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question. Short prompts match verbatim only. `fields=("prompt", "final_text")` also checks replies against eval answers and references. The report separates verbatim hits from near copies and counts hits per field, and returns the clean rows with the first offenders.
+**Decontamination.** Four rules between a dataset's rows and any evaluation source (row lists, JSONL paths, or platform dataset ids), each counted on its own and a row counted once. A row is contaminated when it shares a `scenario_id` or `task_id` with an eval row (`n_same_task`: a task is a situation, not a string, so a rephrasing of an eval situation is the eval situation), when it is an eval prompt verbatim (`n_exact`), or when one eval text covers at least 80% of its words in shared 8-grams (`n_near`; `overlap=`, the Llama 2 rule; one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question, and short prompts match verbatim only). `fields=("prompt", "final_text")` also checks replies against eval answers and references. Word overlap does not see a paraphrase: a holdout written by re-running the generator was 70% within 0.85 cosine of the training batch, and the 8-gram rule flagged 4 of its 101 prompts where a semantic pass flagged 16. Pass `embedder=` (any callable from a list of texts to one vector per text, so nothing is imported; with sentence-transformers, `embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist()`) and rows whose prompt is within `similarity=` (0.85 cosine) of an eval prompt are flagged as `n_semantic`. That flag means the two prompts read alike, not that they are the same task: "cancel one reservation" and "cancel three reservations" for different customers score 0.93 with no shared answer. So the task-id rule decides first, the semantic pass only looks across different task ids, and `report["notes"]` says the flag is a question to check. The default stays lexical; the threshold was read off BGE (unrelated prompts score about 0.55 there) and needs picking for another model, so when the eval rows carry task ids the pass measures how alike distinct tasks read to your embedder (the 99th percentile of similarity over eval-prompt pairs with different task ids) and `notes` says it, and says when `similarity=` sits below it, since a threshold there flags tasks that merely share a domain. The report returns the clean rows with the first offenders, their coverage or similarity, and hits per field.
 
 **Intervals and comparison.** Every pass@1 carries a 95% interval from a bootstrap over tasks (`pass_at(rows).ci95`), and `metric_summary` / `marker_summary` do the same for markers. pass^k and pass@k carry their own (`pass_pow_k_ci95`, `pass_at_k_ci95`), a bootstrap over the k-eligible groups. Markers come from the judge: return `{"reward": ..., "markers": {"name": value}}` from a `grader=` or `run_judge` callable and they land on `row["markers"]`, which is what `marker_summary`, `delta_report` and `from_row` read. `compare_runs` pairs the tasks two runs share, bootstraps the paired difference, and adds a sign-flip permutation p-value; fewer than five shared tasks falls back to an unpaired test and says so. Tasks on one side only are dropped from a paired comparison; `note` says how many and `paired_share` is the fraction that paired, so a verdict over a quarter of the eval reads as one. The verdict `no_difference_detected` means the interval covers zero, not that the runs are equal. A task is a situation, not a string: every report (`pass_at`, `compare_runs`, `delta_report`, `eval_variance`, `curriculum`, `group_signal`, the exporters) groups rows by `wai.task_key(row)`, the engine's `scenario_id` when the row has one, so repeats and rephrasings of one situation count as one task and the same rows give the same task count everywhere. `pass_at(rows).config` and `delta_report(...)["config"]` say what the rows were produced with (temperature, reply budget, policy and judge versions), and `delta_report` warns when the two sides differ.
 

{whileai-0.62 → whileai-0.63}/README.md

@@ -183,6 +183,18 @@ print(wai.pass_at(scored.rows), *scored.warnings)  # a hollow run says so here
 `scored.warnings` is new: no rollout called a tool, a declared tool no
 rollout touched, a marker that fired on no row. A 1.00 on a run like that is not a result; the note names the fix.
 
+A declared tool the world cannot answer is the quiet version of the same
+failure: with `execute=`, a tool that is in the schema but has no branch in
+your function fails exactly like a world fault, the agent reports the miss
+honestly, and a candour rubric rewards the row. Every run now records calls
+and successes per tool in `data.coverage["tools"]` (`n`, `ok`, `fault_n`, and
+`injected` for faults the run scheduled itself), lists the tools that never
+work in `data.coverage["dead_tools"]` (the Wilson 95% upper bound on the
+success rate is under 0.30, so 0 of 9 or 4 of 612 is dead and 0 of 3 or 2 of
+5 is not), adds `dead_tools` to `data.degraded`, and puts the names and the
+one fix that applies to your world in `data.warnings` and `data.report()`.
+Steps with no recorded result are not evidence and never accuse a tool.
+
 ### Bring a model
 
 Bring your own model. Any OpenAI-compatible chat endpoint that returns tool
@@ -191,7 +203,31 @@ your key. To put a number on a model you serve (`wai.serve`, or your own
 vLLM), make it the agent: `wai.simulate(tasks=pinned,
 agent=wai.local_model(endpoint, name, tools=TOOLS, system=POLICY,
 thinking=False))`, and run both arms of a before/after through that same
-call so the only difference is the weights.
+call so the only difference is the weights. `thinking=False` reaches the
+simulated user as well when the agent's own model plays it (the default),
+and whatever a user model still emits as `<think>` is stripped before it
+becomes a user turn; the run reports those under
+`data.search["user_think"]` (counts and shares of the user turns, zeros when
+none) and says so in `data.warnings`. `delta_report` fails a before/after
+whose arms differ in how often they answered at all (`"answered"` in
+`not_comparable`, a two-proportion test at p < 0.01 and a gap over the
+re-run band or 10 points), which is what a reasoning base against an
+adapter trained on think-free targets does under one shared token budget:
+set `thinking=` the same on both arms.
+
+Two `local_model` knobs the situation writer cannot guess for you.
+`result_shapes={tool_name: example_result}` pins what a tool returns, so a
+policy branch that only exists for some results ("credits over $200 go to
+`escalate_to_human`") is reached on purpose instead of by luck. Ids, dates
+and people are re-drawn per call and a number moves by up to about a third
+of itself (`900.0` lands in roughly 600 to 1200), so pick a template value
+whose whole range sits on one side of the threshold and run the same pinned
+tasks under one shape per side. `fault_plans={message: {tool: {"mode":
+"timeout", "rate": 1.0}}}` replays a known fault schedule; `simulate()`
+writes these from `fault_rate=`, so pass your own only to replay one.
+`timeout=` is 300 s by default, enough for a served model that scaled to
+zero to answer its first request; when a call still times out the run says
+so in `data.warnings` with the fix.
 
 ```bash
 export OPENAI_API_KEY=...
@@ -491,6 +527,7 @@ Pass `spec=` if you have a local tools-and-system-prompt folder of your own: a d
 | `user_model` | `None` | Who plays the simulated user in follow-up turns. `None` is the agent's own model; an `openai:`/`vllm:` spec moves that job to another model |
 | `traces` | `None` | Graded traces of the deployed agent — a list of plain row dicts or a JSONL path. Aims the coverage grid at the behaviors those traces show and keeps the sources out of the generated rows. See [Close the loop](#close-the-loop-aim-the-budget-with-traces) |
 | `tasks` | `None` | Re-run a previous run's task set instead of drawing a new one: that run, its rows, or its JSONL path. k is **not** inherited — see [Same tasks, new prompt](#trust-the-numbers) |
+| `timeout` | `300` | Seconds per agent completion, for `local_model` and every model spec. A served model that scaled to zero takes two to three minutes to answer its first request, so a shorter value drops the first pass; a timed-out call is named in `data.warnings` with the fix |
 | `logprobs` | `False` | Ask the rollout model for the log-probability of every token it generates. Each agent turn's step gets `logprob` and `n_tokens`, the row gets the totals. `"tokens"` keeps the per-token list. Model backends only |
 | `sampling` | `None` | How your own callable agent samples, `{"temperature": 0.7, "max_tokens": 1024, "model": "my-model"}`, recorded on every row as given. A model backend records its own and ignores this |
 | `reproducible` | `False` | Same seed, same concurrency, same agent: same rows. Runs batch by batch, so uneven latency costs throughput. Needs the clock off. `concurrency: 1` always runs this way |
@@ -866,7 +903,10 @@ rows, report = wai.write_rubrics(rows, domain="refunds")  # per-prompt criteria
 scored = wai.run_judge(rows, wai.rubric_judge())  # a verdict per criterion; markers rubric:<item>
 clean, report = wai.decontaminate(
     train_rows, against=[eval_rows]
-)  # 8-gram overlap with the eval set
+)  # same task id, verbatim, or 8-gram overlap with the eval set
+clean, report = wai.decontaminate(
+    train_rows, against=[eval_rows], embedder=embed, similarity=0.85
+)  # plus a semantic pass: n_semantic counted apart, a flag means "reads alike"
 wai.style_markers(rows)  # no_boilerplate, no_hedging, no_apology, no_sycophancy, answered
 wai.style_report(rows)["warnings"]  # "reward pays for hedging (corr +0.41 ...)"
 wai.refusal_report(benign_rows)  # over-refusal rate with a Wilson interval
@@ -887,7 +927,7 @@ wai.grounding_report(rows)  # grounded rate, and the invented values by tool and
 
 **Judge trust.** Label 50 to 100 rows by hand with `attach_labels` (0/1) -- `report["ok"]` means measured and clean, so with no labels it is `False` and the report says the judge is unmeasured rather than untrustworthy (`format_judge_trust` prints `NOT MEASURED`). The report gives agreement with a Wilson interval and Cohen's kappa, agreement on two task halves (tune the rubric on one, read the other), judge pass rate on short versus long replies within the same human label (length bias the humans rule out), and, with the judge callable, a re-judge of a sample as-is (consistency) and with neutral filler appended (a flip means the judge reads length). Disagreements come back as a review queue. `format_judge_trust(report)` prints it. `probes="all"` (or a list) tries the reward hacks a policy finds first on the judge on purpose: filler, the rubric's own words stuffed in, a claim of success with no evidence, the ask echoed back, a well-formed tool call with empty arguments, a sycophantic opener, a polite refusal. An additive probe is exploitable when failing replies start passing; a replacement probe when a reply with no content passes. `report["exploitable_by"]` names the holes at or over 10%, and a policy trained on this judge will find those same holes. Standalone: `wai.judge_probes(rows, judge, rubric=...)`. The gold set needs both passes and failures; with one class only the report says so and skips the kappa and length flags. With the hosted judge, call `wai.grade` once first (or `whileai.simulations.score.grade_llm.warm_judge`; it is not re-exported) so the cold start, two to three minutes, is not counted as timeouts.
 
-**Decontamination.** Word 8-gram overlap between a dataset's prompts and any evaluation source: row lists, JSONL paths, or platform dataset ids. A row is contaminated when it is an eval prompt verbatim or when one eval text covers at least 80% of its words (`overlap=`, the Llama 2 rule); one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question. Short prompts match verbatim only. `fields=("prompt", "final_text")` also checks replies against eval answers and references. The report separates verbatim hits from near copies and counts hits per field, and returns the clean rows with the first offenders.
+**Decontamination.** Four rules between a dataset's rows and any evaluation source (row lists, JSONL paths, or platform dataset ids), each counted on its own and a row counted once. A row is contaminated when it shares a `scenario_id` or `task_id` with an eval row (`n_same_task`: a task is a situation, not a string, so a rephrasing of an eval situation is the eval situation), when it is an eval prompt verbatim (`n_exact`), or when one eval text covers at least 80% of its words in shared 8-grams (`n_near`; `overlap=`, the Llama 2 rule; one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question, and short prompts match verbatim only). `fields=("prompt", "final_text")` also checks replies against eval answers and references. Word overlap does not see a paraphrase: a holdout written by re-running the generator was 70% within 0.85 cosine of the training batch, and the 8-gram rule flagged 4 of its 101 prompts where a semantic pass flagged 16. Pass `embedder=` (any callable from a list of texts to one vector per text, so nothing is imported; with sentence-transformers, `embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist()`) and rows whose prompt is within `similarity=` (0.85 cosine) of an eval prompt are flagged as `n_semantic`. That flag means the two prompts read alike, not that they are the same task: "cancel one reservation" and "cancel three reservations" for different customers score 0.93 with no shared answer. So the task-id rule decides first, the semantic pass only looks across different task ids, and `report["notes"]` says the flag is a question to check. The default stays lexical; the threshold was read off BGE (unrelated prompts score about 0.55 there) and needs picking for another model, so when the eval rows carry task ids the pass measures how alike distinct tasks read to your embedder (the 99th percentile of similarity over eval-prompt pairs with different task ids) and `notes` says it, and says when `similarity=` sits below it, since a threshold there flags tasks that merely share a domain. The report returns the clean rows with the first offenders, their coverage or similarity, and hits per field.
 
 **Intervals and comparison.** Every pass@1 carries a 95% interval from a bootstrap over tasks (`pass_at(rows).ci95`), and `metric_summary` / `marker_summary` do the same for markers. pass^k and pass@k carry their own (`pass_pow_k_ci95`, `pass_at_k_ci95`), a bootstrap over the k-eligible groups. Markers come from the judge: return `{"reward": ..., "markers": {"name": value}}` from a `grader=` or `run_judge` callable and they land on `row["markers"]`, which is what `marker_summary`, `delta_report` and `from_row` read. `compare_runs` pairs the tasks two runs share, bootstraps the paired difference, and adds a sign-flip permutation p-value; fewer than five shared tasks falls back to an unpaired test and says so. Tasks on one side only are dropped from a paired comparison; `note` says how many and `paired_share` is the fraction that paired, so a verdict over a quarter of the eval reads as one. The verdict `no_difference_detected` means the interval covers zero, not that the runs are equal. A task is a situation, not a string: every report (`pass_at`, `compare_runs`, `delta_report`, `eval_variance`, `curriculum`, `group_signal`, the exporters) groups rows by `wai.task_key(row)`, the engine's `scenario_id` when the row has one, so repeats and rephrasings of one situation count as one task and the same rows give the same task count everywhere. `pass_at(rows).config` and `delta_report(...)["config"]` say what the rows were produced with (temperature, reply budget, policy and judge versions), and `delta_report` warns when the two sides differ.
 

{whileai-0.62 → whileai-0.63}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "whileai"
-version = "0.62"
+version = "0.63"
 description = "While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt."
 readme = "README.md"
 license = {text = "Apache-2.0"}

{whileai-0.62 → whileai-0.63}/whileai/simulations/__init__.py

@@ -80,11 +80,14 @@ from .ingest.traces import (
     drop_leaky_rows,
     flaw_rows,
     format_trace_report,
+    infer_harness,
     leakage_report,
     load_traces,
     mine_traces,
+    opening_share,
     simulate_from_traces,
     split_pseudo_production,
+    tools_from_traces,
     trace_report,
 )
 from .monitor import HackMonitor, format_hack_monitor
@@ -175,6 +178,7 @@ from .score.stats import (
     detectable_effect,
     eval_variance,
     holdout_size,
+    marker_names,
     marker_summary,
     metric_summary,
     task_key,
@@ -310,6 +314,7 @@ __all__ = [
     "holdout_size",
     "hosted_model",
     "import_hf",
+    "infer_harness",
     "inspect",
     "issue_delegated_credential",
     "judge_agreement",
@@ -329,6 +334,7 @@ __all__ = [
     "loss_mask",
     "mark_grounding",
     "mark_rows",
+    "marker_names",
     "marker_summary",
     "mean_kl",
     "metric_summary",
@@ -339,6 +345,7 @@ __all__ = [
     "normalize_judge_result",
     "novelty",
     "open_ended_probes",
+    "opening_share",
     "optimize",
     "outcome_check",
     "pairwise_judge",
@@ -393,6 +400,7 @@ __all__ = [
     "to_row",
     "to_trl",
     "tool_call_roundtrip",
+    "tools_from_traces",
     "trace_flag_report",
     "trace_flags",
     "trace_markers",

{whileai-0.62 → whileai-0.63}/whileai/simulations/export.py

@@ -13,6 +13,16 @@ target model emits thinking tokens. ``training_rows`` closes that gap:
 * strips ``<think>`` blocks from assistant turns, so a thinking rollout
   model never teaches a non-thinking student to emit them.
 
+The strip cuts the other way too. A reasoning base (Qwen3) trained on
+think-free targets learns to print an empty ``<think></think>`` and
+answer at once. That is fine until the eval, where that adapter meets
+the untrained base under one shared ``max_tokens``: the base reasons and
+runs out of budget, the adapter answers, and the delta is a win over
+replies the base never produced (#297). ``delta_report`` fails that
+comparison; the fix is ``thinking=`` set the same on both arms, or
+``strip_think=False`` when the student is a reasoning model and should
+keep reasoning.
+
 Two wire shapes come out of here, and they are not the same shape:
 
 ``format="openai"`` (the default)
@@ -411,6 +421,14 @@ def training_rows(
     its policy trains an agent that never saw its rules. ``mask_mode``
     picks which assistant turns carry loss (see ``loss_mask``).
 
+    ``strip_think=True`` (the default) removes ``<think>`` blocks from the
+    assistant turns. On a reasoning base such as Qwen3 that teaches the
+    adapter to emit an empty ``<think></think>`` and answer at once, so
+    at eval it answers while the untrained base is still reasoning under
+    the same ``max_tokens`` (#297). Pass ``strip_think=False`` when the
+    student should keep reasoning, and set ``thinking=`` the same on both
+    arms of the eval either way.
+
     ``unroll=True`` turns an N-turn conversation into N samples, the
     k-th ending at the k-th assistant turn with loss on that turn only
     (rlhf-book ch. 4, multi-turn masking). Every earlier agent turn then

{whileai-0.62 → whileai-0.63}/whileai/simulations/generate/agents.py

@@ -17,6 +17,7 @@ from urllib.parse import urlparse
 from whileai._env import getenv
 from whileai.auth import SIGN_IN_URL
 
+from ..text import split_reasoning
 from ..world.sandbox import MockEnvironment
 from .anthropic_backend import ANTHROPIC_BASE_URL, is_anthropic_url
 from .anthropic_backend import DEFAULT_MODEL as ANTHROPIC_DEFAULT_MODEL
@@ -780,8 +781,13 @@ def _strip_tool_markup(text: str) -> str:
     return cleaned.strip()
 
 
-_THINK_BLOCK = re.compile(r"<think>.*?</think>\s*", re.S | re.I)
-_THINK_OPEN = re.compile(r"<think>.*\Z", re.S | re.I)
+#: A simulated-user turn that carried reasoning and left fewer spoken
+#: characters than this was reasoning with no spoken line: the writer
+#: thought and never typed, so the turn is retried, never emitted as a
+#: fragment. The floor applies only to turns that carried reasoning; a
+#: bare ``yes`` or ``order 4821`` with no ``<think>`` is a real user turn
+#: and passes untouched (#284).
+_MIN_SPOKEN = 25
 
 
 def _strip_think(text: str) -> str:
@@ -789,8 +795,23 @@ def _strip_think(text: str) -> str:
     an unclosed ``<think>`` (the token cap landed inside it) goes to the
     end. What is left is the reply, which is what a grader, a marker and
     the next turn's history should see (#264)."""
-    text = _THINK_BLOCK.sub("", text)
-    return _THINK_OPEN.sub("", text)
+    return split_reasoning(text)[0]
+
+
+def _note_user_turn(turn_stats: dict | None, closed: int, unclosed: bool) -> None:
+    """One simulated-user reply came back: count the turn, and count it
+    again as stripped when it carried reasoning, and as unclosed when the
+    reasoning was cut off. The run reports counts and shares per arm
+    (#284: 18 unclosed on one arm, 0 on the other)."""
+    if not turn_stats:
+        return
+    lock = turn_stats.get("lock")
+    with lock if lock is not None else contextlib.nullcontext():
+        turn_stats["user_turns"] = turn_stats.get("user_turns", 0) + 1
+        if closed or unclosed:
+            turn_stats["user_think_stripped"] = turn_stats.get("user_think_stripped", 0) + 1
+        if unclosed:
+            turn_stats["user_think_unclosed"] = turn_stats.get("user_think_unclosed", 0) + 1
 
 
 def _spoken_text(reply: dict) -> str:
@@ -897,9 +918,21 @@ def _want_followup(
 
     A question or refusal earns an answer while the turn budget has
     room: the depth cap is ``budget // 2`` user turns (avg_turns=12
-    allows 6, avg_turns=4 allows 2). After a completed action about
-    half of people react or ask the next thing, when the thread has
-    budget for it. Short-budget threads still end on the agent.
+    allows 6, avg_turns=4 allows 2). Otherwise the thread continues with
+    probability ``1 - 1/cap``, which makes the mean depth track the cap.
+    A budget under 4 still ends on the agent.
+
+    Before this, any reply that was not a question, a refusal, or a
+    recognised success ended the thread, and a completed action only
+    continued on a fixed coin flip. "Your reservation has been cancelled"
+    matches none of those, so threads died at one user turn and the mean
+    sat near 1.5 whatever ``avg_turns`` said: measured 1.54 at avg_turns=6
+    and 1.52 at avg_turns=10 on the same spec. That silently caps every
+    behaviour that needs three turns to happen at all. Confirm-before-
+    acting is the clearest case: the user asks, the agent names the action
+    and asks, the user says yes, the agent acts. At 1.5 user turns most
+    rollouts never reach the write, so the rule is never exercised and the
+    training set cannot demonstrate it.
     """
     cap = max(2, int(budget) // 2)
     if int(user_turns) >= cap:
@@ -909,10 +942,15 @@ def _want_followup(
         return True
     if _AGENT_REFUSAL.search(text):
         return True
-    if int(budget) >= 4 and _AGENT_SUCCESS.search(text):
-        digest = hashlib.sha256(f"{message}:{turn_i}:react".encode()).hexdigest()
-        return int(digest[:8], 16) % 2 == 0
-    return False
+    if int(budget) < 4:
+        # A short thread still ends on the agent: with room for one user line
+        # there is nothing a second one could be for.
+        return False
+    # Geometric with p = 1 - 1/cap: a thread of cap turns in expectation,
+    # deterministic in the message and turn so a seeded run reproduces.
+    digest = hashlib.sha256(f"{message}:{turn_i}:react".encode()).hexdigest()
+    draw = int(digest[:8], 16) / float(1 << 32)
+    return draw < (1.0 - 1.0 / float(cap))
 
 
 # Follow-up user turns only. Opener temperature lives on the writer (0.45–1.05).
@@ -1052,11 +1090,6 @@ _CONFIRM_ONLY = re.compile(
     re.I,
 )
 _AGENT_QUESTION = re.compile(r"\?\s*$|\b(which|what|who|where|can you|could you)\b", re.I)
-_AGENT_SUCCESS = re.compile(
-    r"\b(done|created|opened|fixed|all set|i (have|'ve|just)|"
-    r"successfully|completed)\b",
-    re.I,
-)
 _AGENT_REFUSAL = re.compile(
     r"\b(can't|cannot|won't|unable|not allowed|against (the )?(policy|rules?))\b",
     re.I,
@@ -1220,7 +1253,16 @@ def _user_followup(
     tools: list | None = None,
     force: bool = False,
     persona_tags: dict | None = None,
+    extra: Mapping[str, Any] | None = None,
+    turn_stats: dict | None = None,
 ) -> str:
+    """The simulated user's next line, or ``""`` when the writer produced
+    none worth keeping. ``extra`` carries the agent's request fields
+    (``thinking=False`` on ``local_model``) so the user model is asked not
+    to reason either; whatever it still emits as ``<think>`` is stripped
+    before the words become a user turn, an unclosed block is dropped
+    whole, and a turn that was reasoning with no spoken line is retried
+    rather than emitted as a fragment (#284)."""
     from .generator import _realize_typed_message, _strip_directive_phrases, clean_user_message
 
     trace = _render_user_trace(messages, steps)
@@ -1275,10 +1317,14 @@ def _user_followup(
     # Floor, not ceiling: a 5s wait on a busy endpoint silently killed
     # every follow-up and collapsed whole datasets to single-turn.
     wait = max(8.0 if asked else 5.0, min(30.0, float(timeout or 30) / 2))
-    attempts = (
+    attempts = list(
         (body, retry_body, retry_body) if force else ((body, retry_body) if asked else (body,))
     )
-    for attempt, content in enumerate(attempts):
+    retried_for_reasoning = False
+    attempt = 0
+    while attempt < len(attempts):
+        content = attempts[attempt]
+        attempt += 1
         try:
             reply = complete(
                 base_url,
@@ -1290,15 +1336,23 @@ def _user_followup(
                 tools=None,
                 api_key=api_key,
                 temperature=(_RESPONSE_TEMP_LO + _RESPONSE_TEMP_HI) / 2,
-                max_tokens=120 if attempt else 180,
+                max_tokens=120 if attempt > 1 else 180,
                 timeout=wait,
+                extra=extra,
             )
         except Exception:
             continue
+        spoken, closed, unclosed = split_reasoning(reply.get("content") or "")
+        _note_user_turn(turn_stats, closed, unclosed)
+        if (closed or unclosed) and len(spoken.strip()) < _MIN_SPOKEN:
+            # Reasoning and no spoken line. One more try with the short
+            # prompt; never a fragment, never empty user speech.
+            if not retried_for_reasoning:
+                retried_for_reasoning = True
+                attempts.append(retry_body)
+            continue
         text = _scrub_ai_traces(
-            _realize_typed_message(
-                _strip_directive_phrases(clean_user_message(reply.get("content") or "")), tags
-            )
+            _realize_typed_message(_strip_directive_phrases(clean_user_message(spoken)), tags)
         )
         if _accept_followup(text, prior, agent_text) and not _repeats_user_history(text, messages):
             return text
@@ -1350,6 +1404,8 @@ def _human_answer(
     api_key: str | None,
     timeout: float,
     stance: str = "",
+    extra: Mapping[str, Any] | None = None,
+    turn_stats: dict | None = None,
 ) -> str:
     """The simulated user answers the agent's question, in character.
 
@@ -1389,9 +1445,20 @@ def _human_answer(
         },
     ]
     reply = complete(
-        base_url, model, msgs, api_key=api_key, temperature=0.9, timeout=timeout, max_tokens=120
+        base_url,
+        model,
+        msgs,
+        api_key=api_key,
+        temperature=0.9,
+        timeout=timeout,
+        max_tokens=120,
+        extra=extra,
     )
-    return (_spoken_text(reply) or "").strip()
+    spoken, closed, unclosed = split_reasoning(reply.get("content") or "")
+    _note_user_turn(turn_stats, closed, unclosed)
+    if (closed or unclosed) and len(spoken.strip()) < _MIN_SPOKEN:
+        return ""
+    return _strip_tool_markup(spoken).strip()
 
 
 def _answer_tool_call(env: Any, execute: Callable | None, tool: str, arguments: dict) -> dict:
@@ -1421,6 +1488,12 @@ def _answer_tool_call(env: Any, execute: Callable | None, tool: str, arguments:
 #: says otherwise. Recorded on every row under ``sampling`` (rlhf-book ch. 9:
 #: rejection sampling is run at 0.7 to 1.0; the row has to say what it was).
 LOCAL_MODEL_TEMPERATURE = 0.8
+# Seconds one completion may take. A served model that scaled to zero
+# takes two to three minutes to answer its first request (113 s measured
+# on the account's own endpoint, #302; the hosted judge is the same
+# shape), and the old 60 s dropped every rollout of the first pass and
+# returned an empty run that looked finished.
+LOCAL_MODEL_TIMEOUT = 300.0
 
 
 def reply_budget(max_tokens: int | None = None) -> int:
@@ -1451,7 +1524,7 @@ def local_model(
     opening_rate: float = 0.0,
     human_tools: set | None = None,
     execute: Callable | None = None,
-    timeout: float = 60,
+    timeout: float = LOCAL_MODEL_TIMEOUT,
     max_tokens: int | None = None,
     user_model: str | None = None,
     thinking: bool | None = None,
@@ -1463,8 +1536,50 @@ def local_model(
     ``chat_template_kwargs={"enable_thinking": False}`` so the reply is
     the answer, not the reasoning, the way the hosted Qwen path already
     does; ``True`` asks for it; ``None`` (the default) sends nothing and
-    leaves the server's default. Either way ``<think>`` markup never
-    reaches ``step["text"]`` or ``final_text``.
+    leaves the server's default. The same field goes to the simulated
+    user when the agent's own model plays it (the default) or
+    ``user_model`` sits on the same endpoint, so the customer is asked
+    not to reason either; a ``user_model`` on another endpoint keeps
+    that server's default. Either way ``<think>`` markup never reaches
+    ``step["text"]``, ``final_text``, or a user turn (``step["user"]`` and
+    the ``messages`` history): what the user model still emits as
+    reasoning is stripped before it becomes speech, and a turn that was
+    reasoning with no spoken line is retried, then dropped (#284). The run
+    reports those under ``search["user_think"]``: ``user_turns``,
+    ``stripped`` and ``unclosed`` as counts, ``stripped_share`` and
+    ``unclosed_share`` as shares of the user turns, zeros when none.
+
+    ``result_shapes`` pins what a tool returns: ``{tool_name: example
+    result dict}``. The sandbox fills the example on every call instead
+    of inventing a record, so a policy branch that only exists for some
+    tool results (a credit over $200 must be escalated) is reached on
+    purpose rather than by luck. Field names and free text stay as
+    written; ids, dates and people are re-drawn per call, and a number
+    moves by up to about a third of itself (``900.0`` lands in roughly
+    600 to 1200, ``90.0`` in 60 to 120), so pick a template value whose
+    whole range sits on the side of the threshold you want. An argument
+    that shares a key with the template is echoed back (``invoice_id``
+    in, same ``invoice_id`` out). To measure a branch, run the same
+    pinned tasks under two shapes, one per side of the rule. Without it
+    the situation writer drafts an example per tool (``write_result_shapes``)
+    and the branch is exercised at random.
+
+    ``fault_plans`` schedules faults per ask: ``{message: {tool_name:
+    {"mode": "timeout", "rate": 1.0}}}``, keyed by the exact user message,
+    with ``mode`` one of ``timeout``, ``malformed``, ``stale`` or
+    ``permission_denied`` and ``rate`` the chance the fault fires on a
+    call. The plan may also carry ``world_state``, ``stance``, ``tone``
+    and ``texture``, which are popped off and shape the world and the
+    simulated user for that ask. ``simulate()`` writes these itself from
+    ``fault_rate=``; pass your own only to replay a known plan (``tasks=``
+    does this for you).
+
+    ``timeout`` is seconds per completion, ``LOCAL_MODEL_TIMEOUT`` (300)
+    by default: a served model that scaled to zero takes two to three
+    minutes to answer its first request, and a timeout under that drops
+    every rollout of the first pass. When a call still times out the run
+    says so in ``data.warnings`` with the fix (raise ``timeout=``, or send
+    one throwaway request first so the endpoint is warm).
     """
     local = threading.local()
     plans = fault_plans if fault_plans is not None else {}
@@ -1479,6 +1594,9 @@ def local_model(
         user_key: str | None = None
     else:
         user_url, user_name, user_key = base_url, model, api_key
+    # thinking= reaches the user simulator on the agent's own endpoint.
+    # Another server has its own template fields, so it keeps its default.
+    user_extras = extras if str(user_url).rstrip("/") == str(base_url).rstrip("/") else None
     shapes = result_shapes if result_shapes is not None else {}
     cap = default_max_turns(n_tools=len(tools)) if max_turns is None else max(1, int(max_turns))
     min_users = max(1, min(int(min_user_turns), max(1, cap // 2)))
@@ -1590,6 +1708,8 @@ def local_model(
                             api_key=user_key,
                             timeout=timeout,
                             stance=stance,
+                            extra=user_extras,
+                            turn_stats=turn_stats,
                         )
                         # an empty answer used to default to "go ahead", which
                         # silently taught the agent that asking always clears
@@ -1682,6 +1802,8 @@ def local_model(
                     persona_tags=persona_tags,
                     tools=tools,
                     force=force_followup,
+                    extra=user_extras,
+                    turn_stats=turn_stats,
                 )
                 if follow:
                     last_user = follow