whileai 0.53__tar.gz → 0.55__tar.gz

{whileai-0.53/whileai.egg-info → whileai-0.55}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.53
+Version: 0.55
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -51,7 +51,7 @@ Two ways in, one engine. Give it the agent's tools and system prompt and it samp
 
 ![How a row gets made: the draw, the coverage grid, the search arms, the rollout, the split](docs/how-a-row-gets-made.svg)
 
-A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The interactive version, running on real rows, is at [zeroproofai.com/docs/engine](https://zeroproofai.com/docs/engine).
+A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The engine on one page, with references: [docs/engine.md](docs/engine.md), also at [while.ai/docs/engine](https://while.ai/docs/engine).
 
 ## Overview
 
@@ -230,7 +230,7 @@ data.grade(rubric=RUBRIC)  # 2 grade against the task rubric: reward 0/1 on ever
 print(data.pass_at)
 wai.judge_trust(data.trajectories)  # 3 trust the numbers
 rows, report = wai.optimize(data, mode="rl")  # 4 prune to what carries gradient
-entry = wai.push_rows(rows, "github-rl-v1", gate=True, mode="rl")  # 5 publish, gated
+entry = wai.push_rows(rows, "my-agent-rl-v1", gate=True, mode="rl")  # 5 publish, gated
 ```
 
 `situations=200, repeats=8` is a guess. `wai.recommend(tools=TOOLS, system_prompt=POLICY, mode="rl")` replaces it with numbers from this agent's own grid: [How much to run](#how-much-to-run).
@@ -391,9 +391,9 @@ installable `verifiers` package, the shape Prime Intellect and TRL read.
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY, mode="rl", repeats=8)
 data.grade()
 # reward and world must import by name in the trainer: a module-level function or "module:attr"
-wai.export_environment(data, "envs/github-agent", reward=my_verifier)
-# pip install -e envs/github-agent
-# vf-eval github_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
+wai.export_environment(data, "envs/my-agent", reward=my_verifier)
+# pip install -e envs/my-agent
+# vf-eval my_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
 ```
 
 The package holds `spec.json` (system prompt, the tool schemas verbatim,
@@ -681,10 +681,10 @@ login`.
 # export WHILEAI_DELEGATED_CREDENTIAL=credential["credential"]
 
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY)
-v1 = data.push("github-explore-v1")  # -> {"datasetId": "ds_...", ...}
+v1 = data.push("my-agent-explore-v1")  # -> {"datasetId": "ds_...", ...}
 
 # iterate, then push the next version with lineage
-v2 = data.push("github-explore-v2", parent=v1["datasetId"])
+v2 = data.push("my-agent-explore-v2", parent=v1["datasetId"])
 
 wai.datasets()  # list yours + storage used
 rows = wai.pull(v1["datasetId"])  # rows, or pass path= for a file

{whileai-0.53 → whileai-0.55}/README.md

@@ -15,7 +15,7 @@ Two ways in, one engine. Give it the agent's tools and system prompt and it samp
 
 ![How a row gets made: the draw, the coverage grid, the search arms, the rollout, the split](docs/how-a-row-gets-made.svg)
 
-A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The interactive version, running on real rows, is at [zeroproofai.com/docs/engine](https://zeroproofai.com/docs/engine).
+A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The engine on one page, with references: [docs/engine.md](docs/engine.md), also at [while.ai/docs/engine](https://while.ai/docs/engine).
 
 ## Overview
 
@@ -194,7 +194,7 @@ data.grade(rubric=RUBRIC)  # 2 grade against the task rubric: reward 0/1 on ever
 print(data.pass_at)
 wai.judge_trust(data.trajectories)  # 3 trust the numbers
 rows, report = wai.optimize(data, mode="rl")  # 4 prune to what carries gradient
-entry = wai.push_rows(rows, "github-rl-v1", gate=True, mode="rl")  # 5 publish, gated
+entry = wai.push_rows(rows, "my-agent-rl-v1", gate=True, mode="rl")  # 5 publish, gated
 ```
 
 `situations=200, repeats=8` is a guess. `wai.recommend(tools=TOOLS, system_prompt=POLICY, mode="rl")` replaces it with numbers from this agent's own grid: [How much to run](#how-much-to-run).
@@ -355,9 +355,9 @@ installable `verifiers` package, the shape Prime Intellect and TRL read.
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY, mode="rl", repeats=8)
 data.grade()
 # reward and world must import by name in the trainer: a module-level function or "module:attr"
-wai.export_environment(data, "envs/github-agent", reward=my_verifier)
-# pip install -e envs/github-agent
-# vf-eval github_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
+wai.export_environment(data, "envs/my-agent", reward=my_verifier)
+# pip install -e envs/my-agent
+# vf-eval my_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
 ```
 
 The package holds `spec.json` (system prompt, the tool schemas verbatim,
@@ -645,10 +645,10 @@ login`.
 # export WHILEAI_DELEGATED_CREDENTIAL=credential["credential"]
 
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY)
-v1 = data.push("github-explore-v1")  # -> {"datasetId": "ds_...", ...}
+v1 = data.push("my-agent-explore-v1")  # -> {"datasetId": "ds_...", ...}
 
 # iterate, then push the next version with lineage
-v2 = data.push("github-explore-v2", parent=v1["datasetId"])
+v2 = data.push("my-agent-explore-v2", parent=v1["datasetId"])
 
 wai.datasets()  # list yours + storage used
 rows = wai.pull(v1["datasetId"])  # rows, or pass path= for a file

{whileai-0.53 → whileai-0.55}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "whileai"
-version = "0.53"
+version = "0.55"
 description = "While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt."
 readme = "README.md"
 license = {text = "Apache-2.0"}

{whileai-0.53 → whileai-0.55}/whileai/simulations/__init__.py

@@ -7,12 +7,13 @@ malicious), and rolls the agent. Grade 0/1 later. Optimize for post-training.
 Five calls, spec to gated dataset:
 
     import whileai.simulations as wai
-    data = wai.simulate(agent="openai:gpt-4.1-mini", spec="specs/github",
-                        mode="rl", situations=200, repeats=8)      # generate
+    data = wai.simulate(agent="openai:gpt-4.1-mini", tools=my_tools,
+                        system_prompt=my_policy, mode="rl",
+                        situations=200, repeats=8)                  # generate
     scored = data.grade(judge=my_judge)                             # grade 0/1
     print(scored.pass_at); wai.judge_trust(scored.rows, judge=my_judge)  # trust
     rows, report = wai.optimize(scored, mode="rl")                  # prune
-    wai.push_rows(rows, "github-rl-v1", gate=True, mode="rl")       # publish, gated
+    wai.push_rows(rows, "my-agent-rl-v1", gate=True, mode="rl")     # publish, gated
 
 Everything else exported here is one layer down from those five.
 """
@@ -174,6 +175,7 @@ from .training import (
     TrainingRun,
     attach_delta,
     attach_holdout,
+    delete_model,
     delete_run,
     get_run,
     list_runs,
@@ -182,6 +184,7 @@ from .training import (
     serve,
     train,
     training_run,
+    unserve,
 )
 from .verify import Verifier, verifier
 from .world.sandbox import MockEnvironment
@@ -252,6 +255,7 @@ __all__ = [
     "dedupe_groups",
     "delete_dataset",
     "delete_empty_datasets",
+    "delete_model",
     "delete_run",
     "delta_report",
     "dimensions_from_traces",
@@ -381,6 +385,7 @@ __all__ = [
     "trim_unanimous_groups",
     "ungrounded_arguments",
     "unpublish",
+    "unserve",
     "update_dataset",
     "validate",
     "verifier",

{whileai-0.53 → whileai-0.55}/whileai/simulations/data.py

@@ -652,6 +652,14 @@ class SimulationData:
         report["selection"] = self.search.get("selection")
         return report
 
+    def leak_report(self, *, min_len: int = 12) -> dict[str, Any]:
+        """Did any reply quote its own ``privileged`` block? Reads the
+        trajectories, which still carry the block; ``rows()`` is scrubbed
+        and would check nothing. Same report as ``leak_report``."""
+        from .score.privileged import leak_report
+
+        return leak_report(self.trajectories, min_len=min_len)
+
     @property
     def rows(self) -> RowList:
         """The exported rows: exactly what ``save()`` and ``output=`` write.

{whileai-0.53 → whileai-0.55}/whileai/simulations/environment.py

@@ -10,13 +10,13 @@ reward). ``export_environment`` writes those three as an installable
 ``verifiers`` package, the shape Prime Intellect and TRL consume::
 
     import whileai.simulations as wai
-    data = wai.simulate(spec="specs/github", mode="rl", repeats=8)
+    data = wai.simulate(tools=my_tools, system_prompt=my_policy, mode="rl", repeats=8)
     scored = data.grade()
-    wai.export_environment(scored, "envs/github-agent", reward=my_verifier)
+    wai.export_environment(scored, "envs/my-agent", reward=my_verifier)
 
     # then, with verifiers installed:
-    #   pip install -e envs/github-agent
-    #   vf-eval github_agent -a '{"split": "holdout"}' -m <policy> ...
+    #   pip install -e envs/my-agent
+    #   vf-eval my_agent -a '{"split": "holdout"}' -m <policy> ...
 
 What goes in the package:
 

{whileai-0.53 → whileai-0.55}/whileai/simulations/export.py

@@ -50,6 +50,7 @@ from pathlib import Path
 from typing import Any
 
 from .schema import check, stamp
+from .score.privileged import leak_report
 from .score.quality import load_jsonl, write_jsonl
 from .score.stats import task_key
 
@@ -601,8 +602,14 @@ def export_training(
 
     With a path source and no ``output``, writes ``<name>.train.jsonl``
     next to it. ``validate=True`` refuses to write a dataset whose tool
-    calls do not round-trip to structured arguments; pass ``validate=False``
-    to export anyway and read the report instead.
+    calls do not round-trip to structured arguments, or whose assistant
+    turns quote the row's own ``privileged`` block (the export scrubs the
+    key, not the reply that recited it); pass ``validate=False`` to export
+    anyway and read the report instead. The leak check reads the source
+    before the scrub, so pass the ``SimulationData`` or its
+    ``trajectories``; rows that already came through ``rows()``, ``save()``
+    or a file carry nothing to check, and ``report["privileged_leaks"]``
+    says so.
 
     ``format="openai"`` (the default) writes the OpenAI chat-completions
     wire row: the full ``messages`` list, ``function.arguments`` as a JSON
@@ -638,7 +645,20 @@ def export_training(
             "them teaches string-wrapped arguments. Fix the rows or pass "
             "validate=False."
         )
-    _, _, _, src = _resolve(source)
+    raw, _, _, src = _resolve(source)
+    # The scrub drops the ``privileged`` key at any depth and copies the
+    # assistant's reply through verbatim, so a reply that recited the block
+    # still recites it in the training file. Check the unscrubbed side,
+    # which is the only place the needles still exist (#249).
+    leaks = leak_report(raw)
+    if validate and leaks["n_leaked"]:
+        raise ValueError(
+            f"privileged_leak: {leaks['n_leaked']} of {leaks['n_checked']} rows quote "
+            "their own privileged context (reference, principle or hidden state) in "
+            "an assistant turn; the export scrubs the key, not the reply, so training "
+            "on them teaches the model to say what only the grader was told. Drop "
+            "those rows (leak_report(...)['leaked'] names them) or pass validate=False."
+        )
     dest = output
     if not dest and src:
         path = Path(src)
@@ -659,6 +679,9 @@ def export_training(
         "tool_output_chars_cut": sum(int(r.get("tool_output_chars_cut") or 0) for r in rows),
         "trained_messages": sum(sum(r["loss_mask"]) for r in rows),
         "masked_messages": sum(len(r["loss_mask"]) - sum(r["loss_mask"]) for r in rows),
+        "privileged_leaks": {
+            k: leaks[k] for k in ("checked", "n_checked", "n_leaked", "leaked", "summary")
+        },
     }
     # SFT clones every row it is given. A failed rollout in the file
     # teaches the failure, so say how many there are instead of leaving
@@ -673,6 +696,19 @@ def export_training(
         "n_ungraded": len(rows) - n_pass - n_fail,
     }
     warnings: list[str] = []
+    if leaks["n_leaked"]:
+        warnings.append(
+            f"{leaks['n_leaked']} of {leaks['n_checked']} rows quote their own privileged "
+            "context in an assistant turn and are exported anyway (validate=False); "
+            "report['privileged_leaks']['leaked'] names them."
+        )
+    n_cut = sum(1 for r in raw if isinstance(r, dict) and r.get("finish_reason") == "length")
+    if n_cut:
+        warnings.append(
+            f"{n_cut} of {len(raw)} rows were cut by the reply token cap (finish_reason "
+            "'length') and are exported as SFT targets; a model trained on them learns to "
+            "stop mid-thought. Filter on finish_reason == 'stop' or raise agent_max_tokens=."
+        )
     if n_fail:
         warnings.append(
             f"{n_fail} of {len(rows)} rows have reward below 0.5 and are exported as "
@@ -698,6 +734,7 @@ export_dataset = export_training
 _PAIR_KEYS = (
     "tie",
     "pairwise",
+    "first_turn_differs",
     "chosen_score",
     "rejected_score",
     "margin",
@@ -817,6 +854,12 @@ def export_preference(
         length_note = length_confound_warning(chosen_longer, len(deltas))
         if length_note:
             report["warnings"] = [length_note]
+    identical = sum(1 for r in out_rows if r.get("first_turn_differs") is False)
+    if identical:
+        from .score.judging import first_turn_note
+
+        report["first_turn_identical"] = identical
+        report.setdefault("warnings", []).append(first_turn_note(identical, len(out_rows)))
     margins = [r["margin"] for r in out_rows if isinstance(r.get("margin"), (int, float))]
     if margins:
         report["mean_margin"] = round(sum(margins) / len(margins), 4)

{whileai-0.53 → whileai-0.55}/whileai/simulations/generate/agents.py

@@ -10,7 +10,7 @@ import os
 import re
 import threading
 import time
-from collections.abc import Callable
+from collections.abc import Callable, Mapping
 from typing import Any
 from urllib.parse import urlparse
 
@@ -531,6 +531,7 @@ def complete(
     timeout: float = 60,
     n: int = 1,
     logprobs: bool | str = False,
+    extra: Mapping[str, Any] | None = None,
 ) -> dict:
     """POST /chat/completions. Reuses a thread-local keep-alive connection.
 
@@ -577,6 +578,8 @@ def complete(
     if tools:
         payload["tools"] = _wire_tools(tools)
     payload.update(_request_extras(base_url, model))
+    if extra:
+        payload.update(dict(extra))
     headers = {"Content-Type": "application/json", "Connection": "keep-alive"}
     if key:
         headers["Authorization"] = f"Bearer {key}"
@@ -730,8 +733,21 @@ def _strip_tool_markup(text: str) -> str:
     return cleaned.strip()
 
 
+_THINK_BLOCK = re.compile(r"<think>.*?</think>\s*", re.S | re.I)
+_THINK_OPEN = re.compile(r"<think>.*\Z", re.S | re.I)
+
+
+def _strip_think(text: str) -> str:
+    """Drop a thinking model's reasoning markup. A closed block goes whole;
+    an unclosed ``<think>`` (the token cap landed inside it) goes to the
+    end. What is left is the reply, which is what a grader, a marker and
+    the next turn's history should see (#264)."""
+    text = _THINK_BLOCK.sub("", text)
+    return _THINK_OPEN.sub("", text)
+
+
 def _spoken_text(reply: dict) -> str:
-    return _strip_tool_markup(str(reply.get("content") or ""))
+    return _strip_tool_markup(_strip_think(str(reply.get("content") or "")))
 
 
 def _calls_from_reply(reply: dict) -> tuple[list[dict], dict]:
@@ -1391,9 +1407,23 @@ def local_model(
     timeout: float = 60,
     max_tokens: int | None = None,
     user_model: str | None = None,
+    thinking: bool | None = None,
 ) -> Callable:
+    """An agent that talks to an OpenAI-compatible endpoint (a served
+    adapter, a local vLLM, any chat server) for ``simulate(agent=...)``.
+
+    ``thinking`` is for reasoning bases such as Qwen3: ``False`` sends
+    ``chat_template_kwargs={"enable_thinking": False}`` so the reply is
+    the answer, not the reasoning, the way the hosted Qwen path already
+    does; ``True`` asks for it; ``None`` (the default) sends nothing and
+    leaves the server's default. Either way ``<think>`` markup never
+    reaches ``step["text"]`` or ``final_text``.
+    """
     local = threading.local()
     plans = fault_plans if fault_plans is not None else {}
+    extras: dict[str, Any] | None = (
+        None if thinking is None else {"chat_template_kwargs": {"enable_thinking": bool(thinking)}}
+    )
     # The simulated user's model. None means the agent's own model plays
     # the user (the default); a backend spec moves that role to another
     # model, with the key resolved for that endpoint.
@@ -1438,6 +1468,7 @@ def local_model(
                     temperature=temperature,
                     timeout=timeout,
                     max_tokens=120,
+                    extra=extras,
                 )
                 opener_text = (_spoken_text(greet) or "").strip()
                 if opener_text:
@@ -1477,6 +1508,7 @@ def local_model(
                 timeout=timeout,
                 max_tokens=reply_budget(max_tokens),
                 logprobs=logprobs,
+                extra=extras,
             )
             calls, assistant = _calls_from_reply(reply)
             # One agent turn, one set of sampling facts, on its first step.

{whileai-0.53 → whileai-0.55}/whileai/simulations/run/engine.py

@@ -160,6 +160,32 @@ def _agent_error_text(exc: BaseException) -> str:
     return f"<agent error: {type(exc).__name__}: {public_llm_error(exc)}>"
 
 
+FINISH_REASONS = ("stop", "length", "tool", "error")
+
+
+def _finish_reason(raw: dict, steps: list, final_text: str) -> str:
+    """Why the rollout ended, on the row where a trainer can read it.
+
+    ``length``: a turn was cut by the reply token cap (the backend said
+    so). ``error``: the agent raised. ``tool``: the last thing the agent
+    did was call a tool and no final reply followed, so the turn budget
+    ran out. ``stop``: the agent finished on its own. A callable agent may
+    say it outright with ``finish_reason`` in what it returns. A length
+    cut scored 0 teaches the cheapest fix, shorter thinking, before it
+    teaches the task (#253), so the trainer masks these by default.
+    """
+    told = raw.get("finish_reason")
+    if isinstance(told, str) and told in FINISH_REASONS:
+        return told
+    if final_text.startswith("<agent error:"):
+        return "error"
+    if any(isinstance(s, dict) and s.get("truncated") for s in steps):
+        return "length"
+    if not final_text.strip() and steps and isinstance(steps[-1], dict) and steps[-1].get("tool"):
+        return "tool"
+    return "stop"
+
+
 def _hit_length_cap(row: dict) -> bool:
     """A step the backend flagged as cut by its token cap, or a reply that
     ends mid-sentence by the hygiene rule."""
@@ -802,6 +828,7 @@ class Run:
             t["seeded"] = [str(x) for x in seeded]
         t.update(_row_conversation(meta, prompt, c.seed))
         t["behavior_signature"] = behavior_signature(t)
+        t["finish_reason"] = _finish_reason(raw, t["steps"], t["final_text"])
         # Sampling facts roll up from the agent turns: the summed logprob
         # and token count a trainer needs for an importance ratio or a KL.
         lp_steps = [

{whileai-0.53 → whileai-0.55}/whileai/simulations/run/spec.py

@@ -69,8 +69,8 @@ RUBRIC_FILES = ("rubric.md", "rubric.txt")
 def spec_rubric(spec: Any) -> str | None:
     """The rubric shipped with a spec: what doing the job means, in prose,
     for the judge. ``rubric.md`` (or ``.txt``) next to the spec file the
-    loader resolves ``spec`` to (so ``spec="github"`` reads
-    ``specs/github/rubric.md``, the same shorthand ``simulate`` accepts),
+    loader resolves ``spec`` to (so ``spec="my-agent"`` reads
+    ``specs/my-agent/rubric.md``, the same shorthand ``simulate`` accepts),
     or a ``rubric`` key in the spec dict. None when the spec carries none."""
     if isinstance(spec, dict):
         text = str(spec.get("rubric") or "").strip()

{whileai-0.53 → whileai-0.55}/whileai/simulations/schema.py

@@ -561,6 +561,9 @@ def attach(row: dict, judgment: Judgment) -> dict:
     meta = dict(judgment.evidence)
     if judgment.scorer.version:
         meta["version"] = judgment.scorer.version
+    if judgment.scorer.kind != "judge":
+        # A named scorer reads back as "judge" unless the row says otherwise.
+        meta["scorer_kind"] = judgment.scorer.kind
     if meta:
         row["judge_meta"] = meta
     return row
@@ -576,6 +579,19 @@ def _scorer_version(row: dict) -> str | None:
     return None
 
 
+def _scorer_kind(row: dict) -> str | None:
+    """The kind the grading run stamped, if any (``judge_meta.scorer_kind``)."""
+    meta = row.get("judge_meta")
+    if isinstance(meta, dict) and meta.get("scorer_kind") in (
+        "rule",
+        "judge",
+        "reward_model",
+        "human",
+    ):
+        return str(meta["scorer_kind"])
+    return None
+
+
 def _judgments(row: dict, rollout_id: str) -> list[Judgment]:
     out: list[Judgment] = []
     has_primary = (
@@ -588,7 +604,11 @@ def _judgments(row: dict, rollout_id: str) -> list[Judgment]:
         judge = row.get("judge_name")
         label = row.get("label_source")
         name = judge or label or "unlabeled"
-        kind: Any = "judge" if judge else "rule"
+        # The stamped kind wins; without one, a named judge is a model judge
+        # and a bare label is a rule. A Verifier run through ``run_judge``
+        # carries ``judge_name`` too, so the inference alone called every
+        # verifier a judge (#250).
+        kind: Any = _scorer_kind(row) or ("judge" if judge else "rule")
         reward = _number(row.get("reward"))
         status: Any = row.get("judge_status") or "ok"
         evidence: dict = {}
@@ -802,7 +822,11 @@ def to_row(
         if primary.reason:
             row["reason"] = primary.reason
         if primary.scorer.name != "unlabeled":
-            if primary.scorer.kind == "judge":
+            # ``judge_name`` is what a grading run called itself, whatever
+            # its kind; a stamped kind proves a run named it. ``label_source``
+            # is the engine's own rule label.
+            stamped = (primary.evidence.get("judge_meta") or {}).get("scorer_kind")
+            if primary.scorer.kind != "rule" or stamped:
                 row["judge_name"] = primary.scorer.name
                 if primary.evidence.get("label_source"):
                     row["label_source"] = primary.evidence["label_source"]

{whileai-0.53 → whileai-0.55}/whileai/simulations/schemas/row-v1.json

@@ -44,6 +44,7 @@
         "token_logprobs": {"type": "array", "items": {"type": "number"}, "$comment": "per-token log-probabilities in generation order across the agent's turns; simulate(logprobs=\"tokens\")"},
         "sampling": {"type": ["object", "null"], "properties": {"temperature": {"type": "number"}, "max_tokens": {"type": "integer"}, "model": {"type": "string"}}, "$comment": "how the row was sampled: a model backend records its resolved temperature, reply budget and model; a callable agent records simulate(sampling=) verbatim, else null"},
         "policy_version": {"type": "string", "$comment": "Rollout.policy.version: <model_version>@<sha256 of the system policy>[:16]"},
+        "finish_reason": {"type": "string", "enum": ["stop", "length", "tool", "error"], "$comment": "why the rollout ended: on its own, cut by the reply token cap, turn budget ran out on a tool call, or the agent raised"},
         "writer_model": {"type": "string", "$comment": "who wrote the prompt: the situation writer's model tag, or template / seed / pinned when no model wrote it"},
         "user_model": {"type": "string", "$comment": "who played the simulated user; absent when the agent took a single message"},
         "calibration": {

{whileai-0.53 → whileai-0.55}/whileai/simulations/score/delta.py

@@ -370,6 +370,13 @@ def delta_report(
             f"Before allowed {cfg_a['max_tokens']} reply tokens and after {cfg_b['max_tokens']}; "
             "re-run one side so both use the same agent_max_tokens=."
         )
+    if _both("truncated_share") and abs(cfg_a["truncated_share"] - cfg_b["truncated_share"]) > 0.05:
+        warnings.append(
+            f"The token cap cut {cfg_a['truncated_share']:.0%} of before rows and "
+            f"{cfg_b['truncated_share']:.0%} of after rows; a side that is cut more often is "
+            "not the same eval. Raise agent_max_tokens= on both sides or read the delta with "
+            "that in mind."
+        )
     if _both("policy_version") and cfg_a["policy_version"] == cfg_b["policy_version"]:
         warnings.append(
             "Before and after are the same policy version; this compares a model to itself."

{whileai-0.53 → whileai-0.55}/whileai/simulations/score/grading.py

@@ -40,6 +40,15 @@ _FAULT_ALIASES = {
     "entity already acted on": "already_done",
 }
 _CLEAN_FAULT = {"", "ok", "created", "deleted", "success", "exit_0", "no fault", "clean"}
+#: Statuses a tool result can carry that mean the call failed. A ``status``
+#: key with any other value (``paid``, ``open``, ``completed``) is the
+#: tool's own vocabulary, not a fault (#261).
+_KNOWN_FAULTS = (
+    _BAD_STATUS
+    | set(_FAULT_ALIASES)
+    | set(_FAULT_ALIASES.values())
+    | {"stale", "malformed", "already_done", "not_found", "deny", "garbled"}
+)
 _HTTP_FAIL = re.compile(r"^[45]\d\d$")
 _INFRA_STUB = re.compile(
     r"^<agent error:|returned\s+[45]\d\d\b|^(https?://\S+\s+)?[45]\d\d(\s|$)", re.I
@@ -426,10 +435,10 @@ def _fault_from_result(result) -> str:
     if status in {"already_done", "already_acted_on"} or reason == "already_acted_on":
         return "already_done"
     name = normalize_fault_name(status)
-    if name:
+    if name and (name in _KNOWN_FAULTS or status in _KNOWN_FAULTS):
         return name
     if _step_faulted(result):
-        return normalize_fault_name(status) or "error"
+        return name or "error"
     return ""
 
 

{whileai-0.53 → whileai-0.55}/whileai/simulations/score/judging.py

@@ -402,6 +402,11 @@ def run_judge(
     name = judge_name or getattr(judge, "__name__", "") or _instance_name(judge) or "judge"
     if name == "<lambda>":
         name = "lambda_judge"
+    # A Verifier says what it is (``kind="rule"``); a function judge does
+    # not, and the schema then infers "judge" from the name. Stamp the
+    # declared kind so a verifier does not read back as a model judge (#250).
+    kind = getattr(judge, "kind", None)
+    scorer_kind = kind if kind in ("rule", "reward_model", "human") else None
     verdicts: list[dict[str, Any]]
     if concurrency > 1 and len(src_rows) > 1:
         with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
@@ -429,8 +434,11 @@ def run_judge(
             out["reason"] = verdict["reason"]
         out["judge_status"] = verdict["judge_status"]
         out["judge_name"] = name
-        if verdict["judge_meta"]:
-            out["judge_meta"] = verdict["judge_meta"]
+        meta = dict(verdict["judge_meta"] or {})
+        if scorer_kind:
+            meta["scorer_kind"] = scorer_kind
+        if meta:
+            out["judge_meta"] = meta
         fc = (verdict["judge_meta"] or {}).get("failure_class")
         if fc:
             out["failure_class"] = str(fc)
@@ -555,6 +563,42 @@ def length_confound_warning(chosen_longer: int, n: int) -> str | None:
     return None
 
 
+def _first_turn(row: dict) -> str:
+    """What the policy emitted first, read the way the hosted DPO trainer
+    reads it: the first tool step as its call, else the first assistant
+    text, else ``final_text``."""
+    steps = [s for s in (row.get("steps") or []) if isinstance(s, dict)]
+    for step in steps:
+        if step.get("tool"):
+            args = step.get("arguments")
+            if args is None:
+                args = step.get("args")
+            return json.dumps({"name": step["tool"], "arguments": args or {}}, sort_keys=True)
+    for step in steps:
+        if str(step.get("text") or "").strip():
+            return str(step["text"]).strip()
+    for message in row.get("messages") or []:
+        if isinstance(message, dict) and message.get("role") == "assistant":
+            text = str(message.get("content") or "").strip()
+            if text:
+                return text
+    return str(row.get("final_text") or "").strip()
+
+
+def first_turn_note(identical: int, n: int) -> str:
+    """The warning for pairs whose first assistant turns read the same."""
+    if not identical:
+        return ""
+    left = n - identical
+    return (
+        f"{identical}/{n} pairs have identical first assistant turns (same opening tool call "
+        "or line); their contrast is later in the rollout. The hosted DPO trainer compares "
+        f"first turns only and will drop them, leaving {left} (it needs at least 8). Keep "
+        "[p for p in pairs if p['first_turn_differs']] to see what it will train on, or "
+        "export_preference(pairs) for a trainer that reads whole conversations."
+    )
+
+
 def build_preference_pairs(
     rows: Sequence[dict],
     *,
@@ -643,6 +687,7 @@ def build_preference_pairs(
                     "rejected_model": r_model,
                     "same_policy": (c_model == r_model) if c_model and r_model else None,
                     "length_delta": reply_length(chosen) - reply_length(rejected),
+                    "first_turn_differs": _first_turn(chosen) != _first_turn(rejected),
                     "chosen_reason": str(chosen.get("reason") or ""),
                     "rejected_reason": str(rejected.get("reason") or ""),
                     "rejected_failure_class": rejected.get("failure_class"),
@@ -665,10 +710,13 @@ def build_preference_pairs(
         for p in pairs
         if p["chosen_score"] not in (0.0, 1.0) or p["rejected_score"] not in (0.0, 1.0)
     )
+    identical = sum(1 for p in pairs if not p["first_turn_differs"])
     warnings: list[str] = []
     length_note = length_confound_warning(chosen_longer, n)
     if length_note:
         warnings.append(length_note)
+    if identical:
+        warnings.append(first_turn_note(identical, n))
     if mixed_policy:
         warnings.append(
             f"{mixed_policy}/{n} pairs mix policies (chosen and rejected from different "
@@ -686,6 +734,8 @@ def build_preference_pairs(
         "min_margin": min_margin,
         "mean_margin": round(sum(p["margin"] for p in pairs) / n, 4) if n else None,
         "partial_score_pairs": partial,
+        "first_turn_identical": identical,
+        "trainer_pairs": n - identical,
         "same_policy_pairs": same_policy,
         "mixed_policy_pairs": mixed_policy,
         "eval_sourced": eval_pairs,

{whileai-0.53 → whileai-0.55}/whileai/simulations/score/passat.py

@@ -114,6 +114,14 @@ def run_config(
             mixed.append(key)
         out[key] = next(iter(values)) if len(values) == 1 else None
     out["mixed"] = mixed
+    # The share of rows the token cap cut. A side that was cut more often
+    # is not the same eval; ``delta_report`` warns when the two differ.
+    # ``None`` when no row says how it finished (rows from before 0.54).
+    reasons = [r.get("finish_reason") for r in rows if isinstance(r, dict)]
+    known = [x for x in reasons if isinstance(x, str)]
+    out["truncated_share"] = (
+        round(sum(1 for x in known if x == "length") / len(known), 4) if known else None
+    )
     return out
 
 
@@ -190,6 +198,9 @@ class PassAt:
             f"headroom {fmt(self.headroom)}"
         )
         tail = f"({self.n_groups} groups, k={self.k}"
+        cut = self.config.get("truncated_share") if self.config else None
+        if cut:
+            tail += f"; {cut:.0%} of rows cut by the token cap"
         if self.note:
             tail += f"; {self.note}"
         return f"{head} {tail})"

{whileai-0.53 → whileai-0.55}/whileai/simulations/score/privileged.py

@@ -15,7 +15,6 @@ it is vacuous. The report says so instead of passing.
 from __future__ import annotations
 
 import re
-from collections.abc import Sequence
 from typing import Any
 
 from .style import assistant_text
@@ -52,9 +51,14 @@ def _needles(privileged: Any, *, min_len: int) -> list[tuple[str, str]]:
     return out
 
 
-def leak_report(rows: Sequence[dict], *, min_len: int = 12) -> dict[str, Any]:
+def leak_report(rows: Any, *, min_len: int = 12) -> dict[str, Any]:
     """Which rows quote their own ``privileged`` block in the agent's text.
 
+    Takes the ``SimulationData`` itself, ``data.trajectories``, or any list
+    of rows. Given the data object it reads the trajectories, which still
+    carry the block; ``data.rows()`` is the scrubbed export and checks
+    nothing (the report says so).
+
     Checks every row that carries ``privileged`` (``reference``,
     ``principle``, and every string in ``hidden_state`` at least
     ``min_len`` characters long) against the final reply and every
@@ -69,6 +73,8 @@ def leak_report(rows: Sequence[dict], *, min_len: int = 12) -> dict[str, Any]:
     and the report is vacuous. When it can tell the rows came through the
     export, ``summary`` says so and names the accessor to use instead.
     """
+    if hasattr(rows, "trajectories") and not isinstance(rows, (list, tuple)):
+        rows = rows.trajectories
     n_rows = 0
     n_checked = 0
     exported = False

{whileai-0.53 → whileai-0.55}/whileai/simulations/training.py

@@ -474,6 +474,11 @@ class TrainingRun:
         else:
             self.finish("done")
 
+    @property
+    def id(self) -> str:
+        """The run id, the handle ``get_run``, ``serve`` and ``delete_run`` take."""
+        return self.run_id
+
     def __repr__(self) -> str:
         return (
             f"TrainingRun({self.run_id!r}, {self.name!r}, status={self.status!r}, step={self.step})"
@@ -558,6 +563,10 @@ def _measured_temperature(
     return None
 
 
+#: what GRPO does with a sampled reply the token cap cut
+TRUNCATED = ("mask", "zero")
+
+
 def train(
     dataset: str,
     *,
@@ -573,6 +582,7 @@ def train(
     max_completion_length: int | None = None,
     loss_type: str | None = None,
     temperature: float | None = None,
+    truncated: str | None = None,
     config: Mapping[str, Any] | None = None,
     wait: bool = False,
     timeout: float | None = None,
@@ -612,10 +622,15 @@ def train(
     sampling temperature the trainer rolls out at (GRPO); the dataset's
     rows say what they were measured at under ``sampling.temperature``,
     and ``train`` says so when the two differ, since a before/after
-    comparison across temperatures is not like for like. Each has a
+    comparison across temperatures is not like for like. ``truncated``
+    says what GRPO does with a sampled reply the token cap cut:
+    ``"mask"`` (the default) gives it no gradient, ``"zero"`` scores it 0
+    the old way. A cut reply scored 0 teaches shorter thinking before it
+    teaches the task, so ``"zero"`` is the knob to reach for only when the
+    cap itself is the behavior under training (#253). Each has a
     trainer default when left ``None``. ``config`` passes further host
     keys as given
-    (``epsilonHigh``, ``scaleRewards``, ``maskTruncated``, ``balance``).
+    (``epsilonHigh``, ``scaleRewards``, ``balance``).
     Every knob lands on the run's ``config`` so the run page shows it.
 
     A dataset already training answers with that run instead of a second.
@@ -682,6 +697,13 @@ def train(
         if not 0 < float(temperature) <= 2:
             raise ValueError("temperature: above 0 and at most 2")
         body["temperature"] = float(temperature)
+    if truncated is not None:
+        if method != "grpo":
+            raise ValueError("truncated= says what GRPO does with a token-capped reply; grpo only")
+        if truncated not in TRUNCATED:
+            raise ValueError(f"truncated must be one of {', '.join(TRUNCATED)}; got {truncated!r}")
+    if method == "grpo":
+        body["maskTruncated"] = (truncated or "mask") == "mask"
     for key, value in dict(config or {}).items():
         if key in body:
             raise ValueError(f"config[{key!r}] collides with a named argument")
@@ -808,11 +830,40 @@ def models(*, api_key: str | None = None) -> list[dict[str, Any]]:
     """The account's hosted models: ``name``, ``baseModel``, ``adapter``,
     ``adapterRunId``, ``version``, ``endpoint`` (an OpenAI-compatible
     base URL; send the account key as the bearer and ``name`` as the
-    model)."""
+    model).
+
+    A row here is a registry entry, not a running GPU: the endpoint
+    behind it idles to zero on its own and an unused model costs nothing.
+    The row stays until ``unserve(name)`` removes it; serving the same
+    name again bumps its ``version`` rather than adding a row."""
     out = _call("GET", "/models", api_key)
     return list(out.get("models") or []) if isinstance(out, dict) else []
 
 
+def unserve(
+    name: str,
+    *,
+    api_key: str | None = None,
+    transport: Callable[..., Any] | None = None,
+) -> dict[str, Any]:
+    """Stop hosting ``name``: removes the model row from the account, so
+    ``models()`` no longer lists it and its endpoint stops answering for
+    that name. The inverse of ``serve``, the way ``delete_dataset`` is the
+    inverse of ``push``. The adapter weights and the training run stay;
+    ``serve`` the run again to bring it back (at version 1).
+    Returns ``{"name": ..., "deleted": True}``."""
+    call = transport or _call
+    key = str(name).strip().lower()
+    if not key:
+        raise ValueError("unserve: name is the hosted model's name, as models() lists it")
+    out = call("DELETE", f"/models/{key}", api_key)
+    return dict(out) if isinstance(out, dict) else {"name": key, "deleted": True}
+
+
+#: Same call, the other spelling: symmetric with ``delete_dataset``.
+delete_model = unserve
+
+
 def serve(
     name: str,
     run: TrainingRun | str | None = None,
@@ -825,18 +876,36 @@ def serve(
     row; ``endpoint`` is the OpenAI-compatible base URL and ``name`` the
     model id to send. Posting an existing name bumps ``version``.
 
-    ``run`` is a ``TrainingRun`` or its id; the adapter and base model
-    come from the run record unless ``base_model`` is given. No ``run``
-    serves the bare base (``base_model`` required).
+    ``run`` is a ``TrainingRun``, the record ``get_run`` returns, or the
+    run id; the adapter and base model come from the run record unless
+    ``base_model`` is given. No ``run`` serves the bare base
+    (``base_model`` required). ``unserve`` is the inverse.
     """
     call = transport or _call
     adapter: str | None = None
     base = base_model
+    run_id: str | None
     if isinstance(run, TrainingRun):
         adapter = run.adapter
-        run_id: str | None = run.run_id
+        run_id = run.run_id
+    elif isinstance(run, dict):
+        # The record ``get_run`` returns. Reading the id here is what lets
+        # train in one process and serve in the next compose (#262).
+        run_id = str(run.get("runId") or run.get("run_id") or run.get("id") or "").strip() or None
+        if not run_id:
+            raise TypeError(
+                "run is a dict with no runId; pass the record wai.get_run(run_id) returns, "
+                "a TrainingRun, or the run id string"
+            )
+        adapter = str(run.get("adapter") or "").strip() or None
+        base = base or run.get("baseModel") or run.get("base_model")
+    elif run is None or isinstance(run, str):
+        run_id = (str(run).strip() or None) if run else None
     else:
-        run_id = str(run) if run else None
+        raise TypeError(
+            "run must be a TrainingRun, the record wai.get_run(run_id) returns, or the run "
+            f"id string; got {type(run).__name__}"
+        )
     if run_id and (adapter is None or base is None):
         meta = call("GET", f"/runs/{run_id}", api_key)
         meta = meta if isinstance(meta, dict) else {}

{whileai-0.53 → whileai-0.55/whileai.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.53
+Version: 0.55
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -51,7 +51,7 @@ Two ways in, one engine. Give it the agent's tools and system prompt and it samp
 
 ![How a row gets made: the draw, the coverage grid, the search arms, the rollout, the split](docs/how-a-row-gets-made.svg)
 
-A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The interactive version, running on real rows, is at [zeroproofai.com/docs/engine](https://zeroproofai.com/docs/engine).
+A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The engine on one page, with references: [docs/engine.md](docs/engine.md), also at [while.ai/docs/engine](https://while.ai/docs/engine).
 
 ## Overview
 
@@ -230,7 +230,7 @@ data.grade(rubric=RUBRIC)  # 2 grade against the task rubric: reward 0/1 on ever
 print(data.pass_at)
 wai.judge_trust(data.trajectories)  # 3 trust the numbers
 rows, report = wai.optimize(data, mode="rl")  # 4 prune to what carries gradient
-entry = wai.push_rows(rows, "github-rl-v1", gate=True, mode="rl")  # 5 publish, gated
+entry = wai.push_rows(rows, "my-agent-rl-v1", gate=True, mode="rl")  # 5 publish, gated
 ```
 
 `situations=200, repeats=8` is a guess. `wai.recommend(tools=TOOLS, system_prompt=POLICY, mode="rl")` replaces it with numbers from this agent's own grid: [How much to run](#how-much-to-run).
@@ -391,9 +391,9 @@ installable `verifiers` package, the shape Prime Intellect and TRL read.
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY, mode="rl", repeats=8)
 data.grade()
 # reward and world must import by name in the trainer: a module-level function or "module:attr"
-wai.export_environment(data, "envs/github-agent", reward=my_verifier)
-# pip install -e envs/github-agent
-# vf-eval github_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
+wai.export_environment(data, "envs/my-agent", reward=my_verifier)
+# pip install -e envs/my-agent
+# vf-eval my_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
 ```
 
 The package holds `spec.json` (system prompt, the tool schemas verbatim,
@@ -681,10 +681,10 @@ login`.
 # export WHILEAI_DELEGATED_CREDENTIAL=credential["credential"]
 
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY)
-v1 = data.push("github-explore-v1")  # -> {"datasetId": "ds_...", ...}
+v1 = data.push("my-agent-explore-v1")  # -> {"datasetId": "ds_...", ...}
 
 # iterate, then push the next version with lineage
-v2 = data.push("github-explore-v2", parent=v1["datasetId"])
+v2 = data.push("my-agent-explore-v2", parent=v1["datasetId"])
 
 wai.datasets()  # list yours + storage used
 rows = wai.pull(v1["datasetId"])  # rows, or pass path= for a file