whileai 0.52__tar.gz → 0.54__tar.gz

{whileai-0.52/whileai.egg-info → whileai-0.54}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.52
+Version: 0.54
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -51,7 +51,7 @@ Two ways in, one engine. Give it the agent's tools and system prompt and it samp
 
 ![How a row gets made: the draw, the coverage grid, the search arms, the rollout, the split](docs/how-a-row-gets-made.svg)
 
-A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The interactive version, running on real rows, is at [zeroproofai.com/docs/engine](https://zeroproofai.com/docs/engine).
+A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The engine on one page, with references: [docs/engine.md](docs/engine.md), also at [while.ai/docs/engine](https://while.ai/docs/engine).
 
 ## Overview
 
@@ -230,7 +230,7 @@ data.grade(rubric=RUBRIC)  # 2 grade against the task rubric: reward 0/1 on ever
 print(data.pass_at)
 wai.judge_trust(data.trajectories)  # 3 trust the numbers
 rows, report = wai.optimize(data, mode="rl")  # 4 prune to what carries gradient
-entry = wai.push_rows(rows, "github-rl-v1", gate=True, mode="rl")  # 5 publish, gated
+entry = wai.push_rows(rows, "my-agent-rl-v1", gate=True, mode="rl")  # 5 publish, gated
 ```
 
 `situations=200, repeats=8` is a guess. `wai.recommend(tools=TOOLS, system_prompt=POLICY, mode="rl")` replaces it with numbers from this agent's own grid: [How much to run](#how-much-to-run).
@@ -391,9 +391,9 @@ installable `verifiers` package, the shape Prime Intellect and TRL read.
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY, mode="rl", repeats=8)
 data.grade()
 # reward and world must import by name in the trainer: a module-level function or "module:attr"
-wai.export_environment(data, "envs/github-agent", reward=my_verifier)
-# pip install -e envs/github-agent
-# vf-eval github_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
+wai.export_environment(data, "envs/my-agent", reward=my_verifier)
+# pip install -e envs/my-agent
+# vf-eval my_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
 ```
 
 The package holds `spec.json` (system prompt, the tool schemas verbatim,
@@ -681,10 +681,10 @@ login`.
 # export WHILEAI_DELEGATED_CREDENTIAL=credential["credential"]
 
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY)
-v1 = data.push("github-explore-v1")  # -> {"datasetId": "ds_...", ...}
+v1 = data.push("my-agent-explore-v1")  # -> {"datasetId": "ds_...", ...}
 
 # iterate, then push the next version with lineage
-v2 = data.push("github-explore-v2", parent=v1["datasetId"])
+v2 = data.push("my-agent-explore-v2", parent=v1["datasetId"])
 
 wai.datasets()  # list yours + storage used
 rows = wai.pull(v1["datasetId"])  # rows, or pass path= for a file

{whileai-0.52 → whileai-0.54}/README.md

@@ -15,7 +15,7 @@ Two ways in, one engine. Give it the agent's tools and system prompt and it samp
 
 ![How a row gets made: the draw, the coverage grid, the search arms, the rollout, the split](docs/how-a-row-gets-made.svg)
 
-A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The interactive version, running on real rows, is at [zeroproofai.com/docs/engine](https://zeroproofai.com/docs/engine).
+A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The engine on one page, with references: [docs/engine.md](docs/engine.md), also at [while.ai/docs/engine](https://while.ai/docs/engine).
 
 ## Overview
 
@@ -194,7 +194,7 @@ data.grade(rubric=RUBRIC)  # 2 grade against the task rubric: reward 0/1 on ever
 print(data.pass_at)
 wai.judge_trust(data.trajectories)  # 3 trust the numbers
 rows, report = wai.optimize(data, mode="rl")  # 4 prune to what carries gradient
-entry = wai.push_rows(rows, "github-rl-v1", gate=True, mode="rl")  # 5 publish, gated
+entry = wai.push_rows(rows, "my-agent-rl-v1", gate=True, mode="rl")  # 5 publish, gated
 ```
 
 `situations=200, repeats=8` is a guess. `wai.recommend(tools=TOOLS, system_prompt=POLICY, mode="rl")` replaces it with numbers from this agent's own grid: [How much to run](#how-much-to-run).
@@ -355,9 +355,9 @@ installable `verifiers` package, the shape Prime Intellect and TRL read.
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY, mode="rl", repeats=8)
 data.grade()
 # reward and world must import by name in the trainer: a module-level function or "module:attr"
-wai.export_environment(data, "envs/github-agent", reward=my_verifier)
-# pip install -e envs/github-agent
-# vf-eval github_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
+wai.export_environment(data, "envs/my-agent", reward=my_verifier)
+# pip install -e envs/my-agent
+# vf-eval my_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
 ```
 
 The package holds `spec.json` (system prompt, the tool schemas verbatim,
@@ -645,10 +645,10 @@ login`.
 # export WHILEAI_DELEGATED_CREDENTIAL=credential["credential"]
 
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY)
-v1 = data.push("github-explore-v1")  # -> {"datasetId": "ds_...", ...}
+v1 = data.push("my-agent-explore-v1")  # -> {"datasetId": "ds_...", ...}
 
 # iterate, then push the next version with lineage
-v2 = data.push("github-explore-v2", parent=v1["datasetId"])
+v2 = data.push("my-agent-explore-v2", parent=v1["datasetId"])
 
 wai.datasets()  # list yours + storage used
 rows = wai.pull(v1["datasetId"])  # rows, or pass path= for a file

{whileai-0.52 → whileai-0.54}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "whileai"
-version = "0.52"
+version = "0.54"
 description = "While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt."
 readme = "README.md"
 license = {text = "Apache-2.0"}

{whileai-0.52 → whileai-0.54}/whileai/simulations/__init__.py

@@ -7,12 +7,13 @@ malicious), and rolls the agent. Grade 0/1 later. Optimize for post-training.
 Five calls, spec to gated dataset:
 
     import whileai.simulations as wai
-    data = wai.simulate(agent="openai:gpt-4.1-mini", spec="specs/github",
-                        mode="rl", situations=200, repeats=8)      # generate
+    data = wai.simulate(agent="openai:gpt-4.1-mini", tools=my_tools,
+                        system_prompt=my_policy, mode="rl",
+                        situations=200, repeats=8)                  # generate
     scored = data.grade(judge=my_judge)                             # grade 0/1
     print(scored.pass_at); wai.judge_trust(scored.rows, judge=my_judge)  # trust
     rows, report = wai.optimize(scored, mode="rl")                  # prune
-    wai.push_rows(rows, "github-rl-v1", gate=True, mode="rl")       # publish, gated
+    wai.push_rows(rows, "my-agent-rl-v1", gate=True, mode="rl")     # publish, gated
 
 Everything else exported here is one layer down from those five.
 """

{whileai-0.52 → whileai-0.54}/whileai/simulations/data.py

@@ -652,6 +652,14 @@ class SimulationData:
         report["selection"] = self.search.get("selection")
         return report
 
+    def leak_report(self, *, min_len: int = 12) -> dict[str, Any]:
+        """Did any reply quote its own ``privileged`` block? Reads the
+        trajectories, which still carry the block; ``rows()`` is scrubbed
+        and would check nothing. Same report as ``leak_report``."""
+        from .score.privileged import leak_report
+
+        return leak_report(self.trajectories, min_len=min_len)
+
     @property
     def rows(self) -> RowList:
         """The exported rows: exactly what ``save()`` and ``output=`` write.

{whileai-0.52 → whileai-0.54}/whileai/simulations/environment.py

@@ -10,13 +10,13 @@ reward). ``export_environment`` writes those three as an installable
 ``verifiers`` package, the shape Prime Intellect and TRL consume::
 
     import whileai.simulations as wai
-    data = wai.simulate(spec="specs/github", mode="rl", repeats=8)
+    data = wai.simulate(tools=my_tools, system_prompt=my_policy, mode="rl", repeats=8)
     scored = data.grade()
-    wai.export_environment(scored, "envs/github-agent", reward=my_verifier)
+    wai.export_environment(scored, "envs/my-agent", reward=my_verifier)
 
     # then, with verifiers installed:
-    #   pip install -e envs/github-agent
-    #   vf-eval github_agent -a '{"split": "holdout"}' -m <policy> ...
+    #   pip install -e envs/my-agent
+    #   vf-eval my_agent -a '{"split": "holdout"}' -m <policy> ...
 
 What goes in the package:
 

{whileai-0.52 → whileai-0.54}/whileai/simulations/export.py

@@ -50,6 +50,7 @@ from pathlib import Path
 from typing import Any
 
 from .schema import check, stamp
+from .score.privileged import leak_report
 from .score.quality import load_jsonl, write_jsonl
 from .score.stats import task_key
 
@@ -601,8 +602,14 @@ def export_training(
 
     With a path source and no ``output``, writes ``<name>.train.jsonl``
     next to it. ``validate=True`` refuses to write a dataset whose tool
-    calls do not round-trip to structured arguments; pass ``validate=False``
-    to export anyway and read the report instead.
+    calls do not round-trip to structured arguments, or whose assistant
+    turns quote the row's own ``privileged`` block (the export scrubs the
+    key, not the reply that recited it); pass ``validate=False`` to export
+    anyway and read the report instead. The leak check reads the source
+    before the scrub, so pass the ``SimulationData`` or its
+    ``trajectories``; rows that already came through ``rows()``, ``save()``
+    or a file carry nothing to check, and ``report["privileged_leaks"]``
+    says so.
 
     ``format="openai"`` (the default) writes the OpenAI chat-completions
     wire row: the full ``messages`` list, ``function.arguments`` as a JSON
@@ -638,7 +645,20 @@ def export_training(
             "them teaches string-wrapped arguments. Fix the rows or pass "
             "validate=False."
         )
-    _, _, _, src = _resolve(source)
+    raw, _, _, src = _resolve(source)
+    # The scrub drops the ``privileged`` key at any depth and copies the
+    # assistant's reply through verbatim, so a reply that recited the block
+    # still recites it in the training file. Check the unscrubbed side,
+    # which is the only place the needles still exist (#249).
+    leaks = leak_report(raw)
+    if validate and leaks["n_leaked"]:
+        raise ValueError(
+            f"privileged_leak: {leaks['n_leaked']} of {leaks['n_checked']} rows quote "
+            "their own privileged context (reference, principle or hidden state) in "
+            "an assistant turn; the export scrubs the key, not the reply, so training "
+            "on them teaches the model to say what only the grader was told. Drop "
+            "those rows (leak_report(...)['leaked'] names them) or pass validate=False."
+        )
     dest = output
     if not dest and src:
         path = Path(src)
@@ -659,6 +679,9 @@ def export_training(
         "tool_output_chars_cut": sum(int(r.get("tool_output_chars_cut") or 0) for r in rows),
         "trained_messages": sum(sum(r["loss_mask"]) for r in rows),
         "masked_messages": sum(len(r["loss_mask"]) - sum(r["loss_mask"]) for r in rows),
+        "privileged_leaks": {
+            k: leaks[k] for k in ("checked", "n_checked", "n_leaked", "leaked", "summary")
+        },
     }
     # SFT clones every row it is given. A failed rollout in the file
     # teaches the failure, so say how many there are instead of leaving
@@ -673,6 +696,19 @@ def export_training(
         "n_ungraded": len(rows) - n_pass - n_fail,
     }
     warnings: list[str] = []
+    if leaks["n_leaked"]:
+        warnings.append(
+            f"{leaks['n_leaked']} of {leaks['n_checked']} rows quote their own privileged "
+            "context in an assistant turn and are exported anyway (validate=False); "
+            "report['privileged_leaks']['leaked'] names them."
+        )
+    n_cut = sum(1 for r in raw if isinstance(r, dict) and r.get("finish_reason") == "length")
+    if n_cut:
+        warnings.append(
+            f"{n_cut} of {len(raw)} rows were cut by the reply token cap (finish_reason "
+            "'length') and are exported as SFT targets; a model trained on them learns to "
+            "stop mid-thought. Filter on finish_reason == 'stop' or raise agent_max_tokens=."
+        )
     if n_fail:
         warnings.append(
             f"{n_fail} of {len(rows)} rows have reward below 0.5 and are exported as "

{whileai-0.52 → whileai-0.54}/whileai/simulations/run/engine.py

@@ -160,6 +160,32 @@ def _agent_error_text(exc: BaseException) -> str:
     return f"<agent error: {type(exc).__name__}: {public_llm_error(exc)}>"
 
 
+FINISH_REASONS = ("stop", "length", "tool", "error")
+
+
+def _finish_reason(raw: dict, steps: list, final_text: str) -> str:
+    """Why the rollout ended, on the row where a trainer can read it.
+
+    ``length``: a turn was cut by the reply token cap (the backend said
+    so). ``error``: the agent raised. ``tool``: the last thing the agent
+    did was call a tool and no final reply followed, so the turn budget
+    ran out. ``stop``: the agent finished on its own. A callable agent may
+    say it outright with ``finish_reason`` in what it returns. A length
+    cut scored 0 teaches the cheapest fix, shorter thinking, before it
+    teaches the task (#253), so the trainer masks these by default.
+    """
+    told = raw.get("finish_reason")
+    if isinstance(told, str) and told in FINISH_REASONS:
+        return told
+    if final_text.startswith("<agent error:"):
+        return "error"
+    if any(isinstance(s, dict) and s.get("truncated") for s in steps):
+        return "length"
+    if not final_text.strip() and steps and isinstance(steps[-1], dict) and steps[-1].get("tool"):
+        return "tool"
+    return "stop"
+
+
 def _hit_length_cap(row: dict) -> bool:
     """A step the backend flagged as cut by its token cap, or a reply that
     ends mid-sentence by the hygiene rule."""
@@ -802,6 +828,7 @@ class Run:
             t["seeded"] = [str(x) for x in seeded]
         t.update(_row_conversation(meta, prompt, c.seed))
         t["behavior_signature"] = behavior_signature(t)
+        t["finish_reason"] = _finish_reason(raw, t["steps"], t["final_text"])
         # Sampling facts roll up from the agent turns: the summed logprob
         # and token count a trainer needs for an importance ratio or a KL.
         lp_steps = [

{whileai-0.52 → whileai-0.54}/whileai/simulations/run/spec.py

@@ -69,8 +69,8 @@ RUBRIC_FILES = ("rubric.md", "rubric.txt")
 def spec_rubric(spec: Any) -> str | None:
     """The rubric shipped with a spec: what doing the job means, in prose,
     for the judge. ``rubric.md`` (or ``.txt``) next to the spec file the
-    loader resolves ``spec`` to (so ``spec="github"`` reads
-    ``specs/github/rubric.md``, the same shorthand ``simulate`` accepts),
+    loader resolves ``spec`` to (so ``spec="my-agent"`` reads
+    ``specs/my-agent/rubric.md``, the same shorthand ``simulate`` accepts),
     or a ``rubric`` key in the spec dict. None when the spec carries none."""
     if isinstance(spec, dict):
         text = str(spec.get("rubric") or "").strip()

{whileai-0.52 → whileai-0.54}/whileai/simulations/schema.py

@@ -561,6 +561,9 @@ def attach(row: dict, judgment: Judgment) -> dict:
     meta = dict(judgment.evidence)
     if judgment.scorer.version:
         meta["version"] = judgment.scorer.version
+    if judgment.scorer.kind != "judge":
+        # A named scorer reads back as "judge" unless the row says otherwise.
+        meta["scorer_kind"] = judgment.scorer.kind
     if meta:
         row["judge_meta"] = meta
     return row
@@ -576,6 +579,19 @@ def _scorer_version(row: dict) -> str | None:
     return None
 
 
+def _scorer_kind(row: dict) -> str | None:
+    """The kind the grading run stamped, if any (``judge_meta.scorer_kind``)."""
+    meta = row.get("judge_meta")
+    if isinstance(meta, dict) and meta.get("scorer_kind") in (
+        "rule",
+        "judge",
+        "reward_model",
+        "human",
+    ):
+        return str(meta["scorer_kind"])
+    return None
+
+
 def _judgments(row: dict, rollout_id: str) -> list[Judgment]:
     out: list[Judgment] = []
     has_primary = (
@@ -588,7 +604,11 @@ def _judgments(row: dict, rollout_id: str) -> list[Judgment]:
         judge = row.get("judge_name")
         label = row.get("label_source")
         name = judge or label or "unlabeled"
-        kind: Any = "judge" if judge else "rule"
+        # The stamped kind wins; without one, a named judge is a model judge
+        # and a bare label is a rule. A Verifier run through ``run_judge``
+        # carries ``judge_name`` too, so the inference alone called every
+        # verifier a judge (#250).
+        kind: Any = _scorer_kind(row) or ("judge" if judge else "rule")
         reward = _number(row.get("reward"))
         status: Any = row.get("judge_status") or "ok"
         evidence: dict = {}
@@ -802,7 +822,11 @@ def to_row(
         if primary.reason:
             row["reason"] = primary.reason
         if primary.scorer.name != "unlabeled":
-            if primary.scorer.kind == "judge":
+            # ``judge_name`` is what a grading run called itself, whatever
+            # its kind; a stamped kind proves a run named it. ``label_source``
+            # is the engine's own rule label.
+            stamped = (primary.evidence.get("judge_meta") or {}).get("scorer_kind")
+            if primary.scorer.kind != "rule" or stamped:
                 row["judge_name"] = primary.scorer.name
                 if primary.evidence.get("label_source"):
                     row["label_source"] = primary.evidence["label_source"]

{whileai-0.52 → whileai-0.54}/whileai/simulations/schemas/row-v1.json

@@ -44,6 +44,7 @@
         "token_logprobs": {"type": "array", "items": {"type": "number"}, "$comment": "per-token log-probabilities in generation order across the agent's turns; simulate(logprobs=\"tokens\")"},
         "sampling": {"type": ["object", "null"], "properties": {"temperature": {"type": "number"}, "max_tokens": {"type": "integer"}, "model": {"type": "string"}}, "$comment": "how the row was sampled: a model backend records its resolved temperature, reply budget and model; a callable agent records simulate(sampling=) verbatim, else null"},
         "policy_version": {"type": "string", "$comment": "Rollout.policy.version: <model_version>@<sha256 of the system policy>[:16]"},
+        "finish_reason": {"type": "string", "enum": ["stop", "length", "tool", "error"], "$comment": "why the rollout ended: on its own, cut by the reply token cap, turn budget ran out on a tool call, or the agent raised"},
         "writer_model": {"type": "string", "$comment": "who wrote the prompt: the situation writer's model tag, or template / seed / pinned when no model wrote it"},
         "user_model": {"type": "string", "$comment": "who played the simulated user; absent when the agent took a single message"},
         "calibration": {

{whileai-0.52 → whileai-0.54}/whileai/simulations/score/delta.py

@@ -370,6 +370,13 @@ def delta_report(
             f"Before allowed {cfg_a['max_tokens']} reply tokens and after {cfg_b['max_tokens']}; "
             "re-run one side so both use the same agent_max_tokens=."
         )
+    if _both("truncated_share") and abs(cfg_a["truncated_share"] - cfg_b["truncated_share"]) > 0.05:
+        warnings.append(
+            f"The token cap cut {cfg_a['truncated_share']:.0%} of before rows and "
+            f"{cfg_b['truncated_share']:.0%} of after rows; a side that is cut more often is "
+            "not the same eval. Raise agent_max_tokens= on both sides or read the delta with "
+            "that in mind."
+        )
     if _both("policy_version") and cfg_a["policy_version"] == cfg_b["policy_version"]:
         warnings.append(
             "Before and after are the same policy version; this compares a model to itself."

{whileai-0.52 → whileai-0.54}/whileai/simulations/score/judging.py

@@ -402,6 +402,11 @@ def run_judge(
     name = judge_name or getattr(judge, "__name__", "") or _instance_name(judge) or "judge"
     if name == "<lambda>":
         name = "lambda_judge"
+    # A Verifier says what it is (``kind="rule"``); a function judge does
+    # not, and the schema then infers "judge" from the name. Stamp the
+    # declared kind so a verifier does not read back as a model judge (#250).
+    kind = getattr(judge, "kind", None)
+    scorer_kind = kind if kind in ("rule", "reward_model", "human") else None
     verdicts: list[dict[str, Any]]
     if concurrency > 1 and len(src_rows) > 1:
         with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
@@ -429,8 +434,11 @@ def run_judge(
             out["reason"] = verdict["reason"]
         out["judge_status"] = verdict["judge_status"]
         out["judge_name"] = name
-        if verdict["judge_meta"]:
-            out["judge_meta"] = verdict["judge_meta"]
+        meta = dict(verdict["judge_meta"] or {})
+        if scorer_kind:
+            meta["scorer_kind"] = scorer_kind
+        if meta:
+            out["judge_meta"] = meta
         fc = (verdict["judge_meta"] or {}).get("failure_class")
         if fc:
             out["failure_class"] = str(fc)

{whileai-0.52 → whileai-0.54}/whileai/simulations/score/passat.py

@@ -114,6 +114,14 @@ def run_config(
             mixed.append(key)
         out[key] = next(iter(values)) if len(values) == 1 else None
     out["mixed"] = mixed
+    # The share of rows the token cap cut. A side that was cut more often
+    # is not the same eval; ``delta_report`` warns when the two differ.
+    # ``None`` when no row says how it finished (rows from before 0.54).
+    reasons = [r.get("finish_reason") for r in rows if isinstance(r, dict)]
+    known = [x for x in reasons if isinstance(x, str)]
+    out["truncated_share"] = (
+        round(sum(1 for x in known if x == "length") / len(known), 4) if known else None
+    )
     return out
 
 
@@ -190,11 +198,33 @@ class PassAt:
             f"headroom {fmt(self.headroom)}"
         )
         tail = f"({self.n_groups} groups, k={self.k}"
+        cut = self.config.get("truncated_share") if self.config else None
+        if cut:
+            tail += f"; {cut:.0%} of rows cut by the token cap"
         if self.note:
             tail += f"; {self.note}"
         return f"{head} {tail})"
 
 
+def _nothing_to_score(rows: Sequence[dict]) -> str:
+    """Why no row carried a binary reward.
+
+    "grade first" is right when nothing has been judged, and wrong -- it
+    sends the user back to the step that already ran -- when grading did
+    happen and every row failed. A cold hosted judge does exactly that: all
+    the concurrent calls time out together and the whole set reads as
+    ungraded.
+    """
+    judged = [r for r in rows if isinstance(r, dict) and r.get("judge_status")]
+    failed = [r for r in judged if str(r.get("judge_status")) != "ok"]
+    if not judged or len(failed) != len(judged):
+        return "no binary rewards; grade first"
+    statuses = "/".join(sorted({str(r.get("judge_status")) for r in failed}))
+    reason = next((str(r.get("reason") or "").strip() for r in failed if r.get("reason")), "")
+    tail = f": {reason[:120]}" if reason else ""
+    return f"the judge failed on all {len(failed)} rows ({statuses}){tail}; re-run the judge"
+
+
 def pass_at(
     rows: Sequence[dict] | Any,
     *,
@@ -249,7 +279,7 @@ def pass_at(
             pass_at_k=None,
             n_groups=0,
             n_rows=0,
-            note="no binary rewards; grade first",
+            note=_nothing_to_score(row_list),
             config=run_config(row_list, n_tasks=0, k=int(k or 1)),
         )
 

{whileai-0.52 → whileai-0.54}/whileai/simulations/score/privileged.py

@@ -15,7 +15,6 @@ it is vacuous. The report says so instead of passing.
 from __future__ import annotations
 
 import re
-from collections.abc import Sequence
 from typing import Any
 
 from .style import assistant_text
@@ -52,9 +51,14 @@ def _needles(privileged: Any, *, min_len: int) -> list[tuple[str, str]]:
     return out
 
 
-def leak_report(rows: Sequence[dict], *, min_len: int = 12) -> dict[str, Any]:
+def leak_report(rows: Any, *, min_len: int = 12) -> dict[str, Any]:
     """Which rows quote their own ``privileged`` block in the agent's text.
 
+    Takes the ``SimulationData`` itself, ``data.trajectories``, or any list
+    of rows. Given the data object it reads the trajectories, which still
+    carry the block; ``data.rows()`` is the scrubbed export and checks
+    nothing (the report says so).
+
     Checks every row that carries ``privileged`` (``reference``,
     ``principle``, and every string in ``hidden_state`` at least
     ``min_len`` characters long) against the final reply and every
@@ -63,9 +67,17 @@ def leak_report(rows: Sequence[dict], *, min_len: int = 12) -> dict[str, Any]:
     the block, so the result is vacuous), ``leaked`` (up to 20 rows:
     ``scenario_id``, ``rollout_index``, ``field``, ``needle``) and
     ``summary``. Does not mutate ``rows``.
+
+    Pass ``data.trajectories``, not ``data.rows()``: the export scrubs
+    ``privileged`` at any depth, so exported rows carry nothing to check
+    and the report is vacuous. When it can tell the rows came through the
+    export, ``summary`` says so and names the accessor to use instead.
     """
+    if hasattr(rows, "trajectories") and not isinstance(rows, (list, tuple)):
+        rows = rows.trajectories
     n_rows = 0
     n_checked = 0
+    exported = False
     leaked: list[dict[str, Any]] = []
     for row in rows:
         if not isinstance(row, dict):
@@ -73,6 +85,10 @@ def leak_report(rows: Sequence[dict], *, min_len: int = 12) -> dict[str, Any]:
         n_rows += 1
         needles = _needles(row.get("privileged"), min_len=min_len)
         if not needles:
+            # ``export_row`` always writes ``scenario_id`` (``""`` when the
+            # row has none), so a row that has the key but no privileged
+            # block was scrubbed on the way out rather than never filled.
+            exported = exported or "scenario_id" in row
             continue
         n_checked += 1
         hay = _norm(assistant_text(row))
@@ -94,6 +110,11 @@ def leak_report(rows: Sequence[dict], *, min_len: int = 12) -> dict[str, Any]:
             f"checked 0 of {n_rows} rows: none carried privileged context, "
             "so this says nothing about leaks"
         )
+        if exported:
+            summary += (
+                ". These came through the export (rows(), save(), push()), which scrubs "
+                "privileged at any depth -- pass data.trajectories, which keeps it"
+            )
     elif not n_leaked:
         summary = f"checked {n_checked} of {n_rows} rows: no reply quoted its privileged context"
     else:

{whileai-0.52 → whileai-0.54}/whileai/simulations/score/rubric.py

@@ -26,12 +26,19 @@ import concurrent.futures
 import hashlib
 import json
 import re
+import threading
 from collections.abc import Callable, Mapping, Sequence
 from dataclasses import dataclass, field
 from typing import Any, Literal
 
 from ..generate.agents import complete, parse_backend_spec
-from .grade_llm import JUDGE_TEMPERATURE, _render_payload, judge_spec, judge_version
+from .grade_llm import (
+    JUDGE_TEMPERATURE,
+    _render_payload,
+    judge_spec,
+    judge_version,
+    warm_judge,
+)
 
 Kind = Literal["hard", "principle", "pitfall"]
 KINDS: tuple[str, ...] = ("hard", "principle", "pitfall")
@@ -375,10 +382,28 @@ def rubric_judge(
     ``markers`` (``rubric:<slug>`` = 1.0 met / 0.0 not, and for a pitfall
     1.0 clean / 0.0 exhibited), ``criteria`` (the raw verdicts),
     ``rubric_version`` and the score breakdown. The judge's name folds the
-    rubric version in when one is fixed."""
+    rubric version in when one is fixed.
+
+    The hosted judge scales to zero, so the first row through warms it once
+    (``warm_judge``, a 600s budget) while the rest of the fan-out waits.
+    Without that, ``run_judge``'s eight concurrent calls all raced a
+    container that was still loading its weights and every row came back
+    ``invalid_result`` with a ``TimeoutError``. Warm-up failure is not
+    fatal: the rows are judged anyway and report the real error."""
     resolved = judge_spec(spec=spec)
     url, model = parse_backend_spec(resolved)
     system = str(prompt or "").strip() or RUBRIC_JUDGE_SYSTEM
+    warm_lock = threading.Lock()
+    warmed: list[dict] = []
+
+    def ensure_warm() -> None:
+        # once per judge, and the other workers block here rather than
+        # opening their own request against a cold server
+        if warmed:
+            return
+        with warm_lock:
+            if not warmed:
+                warmed.append(warm_judge(resolved, api_key=api_key))
 
     def judge(row: dict) -> dict[str, Any]:
         use = rubric or rubric_of(row)
@@ -391,6 +416,7 @@ def rubric_judge(
             },
             default=str,
         )
+        ensure_warm()
         try:
             reply = complete(
                 url,

{whileai-0.52 → whileai-0.54}/whileai/simulations/training.py

@@ -558,6 +558,10 @@ def _measured_temperature(
     return None
 
 
+#: what GRPO does with a sampled reply the token cap cut
+TRUNCATED = ("mask", "zero")
+
+
 def train(
     dataset: str,
     *,
@@ -573,6 +577,7 @@ def train(
     max_completion_length: int | None = None,
     loss_type: str | None = None,
     temperature: float | None = None,
+    truncated: str | None = None,
     config: Mapping[str, Any] | None = None,
     wait: bool = False,
     timeout: float | None = None,
@@ -612,10 +617,15 @@ def train(
     sampling temperature the trainer rolls out at (GRPO); the dataset's
     rows say what they were measured at under ``sampling.temperature``,
     and ``train`` says so when the two differ, since a before/after
-    comparison across temperatures is not like for like. Each has a
+    comparison across temperatures is not like for like. ``truncated``
+    says what GRPO does with a sampled reply the token cap cut:
+    ``"mask"`` (the default) gives it no gradient, ``"zero"`` scores it 0
+    the old way. A cut reply scored 0 teaches shorter thinking before it
+    teaches the task, so ``"zero"`` is the knob to reach for only when the
+    cap itself is the behavior under training (#253). Each has a
     trainer default when left ``None``. ``config`` passes further host
     keys as given
-    (``epsilonHigh``, ``scaleRewards``, ``maskTruncated``, ``balance``).
+    (``epsilonHigh``, ``scaleRewards``, ``balance``).
     Every knob lands on the run's ``config`` so the run page shows it.
 
     A dataset already training answers with that run instead of a second.
@@ -682,6 +692,13 @@ def train(
         if not 0 < float(temperature) <= 2:
             raise ValueError("temperature: above 0 and at most 2")
         body["temperature"] = float(temperature)
+    if truncated is not None:
+        if method != "grpo":
+            raise ValueError("truncated= says what GRPO does with a token-capped reply; grpo only")
+        if truncated not in TRUNCATED:
+            raise ValueError(f"truncated must be one of {', '.join(TRUNCATED)}; got {truncated!r}")
+    if method == "grpo":
+        body["maskTruncated"] = (truncated or "mask") == "mask"
     for key, value in dict(config or {}).items():
         if key in body:
             raise ValueError(f"config[{key!r}] collides with a named argument")

{whileai-0.52 → whileai-0.54/whileai.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.52
+Version: 0.54
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -51,7 +51,7 @@ Two ways in, one engine. Give it the agent's tools and system prompt and it samp
 
 ![How a row gets made: the draw, the coverage grid, the search arms, the rollout, the split](docs/how-a-row-gets-made.svg)
 
-A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The interactive version, running on real rows, is at [zeroproofai.com/docs/engine](https://zeroproofai.com/docs/engine).
+A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into `Task`, `Rollout`, `Judgment`, and `Marker`, and every training target is a projection of some of those four. The engine on one page, with references: [docs/engine.md](docs/engine.md), also at [while.ai/docs/engine](https://while.ai/docs/engine).
 
 ## Overview
 
@@ -230,7 +230,7 @@ data.grade(rubric=RUBRIC)  # 2 grade against the task rubric: reward 0/1 on ever
 print(data.pass_at)
 wai.judge_trust(data.trajectories)  # 3 trust the numbers
 rows, report = wai.optimize(data, mode="rl")  # 4 prune to what carries gradient
-entry = wai.push_rows(rows, "github-rl-v1", gate=True, mode="rl")  # 5 publish, gated
+entry = wai.push_rows(rows, "my-agent-rl-v1", gate=True, mode="rl")  # 5 publish, gated
 ```
 
 `situations=200, repeats=8` is a guess. `wai.recommend(tools=TOOLS, system_prompt=POLICY, mode="rl")` replaces it with numbers from this agent's own grid: [How much to run](#how-much-to-run).
@@ -391,9 +391,9 @@ installable `verifiers` package, the shape Prime Intellect and TRL read.
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY, mode="rl", repeats=8)
 data.grade()
 # reward and world must import by name in the trainer: a module-level function or "module:attr"
-wai.export_environment(data, "envs/github-agent", reward=my_verifier)
-# pip install -e envs/github-agent
-# vf-eval github_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
+wai.export_environment(data, "envs/my-agent", reward=my_verifier)
+# pip install -e envs/my-agent
+# vf-eval my_agent -a '{"split": "holdout"}' -m <policy> -b <base url> -k <key var>
 ```
 
 The package holds `spec.json` (system prompt, the tool schemas verbatim,
@@ -681,10 +681,10 @@ login`.
 # export WHILEAI_DELEGATED_CREDENTIAL=credential["credential"]
 
 data = wai.simulate(my_agent, tools=TOOLS, system_prompt=POLICY)
-v1 = data.push("github-explore-v1")  # -> {"datasetId": "ds_...", ...}
+v1 = data.push("my-agent-explore-v1")  # -> {"datasetId": "ds_...", ...}
 
 # iterate, then push the next version with lineage
-v2 = data.push("github-explore-v2", parent=v1["datasetId"])
+v2 = data.push("my-agent-explore-v2", parent=v1["datasetId"])
 
 wai.datasets()  # list yours + storage used
 rows = wai.pull(v1["datasetId"])  # rows, or pass path= for a file