whileai 0.64__tar.gz → 0.65__tar.gz

{whileai-0.64/whileai.egg-info → whileai-0.65}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.64
+Version: 0.65
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -24,6 +24,7 @@ Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: requests>=2.25.0
+Requires-Dist: pydantic>=2.0
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0; extra == "dev"
 Requires-Dist: pytest-cov>=5.0; extra == "dev"
@@ -919,7 +920,8 @@ The platform's "Make training data" button, as one line:
 
 ```python
 wai.send_score("4bf92f3577b34da6", 1.0)  # this run passed
-wai.cuts(agent="my-agent")  # what a cut would hold
+summary = wai.cuts(agent="my-agent")  # what a cut would hold
+print(wai.format_cuts(summary, agent="my-agent"))  # the traces page's sentence
 made = wai.cut(agent="my-agent", kind="rl")  # make it
 wai.pull(made["train"]["datasetId"], "train.jsonl")
 made["holdout"]["datasetId"]  # measure on this, never train on it
@@ -1117,6 +1119,42 @@ run.holdout(before=0.42, after=0.58)  # did it work? the run page opens with thi
 
 A run's page opens with one word — **Better**, **Worse**, **About the same** — over the held-out pass rate before and after. The platform's trainer measures it; a run on your own hardware says it with `run.holdout(before, after)`, or `wai.attach_holdout(run_id, before=..., after=...)` once the run has finished. Pass rates are 0 to 1, so 58% is `0.58`; `metric="loss"` sends held-out loss instead (SFT), where lower is better. `run.delta(...)` and `wai.attach_delta(...)` already measure both sides, so they fill the two numbers in themselves, and add `summary["holdout"]` (also `run.holdout_summary`): each side's pass rate with `n_tasks`, `k` and a `ci95`, plus the delta report's verdict word (`moved`, `moved_unreplicated`, `within_eval_noise`, `no_change_detected`). A hosted run read back with `run.refresh()` has the same block with the interval fields `None` and a note that the platform only returned two numbers.
 
+### Report a run so a person can decide
+
+The platform draws one screen per tracked agent at [while.ai/platform/runs](https://while.ai/platform/runs): the held-out score by version with the frontier model as the line to beat, the training curve, what moved on the behaviors you did not train, the judge checks, live traffic on the served version, and cost. A coding agent fills it with `whileai.platform`; the person reads it and presses Promote. Your agent framework stays yours: `track` takes the agent object you already have (OpenAI Agents SDK, Pydantic AI, LangGraph, Claude Agent SDK) and reads the model, the instructions and the tools off it, or you describe it by hand.
+
+```python
+from whileai.platform import Behavior, Frontier, Harness, Judge, track
+
+tracked = track(
+    "refund-bot",  # or track(my_agent): name, model, prompt and tools come from the object
+    model="Qwen/Qwen3-4B",
+    harness=Harness(instructions=SYSTEM_PROMPT, tools=["lookup_order", "issue_refund"]),
+    frontier=Frontier(name="Sonnet 5", score=81, cost_per_1k=18.0),
+)
+tracked.behavior(
+    Behavior(
+        name="refunds",
+        test_version="v2",
+        n=240,
+        judge=Judge(agreement=0.86, human_n=60, length_bias=0.08),
+        noise_floor=2.4,
+        contamination=0,
+        reward_is_judge=False,
+    )
+)
+
+run = tracked.run("v4", method="GRPO", targets=["refunds"], trained_on=["refunds-grpo"])
+run.log(10, reward=0.41, kl=0.01)  # or trainer.add_callback(wai.TrainerCallback(run))
+run.score("refunds", 83, ci=2.7, n=240)  # every behavior, not only the targets
+run.score("length", 76, ci=2.8, n=120)
+run.finish(hours=2.1, gpu="1xH100", cost_usd=31)
+
+str(tracked.verdict())  # "refunds: v4 beats v3 by 5 (interval excludes zero); 1 regression"
+```
+
+Every object is a pydantic model that validates before it leaves the process, and each one's docstring names the rlhfbook.com chapter it comes from. A *harness* is the instructions, tools and model name around the weights; its fingerprint is its version, so a prompt edit shows up as a new version without anyone naming it (Evaluation: a score is only comparable with its setup held constant). A *behavior* has its own frozen held-out test (`test_version`), a `noise_floor` measured by scoring the same model twice, and a judge checked against people (`agreement` over `human_n`) and for `length_bias`. A *run* is scored on every behavior: `targets` are the claim, the rest are the check (Over-Optimization: verbosity, sycophancy and refusals are what moves when the reward is gamed). `ci` is the half-width of the 95% interval; the verdict is whether the candidate's and the served version's intervals overlap. `tracked.live(day, version=, replies=, flagged=)` reports a day of traffic when you serve the model yourself. Logging buffers and never raises into the training loop. Worked example: `recipes/04-train/report-run/`.
+
 ### Is it hacking the reward right now?
 
 ```python

{whileai-0.64 → whileai-0.65}/README.md

@@ -883,7 +883,8 @@ The platform's "Make training data" button, as one line:
 
 ```python
 wai.send_score("4bf92f3577b34da6", 1.0)  # this run passed
-wai.cuts(agent="my-agent")  # what a cut would hold
+summary = wai.cuts(agent="my-agent")  # what a cut would hold
+print(wai.format_cuts(summary, agent="my-agent"))  # the traces page's sentence
 made = wai.cut(agent="my-agent", kind="rl")  # make it
 wai.pull(made["train"]["datasetId"], "train.jsonl")
 made["holdout"]["datasetId"]  # measure on this, never train on it
@@ -1081,6 +1082,42 @@ run.holdout(before=0.42, after=0.58)  # did it work? the run page opens with thi
 
 A run's page opens with one word — **Better**, **Worse**, **About the same** — over the held-out pass rate before and after. The platform's trainer measures it; a run on your own hardware says it with `run.holdout(before, after)`, or `wai.attach_holdout(run_id, before=..., after=...)` once the run has finished. Pass rates are 0 to 1, so 58% is `0.58`; `metric="loss"` sends held-out loss instead (SFT), where lower is better. `run.delta(...)` and `wai.attach_delta(...)` already measure both sides, so they fill the two numbers in themselves, and add `summary["holdout"]` (also `run.holdout_summary`): each side's pass rate with `n_tasks`, `k` and a `ci95`, plus the delta report's verdict word (`moved`, `moved_unreplicated`, `within_eval_noise`, `no_change_detected`). A hosted run read back with `run.refresh()` has the same block with the interval fields `None` and a note that the platform only returned two numbers.
 
+### Report a run so a person can decide
+
+The platform draws one screen per tracked agent at [while.ai/platform/runs](https://while.ai/platform/runs): the held-out score by version with the frontier model as the line to beat, the training curve, what moved on the behaviors you did not train, the judge checks, live traffic on the served version, and cost. A coding agent fills it with `whileai.platform`; the person reads it and presses Promote. Your agent framework stays yours: `track` takes the agent object you already have (OpenAI Agents SDK, Pydantic AI, LangGraph, Claude Agent SDK) and reads the model, the instructions and the tools off it, or you describe it by hand.
+
+```python
+from whileai.platform import Behavior, Frontier, Harness, Judge, track
+
+tracked = track(
+    "refund-bot",  # or track(my_agent): name, model, prompt and tools come from the object
+    model="Qwen/Qwen3-4B",
+    harness=Harness(instructions=SYSTEM_PROMPT, tools=["lookup_order", "issue_refund"]),
+    frontier=Frontier(name="Sonnet 5", score=81, cost_per_1k=18.0),
+)
+tracked.behavior(
+    Behavior(
+        name="refunds",
+        test_version="v2",
+        n=240,
+        judge=Judge(agreement=0.86, human_n=60, length_bias=0.08),
+        noise_floor=2.4,
+        contamination=0,
+        reward_is_judge=False,
+    )
+)
+
+run = tracked.run("v4", method="GRPO", targets=["refunds"], trained_on=["refunds-grpo"])
+run.log(10, reward=0.41, kl=0.01)  # or trainer.add_callback(wai.TrainerCallback(run))
+run.score("refunds", 83, ci=2.7, n=240)  # every behavior, not only the targets
+run.score("length", 76, ci=2.8, n=120)
+run.finish(hours=2.1, gpu="1xH100", cost_usd=31)
+
+str(tracked.verdict())  # "refunds: v4 beats v3 by 5 (interval excludes zero); 1 regression"
+```
+
+Every object is a pydantic model that validates before it leaves the process, and each one's docstring names the rlhfbook.com chapter it comes from. A *harness* is the instructions, tools and model name around the weights; its fingerprint is its version, so a prompt edit shows up as a new version without anyone naming it (Evaluation: a score is only comparable with its setup held constant). A *behavior* has its own frozen held-out test (`test_version`), a `noise_floor` measured by scoring the same model twice, and a judge checked against people (`agreement` over `human_n`) and for `length_bias`. A *run* is scored on every behavior: `targets` are the claim, the rest are the check (Over-Optimization: verbosity, sycophancy and refusals are what moves when the reward is gamed). `ci` is the half-width of the 95% interval; the verdict is whether the candidate's and the served version's intervals overlap. `tracked.live(day, version=, replies=, flagged=)` reports a day of traffic when you serve the model yourself. Logging buffers and never raises into the training loop. Worked example: `recipes/04-train/report-run/`.
+
 ### Is it hacking the reward right now?
 
 ```python

{whileai-0.64 → whileai-0.65}/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "whileai"
-version = "0.64"
+version = "0.65"
 description = "While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt."
 readme = "README.md"
 license = {text = "Apache-2.0"}
 requires-python = ">=3.10"
-dependencies = ["requests>=2.25.0"]
+dependencies = ["requests>=2.25.0", "pydantic>=2.0"]
 authors = [{name = "While"}]
 keywords = [
     "zp", "zeroproof", "while", "evals", "agent-evals",

{whileai-0.64 → whileai-0.65}/whileai/ingest.py

@@ -39,6 +39,13 @@ from whileai._env import getenv
 _DEFAULT_TRACE_URL = "https://api.zeroproofai.com"
 _GZIP_MAGIC = b"\x1f\x8b"
 
+# The resource attribute that names the dataset. The gate reads
+# `zeroproof.dataset` and nothing else, so sending only the whileai spelling
+# lands every batch in a dataset called `traces` whatever you asked for, with
+# a 202 that says so too late to notice. Both are written: the second costs
+# one attribute and means the rename needs no release.
+_DATASET_KEYS = ("zeroproof.dataset", "whileai.dataset")
+
 
 class WhileIngestError(Exception):
     """Raised when the gate rejects a trace batch."""
@@ -67,7 +74,7 @@ def otel_env(api_key: str, dataset: str = "traces", base_url: str | None = None)
         "OTEL_EXPORTER_OTLP_HEADERS": "x-api-key=" + api_key,
         "OTEL_EXPORTER_OTLP_PROTOCOL": "http/json",
         # Resource attribute the gate reads to name the dataset.
-        "OTEL_RESOURCE_ATTRIBUTES": "whileai.dataset=" + dataset,
+        "OTEL_RESOURCE_ATTRIBUTES": ",".join(k + "=" + dataset for k in _DATASET_KEYS),
     }
 
 
@@ -133,9 +140,9 @@ def ingest_traces(
     Push a local OTLP batch file end to end and return ``{datasetId, dataset,
     rows}``.
 
-    ``dataset`` overrides the dataset name by setting the
-    ``whileai.dataset`` resource attribute on every resourceSpan, which
-    requires reading the batch; leave it unset to send the bytes untouched.
+    ``dataset`` overrides the dataset name by setting the dataset resource
+    attribute on every resourceSpan, which requires reading the batch; leave
+    it unset to send the bytes untouched.
     """
     with open(file, "rb") as fh:
         body = fh.read()
@@ -148,9 +155,9 @@ def ingest_traces(
         for resource_span in batch.get("resourceSpans", []):
             resource = resource_span.setdefault("resource", {})
             attributes = [
-                a for a in resource.get("attributes", []) if a.get("key") != "whileai.dataset"
+                a for a in resource.get("attributes", []) if a.get("key") not in _DATASET_KEYS
             ]
-            attributes.append({"key": "whileai.dataset", "value": {"stringValue": dataset}})
+            attributes += [{"key": k, "value": {"stringValue": dataset}} for k in _DATASET_KEYS]
             resource["attributes"] = attributes
         body = json.dumps(batch).encode("utf-8")