@basein/runner 0.2.11 → 0.3.0

package/docs/t-bench.md

@@ -1,394 +1,406 @@
-# τ²-bench — running it against BaseInstRunner, with Claude Code as the agent
-
-Companion to [benchmark.md](benchmark.md), which recommends MCPMark first and τ²-bench
-second. This is the second half: what τ²-bench actually is, how to read a trajectory,
-and the three structural facts that decide whether the integration measures our product
-or measures nothing at all.
-
-**Read §3 before writing any code.** τ²-bench executes tool calls *inside its own
-orchestrator*. Nothing crosses MCP by default, which means a naive integration produces a
-clean run, a plausible score, and zero recorded steps.
-
----
-
-## 1. Why this benchmark, in one paragraph
-
-MCPMark proves a replayed scenario survives a **fresh instance of the same task**.
-τ²-bench proves it survives a **different task of the same shape** — "cancel order W1234
-for user A" against "cancel order W5678 for user B". Its domains are built out of
-policy-constrained task families over a shared database, which is exactly the surface that
-per-step parameter derivation exists for. Nothing else public tests the *calculated* half
-of calculated replay this directly. It also carries the name recognition that makes a
-result travel.
-
-Domains: `retail` (115 tasks), `airline` (50), `telecom` (114), plus `mock` and
-`banking_knowledge`. Python ≥3.12 <3.14, `uv sync`, LiteLLM-compatible keys in `.env`.
-
----
-
-## 2. Reading a trajectory
-
-The link that prompted this doc:
-
-```
-https://taubench.com/trajectory-visualizer?model=qwen3.5-397b-a17b-think_sierra_2026-03-02&domain=retail&task=0
-```
-
-It is a client-side viewer over the simulation JSON that `tau2 run` writes to
-`data/simulations/`. Two modes — **Trajectories** and **Tasks** — and the URL pins one
-`(model, domain, task)` triple. The same data is browsable locally with `tau2 view`, which
-is the version to use while building the adapter, because it reads your own runs.
-
-An episode is a four-party loop, and the visualizer shows it as one interleaved column:
-
-```
-Agent ──message──▶ User ──message──▶ Agent ──tool_call──▶ Environment ──result──▶ Agent
-```
-
-| Element | What it is | What we care about |
-|---|---|---|
-| **UserMessage** | the *simulated* customer, driven by `--user-llm` | turn 1 carries the intent; later turns supply parameters |
-| **AssistantMessage** | the agent's text, and optionally `tool_calls` | the text is what the COMMUNICATE check reads |
-| **ToolMessage** | the environment's result | executed by the orchestrator, not the agent (§3.1) |
-| **evaluation_criteria.actions** | *one* reference solution, replayed on a fresh env to derive the target DB hash | **not** a required path (§3.2) |
-| reward breakdown | DB / COMMUNICATE / ENV_ASSERTION / NL_ASSERTION | the product of whatever `reward_basis` lists |
-
-Episodes end on a stop token, a user-simulator "task complete" transfer, or `max_steps`.
-
-**The one habit worth forming:** when a task scores 0, open the trajectory and check
-*which* component was 0. A DB failure and a COMMUNICATE failure look identical in the
-aggregate and mean opposite things for us — see §3.3.
-
----
-
-## 3. The three facts that decide the integration
-
-### 3.1 The grade comes from the transcript, not from the world
-
-*Corrected after the first working run. An earlier version of this section said the
-binding constraint was that the orchestrator executes tool calls. That is true — the agent
-returns an `AssistantMessage` containing `tool_calls` and the orchestrator invokes them
-against the domain environment — but it is not the constraint that decides the design.*
-
-The constraint that decides the design is in `evaluator_env.py`: τ² **does not grade the
-environment the episode ran in**. It constructs a *fresh* environment, replays the
-trajectory's `(tool_call, tool_result)` pairs onto it via `set_state`, and hashes that.
-
-> **Consequence:** the reward is earned by what appears in the messages. An agent that
-> executes the whole task perfectly against the live environment and reports it in prose
-> scores **zero**, because the graded environment received nothing.
-
-We measured exactly this. Claude Code solved retail task 0 with a call sequence matching
-the reference trajectory *exactly* — same tools, same item ids, same payment method — and
-scored `DB 0`, `reward 0`. Nothing in the results table said "your harness is broken"; it
-read like a hard task.
-
-τ²-bench also ships no MCP interface: domain tools are Python callables wrapped in `Tool`
-objects and called in-process. So bir sees nothing unless Claude Code makes real MCP calls
-of its own. Those two facts together — the trajectory must carry the calls, *and* Claude
-Code must make them over MCP — are what §4 has to satisfy at once.
-
-### 3.2 Reward is outcome-based, so replay is legitimately scoreable
-
-`evaluation_criteria.actions` is a reference trajectory, replayed on a fresh environment to
-establish a **target database hash**. The DB check compares hashes. Any sequence of tool
-calls reaching an equivalent end state passes. The docs are emphatic: *actions document one
-working solution, not the only acceptable one.*
-
-`ACTION` — exact trajectory match — is a reward component that exists but is **not** in the
-default `reward_basis` for airline, retail or telecom, which is `["DB", "COMMUNICATE"]`.
-
-This is the fact that makes the whole exercise honest. A replayed scenario takes the path
-*a previous run* took, which will not be token-identical to the reference actions. Under an
-action-matching benchmark that would be scored as deviation. Under τ²-bench it is scored as
-what it is: the same outcome, reached differently. We are not exploiting a loophole — we
-are using the benchmark's stated philosophy.
-
-**Check `reward_basis` per task before running.** If a domain or task set includes `ACTION`,
-exclude it and say so, rather than discovering it in the aggregate.
-
-### 3.3 Reward is a *product*, and COMMUNICATE is in the basis
-
-`reward = db_reward × communicate_reward`. `communicate_info` is a list of strings the
-agent must have said to the user, matched as substrings.
-
-Now read that against what direct-mode replay does. From the README: on a scenario whose
-every step is a wrapped MCP tool, the sequence runs through connections the proxies already
-hold — **zero model tokens** — and the model's only job is one tool call that reads the
-results.
-
-> A replay that executes every step perfectly and never *tells the customer the refund
-> amount* scores `1 × 0 = 0`.
-
-The design already accounts for this: `steered_full` means every planned step ran with
-scenario-derived inputs **and the response model was produced**. So the machinery should be
-right. But this is the single highest-value assertion to verify early, because its failure
-signature — every task at exactly 0.0 with a healthy step ledger — is easy to misread as a
-plumbing bug and waste a day on.
-
-τ-bench is, in this specific sense, a *harder* test of replay than MCPMark: MCPMark grades
-the world, τ-bench grades the world **and** what the agent said about it.
-
----
-
-## 4. Architecture — two environments
-
-Both constraints in §3.1 have to hold at once: Claude Code must make **real MCP calls**
-(or bir records nothing), and the **trajectory must carry those calls** (or the grade is
-zero). One environment cannot satisfy both — if Claude Code executes a call and the
-adapter also hands it to the orchestrator, the orchestrator executes it a second time,
-the exchange fails as already-exchanged, and the trajectory records an error the evaluator
-cannot replay.
-
-Two copies of the domain resolve it. Each executes every call exactly once.
-
-```
-tau2 orchestrator ── graded environment ◀── executes the calls the adapter emits
-  ├── user simulator (--user-llm)              (this is what the trajectory records)
-  └── ClaudeCodeAgent
-        ├── ToolBridge ── private copy of the domain ◀── Claude Code acts here
-        │      (in-process HTTP, ephemeral port, one per episode)
-        └── claude -p / --resume, one call per orchestrator turn
-              └── mcp__tau2__* ─▶ bir-proxy ─▶ node mcp-stdio-bridge.mjs ─▶ ToolBridge
-```
-
-The copies start identical — taken after `_initialize_environment` has applied the task's
-setup — and replay the same calls in the same order, so they stay in step. Claude Code's
-results are real; the orchestrator's execution is what gets graded.
-
-### 4.1 Taking the copy
-
-`build_agent` hands the agent `environment.get_tools()`: live `Tool` objects whose `_func`
-is a bound method of the domain's toolkit, which owns the db. Deep-copying **the toolkit**
-yields an independent domain with its own database:
-
-```python
-toolkit = self.tools[0]._func.__self__      # RetailTools, etc.
-shadow = copy.deepcopy(toolkit)
-shadow_tools = list(shadow.get_tools().values())
-```
-
-Copying the toolkit rather than rebuilding the domain from the registry preserves any
-task-specific initialization that would otherwise have to be reconstructed from the task
-object. Verify independence rather than assuming it: write through the copy and confirm
-`environment.get_db_hash()` does not move. `selftest_bridge.py` is that check.
-
-### 4.2 One Claude Code turn becomes N+1 messages
-
-τ²'s validator (`validate_message_format_default`) rejects a message carrying both text and
-tool calls. So a turn in which Claude Code makes four calls and then replies is *five*
-messages, each separated by the orchestrator executing one call. The adapter queues them
-and hands over one per `generate_next_message`; the tool results the orchestrator returns
-are ignored, because Claude Code already saw the equivalent result from its own copy and
-the run is over by then. Their purpose is the trajectory.
-
-> **Stamp each message when you hand it over, not when you build it.** τ² sorts the final
-> trajectory by timestamp. A queue built in one burst carries timestamps *older* than the
-> tool results answering it, so the sort files every assistant message ahead of every tool
-> message and the evaluator refuses to replay it. It is intermittent — when the clock does
-> not tick between batches the timestamps tie and the stable sort happens to be right — so
-> it presents as roughly one run in three dying with an infrastructure error.
-
-Two smaller things in the same area:
-
-- **Give every assistant message a `cost`, even zero.** `get_cost` discards the whole
-  conversation's cost if any single one is `None`, so the tool-call messages carry `0.0`
-  and the reply carries the run's total.
-- **Translate the token counts.** τ² sums `prompt_tokens` / `completion_tokens`; Claude
-  Code reports `input_tokens`, `output_tokens` and two cache figures. Cached reads are real
-  prompt tokens and are most of the prompt on a resumed session, so they belong in the
-  total or every turn after the first looks nearly free.
-
-### 4.3 Passing the policy
-
-The agent constructor receives `domain_policy` as a string. It goes into the session as
-`--append-system-prompt` (or `--append-system-prompt-file` — retail's `policy.md` is long
-enough that a file is the better shape). Do not paste it into the per-turn prompt: it is
-constant across the episode, and repeating it per turn inflates every cost number we are
-trying to measure.
-
-### 4.4 Per-turn invocation
-
-```bash
-# turn 1
-claude -p "$USER_MESSAGE" \
-  --append-system-prompt-file "$POLICY_MD" \
-  --settings "$BIR_SETTINGS_JSON" \
-  --mcp-config "$EPISODE_MCP_JSON" \
-  --output-format json \
-  --permission-mode dontAsk \
-  --allowedTools "mcp__tau2__*" \
-  --max-turns 30
-
-# turns 2..n
-claude -p "$USER_MESSAGE" --resume "$SESSION_ID" --output-format json ...
-```
-
-`--allowedTools "mcp__tau2__*"` and nothing else. The agent has no business touching the
-filesystem or shell in a customer-service episode, and an allow list that says so turns a
-whole class of confusing failures into a clean permission denial.
-
-Check `system/init`'s `mcp_server_errors` on turn 1 of every episode. A `--mcp-config`
-entry that fails validation is skipped *silently*; the episode then runs with no tools and
-scores 0, which reads exactly like a hard task.
-
----
-
-## 5. BIR wiring
-
-```bash
-export BIR_AUTH_URL=https://your-basein-service
-export BIR_CONTROL_URL=http://127.0.0.1:53411      # not discovery
-export BIR_CONTROL_TOKEN=...
-export BIR_REPLAY_ALLOW_SERVERS=tau2
-BIR_REPLAY=1 bir-hooks 2>&1 | tee -a results/tau2-audit.log
-```
-
-`BIR_CONTROL_URL` / `BIR_CONTROL_TOKEN` are mandatory, not convenient: the README calls
-them the only reliable channel for an SDK session, and every episode turn is one. Discovery
-across a few hundred harness-spawned processes is how an arm silently degrades to Tier 2.
-
-`BIR_REPLAY_ALLOW_SERVERS=tau2` is the whole allowance. Direct execution dispatches steps on
-connections that are already open and never reaches the permission system, so the list is
-the blast radius.
-
-### 5.1 The per-prompt run boundary — the structural mismatch
-
-This is the one that will shape the results, and it is worth understanding before the first
-run rather than after.
-
-`onPrompt` rolls a **new run on every prompt**. In τ²-bench, every user-simulator turn is a
-prompt. So an episode of six turns is **six runs**, not one, and matching happens per turn
-against a library of per-turn scenarios.
-
-What follows from that:
-
-- **Turn 1 carries the intent** — "I want to cancel my order" — and is where a match should
-  fire. Later turns are mostly parameter supply ("it's W5678") and are short, cheap, and
-  poor matching material.
-- **A scenario is a turn's tool chain, not an episode's.** Cross-episode reuse therefore
-  happens turn-by-turn, and the savings ledger sums over turns.
-- **Per-run usage watermarking matters more here than anywhere.** Six runs share one
-  Claude Code transcript; summing the file gives turn 6 the cost of turns 1–6. The design
-  doc's `markTranscriptUsage` / `usageSince` delta is not an optimisation in this setting,
-  it is the difference between a real number and a fabricated one. Check `measured: true`.
-
-If the results show matching that fires reliably on turn 1 and rarely after, that is not a
-bug — it is the honest shape of the product against a multi-turn benchmark, and it should
-be reported that way.
-
----
-
-## 6. Running it
-
-```bash
-git clone https://github.com/sierra-research/tau2-bench && cd tau2-bench
-uv sync && cp .env.example .env      # user-simulator key goes here
-
-# smoke: one task, one trial
-tau2 run --domain retail --agent claude_code --user-llm <model> \
-  --num-trials 1 --task-ids 0
-
-tau2 view                            # read the trajectory you just produced
-
-# baseline arm, recording only (BIR_REPLAY unset)
-tau2 run --domain retail --agent claude_code --user-llm <model> \
-  --num-trials 4 --max-concurrency 4
-
-# replay arm, library seeded from the passing runs above
-BIR_REPLAY=1 tau2 run --domain retail --agent claude_code --user-llm <model> \
-  --num-trials 4 --max-concurrency 4
-```
-
-Pin the `--user-llm` model and version across all arms. It is the environment; changing it
-changes the task, and a shifted baseline invalidates every comparison.
-
----
-
-## 7. Verification gates
-
-Run these on `--task-ids 0`, in order. Each has a distinct failure signature, and they are
-ordered so that the cheapest and most misleading failures are caught first.
-
-| # | Gate | Failure looks like | Means |
-|---|---|---|---|
-| 1 | `mcp_servers` in `system/init` lists `tau2` as connected | score 0, no tool calls | config skipped silently |
-| 2 | `bir doctor` green, `metadata.tier == 1` inside a harness-spawned run | recording exists but thin | Tier 2 — hooks not loading |
-| 3 | The proxy recorded the episode's tool calls | empty step stream on a passing task | traffic bypassed bir |
-| 4 | Reward > 0 on a task the reference solves | *every* task exactly 0.0 | the trajectory carries no tool calls (§3.1) |
-| 5 | No duplicated side effects | reward 0, doubled refund in the env | one environment, executed twice (§4) |
-| 6 | Zero retries across a multi-trial run | ~1 run in 3 dies as "infrastructure error" | the timestamp race (§4.2) |
-| 7 | `Avg Cost/Conversation` is a number, not `n/a` | costs silently absent | an assistant message with `cost=None` |
-| 8 | On the replay arm: DB=1 **and** COMMUNICATE=1 | reward 0, step ledger healthy | **§3.3** — replay produced no response text |
-
-**Every one of these fails quietly.** That is the point of the list. Three of them we hit
-for real, and none announced itself:
-
-- **Gate 4** produced a clean table reading `reward 0.0000, DB match 0/1` on a task the
-  agent had solved *perfectly* — its calls matched the reference exactly. It reads like a
-  hard task, not a broken harness.
-- **Gate 6** was absorbed by τ²'s own retry logic, so the run "succeeded" while costing 3×
-  and mixing infrastructure errors into what would look like task variance across a sweep.
-- A fourth, not a gate but worth knowing: on Windows, npm's `claude.cmd` shim routes
-  arguments through `cmd.exe`, which re-parses them. Any user turn containing `&`, `|`,
-  `%`, `^` or a newline arrives truncated, and the agent answers *"your message came
-  through empty."* Call `claude.exe` directly and refuse to fall back to the shim.
-
-The lesson generalises past this benchmark: a harness bug and a hard task produce the same
-number. Read one transcript per configuration before trusting any aggregate.
-
----
-
-## 8. What to measure
-
-Per arm, per domain: reward (mean), pass^k across trials, `$`/episode and `$`/turn from
-`--output-format json`'s `total_cost_usd`, wall-clock per episode, and turn count.
-
-Per replayed turn, from the audit log and the execution ledger: outcome
-(`steered_full` / `diverged` / `not_steered` / `failed`), match similarity against
-`BIR_MIN_STEER_SIMILARITY` (default 0.92), direct vs steered mode, `deriveCostUsd` against
-`sessionCostUsd`, and the per-step verdicts.
-
-Two τ-specific splits worth reporting that MCPMark cannot give us:
-
-- **Reward decomposed.** DB and COMMUNICATE separately, always. An arm where DB holds and
-  COMMUNICATE slips is a precise, fixable finding about what replay omits; the product hides
-  it completely.
-- **Match rate by turn index.** Turn 1 versus turns 2..n (§5.1). This is the number that
-  says whether calculated replay generalises across a *task family* or only across
-  restatements of one task, and it is the question τ-bench was chosen to answer.
-
-Then the control from [benchmark.md](benchmark.md) §7 Phase 3, which matters more here than
-anywhere: τ-bench task families differ by *entity*, so a matcher that ignores which order
-id the customer named will replay cheerfully against the wrong order. Build the near-miss
-set by swapping entities between same-shape tasks — same words, different customer — and
-report the **false replay rate**. In a retail domain, that number has an obvious real-world
-reading, which is exactly why it should be published whatever it says.
-
----
-
-## 9. Known limits
-
-- **The MCP wrapper is ours.** τ²-bench has no MCP interface, so the transport under test is
-  a shim we wrote. A sceptic can question it; the answer is to publish it and to run gate 3
-  and gate 5 in public. MCPMark has no such caveat, which is the other reason it is primary.
-- **The user simulator is an LLM**, so the environment is stochastic across trials. pass^k
-  here mixes agent variance with simulator variance. Pin the simulator model and report it
-  as a limit rather than pretending the variance is all ours.
-- **Six runs per episode** (§5.1) makes every per-run number a per-turn number. Say so in
-  the table headers; a `$`/run figure that a reader takes for `$`/task is off by ~6×.
-- **`banking_knowledge` stays out.** Same reasoning as [benchmark.md](benchmark.md) §5 — it
-  is retrieval over a document set, not a repeatable tool chain, and there is nothing there
-  for a scenario to be.
-- **Voice mode is out of scope.** Full-duplex needs `FullDuplexAgent` and a realtime
-  provider; Claude Code is half-duplex and so is this integration.
-
----
-
-## Sources
-
-- τ²-bench — [repo](https://github.com/sierra-research/tau2-bench) · [agent developer guide](https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/agent/README.md) · [orchestrator guide](https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/orchestrator/README.md) · [evaluation docs](https://github.com/sierra-research/tau2-bench/blob/main/docs/evaluation.md)
-- Papers — [τ²-bench (arXiv:2506.07982)](https://arxiv.org/pdf/2506.07982) · [τ-bench (arXiv:2406.12045)](https://arxiv.org/pdf/2406.12045)
-- [Trajectory visualizer](https://taubench.com/trajectory-visualizer/?model=qwen3.5-397b-a17b-think_sierra_2026-03-02&domain=retail&task=0) · [leaderboard](https://taubench.com/leaderboard/)
-- [Claude Code headless mode](https://code.claude.com/docs/en/headless)
-- This repo — [benchmark.md](benchmark.md) · [calculatedReplay.md](calculatedReplay.md) · [calculatedReplayGuide.md](calculatedReplayGuide.md)
+# τ²-bench — running it against BaseInstRunner, with Claude Code as the agent
+
+Companion to [benchmark.md](benchmark.md), which recommends MCPMark first and τ²-bench
+second. This is the second half: what τ²-bench actually is, how to read a trajectory,
+and the three structural facts that decide whether the integration measures our product
+or measures nothing at all.
+
+**Read §3 before writing any code.** τ²-bench executes tool calls *inside its own
+orchestrator*. Nothing crosses MCP by default, which means a naive integration produces a
+clean run, a plausible score, and zero recorded steps.
+
+---
+
+## 1. Why this benchmark, in one paragraph
+
+MCPMark proves a replayed scenario survives a **fresh instance of the same task**.
+τ²-bench proves it survives a **different task of the same shape** — "cancel order W1234
+for user A" against "cancel order W5678 for user B". Its domains are built out of
+policy-constrained task families over a shared database, which is exactly the surface that
+per-step parameter derivation exists for. Nothing else public tests the *calculated* half
+of calculated replay this directly. It also carries the name recognition that makes a
+result travel.
+
+Domains: `retail` (115 tasks), `airline` (50), `telecom` (114), plus `mock` and
+`banking_knowledge`. Python ≥3.12 <3.14, `uv sync`, LiteLLM-compatible keys in `.env`.
+
+---
+
+## 2. Reading a trajectory
+
+The link that prompted this doc:
+
+```
+https://taubench.com/trajectory-visualizer?model=qwen3.5-397b-a17b-think_sierra_2026-03-02&domain=retail&task=0
+```
+
+It is a client-side viewer over the simulation JSON that `tau2 run` writes to
+`data/simulations/`. Two modes — **Trajectories** and **Tasks** — and the URL pins one
+`(model, domain, task)` triple. The same data is browsable locally with `tau2 view`, which
+is the version to use while building the adapter, because it reads your own runs.
+
+An episode is a four-party loop, and the visualizer shows it as one interleaved column:
+
+```
+Agent ──message──▶ User ──message──▶ Agent ──tool_call──▶ Environment ──result──▶ Agent
+```
+
+| Element | What it is | What we care about |
+|---|---|---|
+| **UserMessage** | the *simulated* customer, driven by `--user-llm` | turn 1 carries the intent; later turns supply parameters |
+| **AssistantMessage** | the agent's text, and optionally `tool_calls` | the text is what the COMMUNICATE check reads |
+| **ToolMessage** | the environment's result | executed by the orchestrator, not the agent (§3.1) |
+| **evaluation_criteria.actions** | *one* reference solution, replayed on a fresh env to derive the target DB hash | **not** a required path (§3.2) |
+| reward breakdown | DB / COMMUNICATE / ENV_ASSERTION / NL_ASSERTION | the product of whatever `reward_basis` lists |
+
+Episodes end on a stop token, a user-simulator "task complete" transfer, or `max_steps`.
+
+**The one habit worth forming:** when a task scores 0, open the trajectory and check
+*which* component was 0. A DB failure and a COMMUNICATE failure look identical in the
+aggregate and mean opposite things for us — see §3.3.
+
+---
+
+## 3. The three facts that decide the integration
+
+### 3.1 The grade comes from the transcript, not from the world
+
+*Corrected after the first working run. An earlier version of this section said the
+binding constraint was that the orchestrator executes tool calls. That is true — the agent
+returns an `AssistantMessage` containing `tool_calls` and the orchestrator invokes them
+against the domain environment — but it is not the constraint that decides the design.*
+
+The constraint that decides the design is in `evaluator_env.py`: τ² **does not grade the
+environment the episode ran in**. It constructs a *fresh* environment, replays the
+trajectory's `(tool_call, tool_result)` pairs onto it via `set_state`, and hashes that.
+
+> **Consequence:** the reward is earned by what appears in the messages. An agent that
+> executes the whole task perfectly against the live environment and reports it in prose
+> scores **zero**, because the graded environment received nothing.
+
+We measured exactly this. Claude Code solved retail task 0 with a call sequence matching
+the reference trajectory *exactly* — same tools, same item ids, same payment method — and
+scored `DB 0`, `reward 0`. Nothing in the results table said "your harness is broken"; it
+read like a hard task.
+
+τ²-bench also ships no MCP interface: domain tools are Python callables wrapped in `Tool`
+objects and called in-process. So bir sees nothing unless Claude Code makes real MCP calls
+of its own. Those two facts together — the trajectory must carry the calls, *and* Claude
+Code must make them over MCP — are what §4 has to satisfy at once.
+
+### 3.2 Reward is outcome-based, so replay is legitimately scoreable
+
+`evaluation_criteria.actions` is a reference trajectory, replayed on a fresh environment to
+establish a **target database hash**. The DB check compares hashes. Any sequence of tool
+calls reaching an equivalent end state passes. The docs are emphatic: *actions document one
+working solution, not the only acceptable one.*
+
+`ACTION` — exact trajectory match — is a reward component that exists but is **not** in the
+default `reward_basis` for airline, retail or telecom, which is `["DB", "COMMUNICATE"]`.
+
+This is the fact that makes the whole exercise honest. A replayed scenario takes the path
+*a previous run* took, which will not be token-identical to the reference actions. Under an
+action-matching benchmark that would be scored as deviation. Under τ²-bench it is scored as
+what it is: the same outcome, reached differently. We are not exploiting a loophole — we
+are using the benchmark's stated philosophy.
+
+**Check `reward_basis` per task before running.** If a domain or task set includes `ACTION`,
+exclude it and say so, rather than discovering it in the aggregate.
+
+### 3.3 Reward is a *product*, and COMMUNICATE is in the basis
+
+`reward = db_reward × communicate_reward`. `communicate_info` is a list of strings the
+agent must have said to the user, matched as substrings.
+
+Now read that against what direct-mode replay does. From the README: on a scenario whose
+every step is a wrapped MCP tool, the sequence runs through connections the proxies already
+hold — **zero model tokens** — and the model's only job is one tool call that reads the
+results.
+
+> A replay that executes every step perfectly and never *tells the customer the refund
+> amount* scores `1 × 0 = 0`.
+
+The design already accounts for this: `steered_full` means every planned step ran with
+scenario-derived inputs **and the response model was produced**. So the machinery should be
+right. But this is the single highest-value assertion to verify early, because its failure
+signature — every task at exactly 0.0 with a healthy step ledger — is easy to misread as a
+plumbing bug and waste a day on.
+
+τ-bench is, in this specific sense, a *harder* test of replay than MCPMark: MCPMark grades
+the world, τ-bench grades the world **and** what the agent said about it.
+
+---
+
+## 4. Architecture — two environments
+
+Both constraints in §3.1 have to hold at once: Claude Code must make **real MCP calls**
+(or bir records nothing), and the **trajectory must carry those calls** (or the grade is
+zero). One environment cannot satisfy both — if Claude Code executes a call and the
+adapter also hands it to the orchestrator, the orchestrator executes it a second time,
+the exchange fails as already-exchanged, and the trajectory records an error the evaluator
+cannot replay.
+
+Two copies of the domain resolve it. Each executes every call exactly once.
+
+```
+tau2 orchestrator ── graded environment ◀── executes the calls the adapter emits
+  ├── user simulator (--user-llm)              (this is what the trajectory records)
+  └── ClaudeCodeAgent
+        ├── ToolBridge ── private copy of the domain ◀── Claude Code acts here
+        │      (in-process HTTP, ephemeral port, one per episode)
+        └── claude -p / --resume, one call per orchestrator turn
+              └── mcp__tau2__* ─▶ bir-proxy ─▶ node mcp-stdio-bridge.mjs ─▶ ToolBridge
+```
+
+The copies start identical — taken after `_initialize_environment` has applied the task's
+setup — and replay the same calls in the same order, so they stay in step. Claude Code's
+results are real; the orchestrator's execution is what gets graded.
+
+### 4.1 Taking the copy
+
+`build_agent` hands the agent `environment.get_tools()`: live `Tool` objects whose `_func`
+is a bound method of the domain's toolkit, which owns the db. Deep-copying **the toolkit**
+yields an independent domain with its own database:
+
+```python
+toolkit = self.tools[0]._func.__self__      # RetailTools, etc.
+shadow = copy.deepcopy(toolkit)
+shadow_tools = list(shadow.get_tools().values())
+```
+
+Copying the toolkit rather than rebuilding the domain from the registry preserves any
+task-specific initialization that would otherwise have to be reconstructed from the task
+object. Verify independence rather than assuming it: write through the copy and confirm
+`environment.get_db_hash()` does not move. `selftest_bridge.py` is that check.
+
+### 4.2 One Claude Code turn becomes N+1 messages
+
+τ²'s validator (`validate_message_format_default`) rejects a message carrying both text and
+tool calls. So a turn in which Claude Code makes four calls and then replies is *five*
+messages, each separated by the orchestrator executing one call. The adapter queues them
+and hands over one per `generate_next_message`; the tool results the orchestrator returns
+are ignored, because Claude Code already saw the equivalent result from its own copy and
+the run is over by then. Their purpose is the trajectory.
+
+> **Stamp each message when you hand it over, not when you build it.** τ² sorts the final
+> trajectory by timestamp. A queue built in one burst carries timestamps *older* than the
+> tool results answering it, so the sort files every assistant message ahead of every tool
+> message and the evaluator refuses to replay it. It is intermittent — when the clock does
+> not tick between batches the timestamps tie and the stable sort happens to be right — so
+> it presents as roughly one run in three dying with an infrastructure error.
+
+Two smaller things in the same area:
+
+- **Give every assistant message a `cost`, even zero.** `get_cost` discards the whole
+  conversation's cost if any single one is `None`, so the tool-call messages carry `0.0`
+  and the reply carries the run's total.
+- **Translate the token counts.** τ² sums `prompt_tokens` / `completion_tokens`; Claude
+  Code reports `input_tokens`, `output_tokens` and two cache figures. Cached reads are real
+  prompt tokens and are most of the prompt on a resumed session, so they belong in the
+  total or every turn after the first looks nearly free.
+
+### 4.3 Passing the policy
+
+The agent constructor receives `domain_policy` as a string. It goes into the session as
+`--append-system-prompt` (or `--append-system-prompt-file` — retail's `policy.md` is long
+enough that a file is the better shape). Do not paste it into the per-turn prompt: it is
+constant across the episode, and repeating it per turn inflates every cost number we are
+trying to measure.
+
+### 4.4 Per-turn invocation
+
+```bash
+# turn 1
+claude -p "$USER_MESSAGE" \
+  --append-system-prompt-file "$POLICY_MD" \
+  --settings "$BIR_SETTINGS_JSON" \
+  --mcp-config "$EPISODE_MCP_JSON" \
+  --output-format json \
+  --permission-mode dontAsk \
+  --allowedTools "mcp__tau2__*" \
+  --max-turns 30
+
+# turns 2..n
+claude -p "$USER_MESSAGE" --resume "$SESSION_ID" --output-format json ...
+```
+
+`--allowedTools "mcp__tau2__*"` and nothing else. The agent has no business touching the
+filesystem or shell in a customer-service episode, and an allow list that says so turns a
+whole class of confusing failures into a clean permission denial.
+
+Check `system/init`'s `mcp_server_errors` on turn 1 of every episode. A `--mcp-config`
+entry that fails validation is skipped *silently*; the episode then runs with no tools and
+scores 0, which reads exactly like a hard task.
+
+---
+
+## 5. BIR wiring
+
+```bash
+export BIR_AUTH_URL=https://your-basein-service
+export BIR_CONTROL_URL=http://127.0.0.1:53411      # not discovery
+export BIR_CONTROL_TOKEN=...
+export BIR_REPLAY_ALLOW_SERVERS=tau2
+BIR_REPLAY=1 bir-hooks 2>&1 | tee -a results/tau2-audit.log
+```
+
+`BIR_CONTROL_URL` / `BIR_CONTROL_TOKEN` are mandatory, not convenient: the README calls
+them the only reliable channel for an SDK session, and every episode turn is one. Discovery
+across a few hundred harness-spawned processes is how an arm silently degrades to Tier 2.
+
+`BIR_REPLAY_ALLOW_SERVERS=tau2` is the whole allowance. Direct execution dispatches steps on
+connections that are already open and never reaches the permission system, so the list is
+the blast radius.
+
+### 5.1 The per-prompt run boundary — the structural mismatch
+
+This is the one that will shape the results, and it is worth understanding before the first
+run rather than after.
+
+`onPrompt` rolls a **new run on every prompt**. In τ²-bench, every user-simulator turn is a
+prompt. So an episode of six turns is **six runs**, not one, and matching happens per turn
+against a library of per-turn scenarios.
+
+What follows from that:
+
+- **Turn 1 carries the intent** — "I want to cancel my order" — and is where a match should
+  fire. Later turns are mostly parameter supply ("it's W5678") and are short, cheap, and
+  poor matching material.
+- **A scenario is a turn's tool chain, not an episode's.** Cross-episode reuse therefore
+  happens turn-by-turn, and the savings ledger sums over turns.
+- **Per-run usage watermarking matters more here than anywhere.** Six runs share one
+  Claude Code transcript; summing the file gives turn 6 the cost of turns 1–6. The design
+  doc's `markTranscriptUsage` / `usageSince` delta is not an optimisation in this setting,
+  it is the difference between a real number and a fabricated one. Check `measured: true`.
+
+If the results show matching that fires reliably on turn 1 and rarely after, that is not a
+bug — it is the honest shape of the product against a multi-turn benchmark, and it should
+be reported that way.
+
+---
+
+## 6. Running it
+
+```bash
+git clone https://github.com/sierra-research/tau2-bench && cd tau2-bench
+uv sync && cp .env.example .env      # user-simulator key goes here
+
+# smoke: one task, one trial
+tau2 run --domain retail --agent claude_code --user-llm <model> \
+  --num-trials 1 --task-ids 0
+
+tau2 view                            # read the trajectory you just produced
+
+# baseline arm, recording only (BIR_REPLAY unset)
+tau2 run --domain retail --agent claude_code --user-llm <model> \
+  --num-trials 4 --max-concurrency 4
+
+# replay arm, library seeded from the passing runs above
+BIR_REPLAY=1 tau2 run --domain retail --agent claude_code --user-llm <model> \
+  --num-trials 4 --max-concurrency 4
+```
+
+On the replay arm, `--max-concurrency 4` means several Claude Code sessions are live at
+once, and every `claude -p` turn starts a `tau2` proxy of its own. A step bir runs itself
+for a session — a direct replay, or the rest of a plan after a divergence — goes only to a
+proxy known to be that session's (one of its calls was reported and joined to the
+session), or to one not yet tied to any session while no other session is live. Otherwise
+the step is not run (recorded as `skipped`) and the plan hands the turn to the agent
+(a direct replay books `failed`, or `fell_back` once a step ran), rather than act on another
+episode's copy of the domain. A direct replay starts with `run_scenario`, before
+the turn's proxy has reported anything, so while other episodes run it hands over: measure
+direct mode with `--max-concurrency 1`, and read hand-overs under concurrency as the
+harness, not the scenario.
+
+Pin the `--user-llm` model and version across all arms. It is the environment; changing it
+changes the task, and a shifted baseline invalidates every comparison.
+
+---
+
+## 7. Verification gates
+
+Run these on `--task-ids 0`, in order. Each has a distinct failure signature, and they are
+ordered so that the cheapest and most misleading failures are caught first.
+
+| # | Gate | Failure looks like | Means |
+|---|---|---|---|
+| 1 | `mcp_servers` in `system/init` lists `tau2` as connected | score 0, no tool calls | config skipped silently |
+| 2 | `bir doctor` green, `metadata.tier == 1` inside a harness-spawned run | recording exists but thin | Tier 2 — hooks not loading |
+| 3 | The proxy recorded the episode's tool calls | empty step stream on a passing task | traffic bypassed bir |
+| 4 | Reward > 0 on a task the reference solves | *every* task exactly 0.0 | the trajectory carries no tool calls (§3.1) |
+| 5 | No duplicated side effects | reward 0, doubled refund in the env | one environment, executed twice (§4) |
+| 6 | Zero retries across a multi-trial run | ~1 run in 3 dies as "infrastructure error" | the timestamp race (§4.2) |
+| 7 | `Avg Cost/Conversation` is a number, not `n/a` | costs silently absent | an assistant message with `cost=None` |
+| 8 | On the replay arm: DB=1 **and** COMMUNICATE=1 | reward 0, step ledger healthy | **§3.3** — replay produced no response text |
+
+**Every one of these fails quietly.** That is the point of the list. Three of them we hit
+for real, and none announced itself:
+
+- **Gate 4** produced a clean table reading `reward 0.0000, DB match 0/1` on a task the
+  agent had solved *perfectly* — its calls matched the reference exactly. It reads like a
+  hard task, not a broken harness.
+- **Gate 6** was absorbed by τ²'s own retry logic, so the run "succeeded" while costing 3×
+  and mixing infrastructure errors into what would look like task variance across a sweep.
+- A fourth, not a gate but worth knowing: on Windows, npm's `claude.cmd` shim routes
+  arguments through `cmd.exe`, which re-parses them. Any user turn containing `&`, `|`,
+  `%`, `^` or a newline arrives truncated, and the agent answers *"your message came
+  through empty."* Call `claude.exe` directly and refuse to fall back to the shim.
+
+The lesson generalises past this benchmark: a harness bug and a hard task produce the same
+number. Read one transcript per configuration before trusting any aggregate.
+
+---
+
+## 8. What to measure
+
+Per arm, per domain: reward (mean), pass^k across trials, `$`/episode and `$`/turn from
+`--output-format json`'s `total_cost_usd`, wall-clock per episode, and turn count.
+
+Per replayed turn, from the audit log and the execution ledger: outcome
+(`steered_full` / `diverged` / `fell_back` / `not_steered` / `failed`), match similarity against
+`BIR_MIN_STEER_SIMILARITY` (default 0.92), direct vs steered mode, `deriveCostUsd` against
+`sessionCostUsd`, and the per-step verdicts.
+
+Two τ-specific splits worth reporting that MCPMark cannot give us:
+
+- **Reward decomposed.** DB and COMMUNICATE separately, always. An arm where DB holds and
+  COMMUNICATE slips is a precise, fixable finding about what replay omits; the product hides
+  it completely.
+- **Match rate by turn index.** Turn 1 versus turns 2..n (§5.1). This is the number that
+  says whether calculated replay generalises across a *task family* or only across
+  restatements of one task, and it is the question τ-bench was chosen to answer.
+
+Then the control from [benchmark.md](benchmark.md) §7 Phase 3, which matters more here than
+anywhere: τ-bench task families differ by *entity*, so a matcher that ignores which order
+id the customer named will replay cheerfully against the wrong order. Build the near-miss
+set by swapping entities between same-shape tasks — same words, different customer — and
+report the **false replay rate**. In a retail domain, that number has an obvious real-world
+reading, which is exactly why it should be published whatever it says.
+
+---
+
+## 9. Known limits
+
+- **The MCP wrapper is ours.** τ²-bench has no MCP interface, so the transport under test is
+  a shim we wrote. A sceptic can question it; the answer is to publish it and to run gate 3
+  and gate 5 in public. MCPMark has no such caveat, which is the other reason it is primary.
+- **The user simulator is an LLM**, so the environment is stochastic across trials. pass^k
+  here mixes agent variance with simulator variance. Pin the simulator model and report it
+  as a limit rather than pretending the variance is all ours.
+- **Six runs per episode** (§5.1) makes every per-run number a per-turn number. Say so in
+  the table headers; a `$`/run figure that a reader takes for `$`/task is off by ~6×.
+- **`banking_knowledge` stays out.** Same reasoning as [benchmark.md](benchmark.md) §5 — it
+  is retrieval over a document set, not a repeatable tool chain, and there is nothing there
+  for a scenario to be.
+- **Voice mode is out of scope.** Full-duplex needs `FullDuplexAgent` and a realtime
+  provider; Claude Code is half-duplex and so is this integration.
+
+---
+
+## Sources
+
+- τ²-bench — [repo](https://github.com/sierra-research/tau2-bench) · [agent developer guide](https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/agent/README.md) · [orchestrator guide](https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/orchestrator/README.md) · [evaluation docs](https://github.com/sierra-research/tau2-bench/blob/main/docs/evaluation.md)
+- Papers — [τ²-bench (arXiv:2506.07982)](https://arxiv.org/pdf/2506.07982) · [τ-bench (arXiv:2406.12045)](https://arxiv.org/pdf/2406.12045)
+- [Trajectory visualizer](https://taubench.com/trajectory-visualizer/?model=qwen3.5-397b-a17b-think_sierra_2026-03-02&domain=retail&task=0) · [leaderboard](https://taubench.com/leaderboard/)
+- [Claude Code headless mode](https://code.claude.com/docs/en/headless)
+- This repo — [benchmark.md](benchmark.md) · [calculatedReplay.md](calculatedReplay.md) · [calculatedReplayGuide.md](calculatedReplayGuide.md)