@basein/runner 0.2.4 → 0.2.6

package/docs/mcpmark.md

@@ -1,752 +1,752 @@
-# MCPMark — running one task against BaseInstRunner
-
-A concrete plan for one task, end to end:
-
-```
-tasks/postgres/easy/employees/hiring_year_summary/
-```
-
-Companion to [t-bench.md](t-bench.md). That document explains why τ²-bench was hard to
-integrate. This one exists because **MCPMark is structurally easier and scores our product
-more honestly**, and §1 is the reason.
-
-Source: [github.com/eval-sys/mcpmark](https://github.com/eval-sys/mcpmark) ·
-[paper](https://huggingface.co/papers/2509.24002) · [mcpmark.ai](https://mcpmark.ai/)
-
----
-
-## 1. Why this benchmark, and why this task
-
-Three structural facts, each of which τ²-bench got wrong for us.
-
-**MCPMark grades the world, not the transcript.** `verify.py` opens a psycopg2 connection
-and queries the actual database. τ²-bench's `evaluator_env.py` builds a *fresh* environment
-and replays the trajectory's `(tool_call, tool_result)` pairs onto it — which is why direct
-mode was structurally penalised there (t-bench.md §3.1). Here, direct mode executes real
-SQL through a real connection, the table really exists, and `verify.py` passes. **Direct
-mode is fully scoreable on MCPMark.** This is the single most important line in this
-document.
-
-**The agent already speaks MCP.** MCPMark spawns `postgres-mcp` over stdio
-(`src/agents/mcpmark_agent.py:1183`). τ²-bench has no MCP interface at all, which is why it
-needed `_shadow_tools`, `mcp-stdio-bridge.mjs` and two copies of the domain. Here
-`bir-proxy` just wraps the server MCPMark was going to spawn anyway. **No bridge.**
-
-**Every task runs four times against a reset state.** That is our replay shape handed over
-for free: trial 1 records, trials 2–4 replay. And the metric MCPMark leads with, `pass^4`,
-collapses for every model (best is 52.56% pass@1 → 33.86% pass^4) purely from run-to-run
-inconsistency — which is exactly what a deterministic replay removes.
-
-### Why `hiring_year_summary` specifically
-
-| | |
-|---|---|
-| Service | `postgres` — pure MCP, no built-in substitute the model would prefer |
-| Difficulty | `L1` / `easy` bucket |
-| Template DB | `employees` |
-| Writes | one `CREATE TABLE` + populate — enough to be a real CRUD test, small enough to debug |
-| Verification | ground truth computed in SQL at verify time, compared row-by-row with 0.1 tolerance on decimals |
-| Prompt | 4 short sections, ~200 words |
-
-It is the smallest task in the repo that still exercises write operations and per-step
-parameter threading. Start here, then widen.
-
----
-
-## 2. The prompt
-
-The user prompt is `description.md` **verbatim**, plus one fixed suffix appended by
-`BaseTaskManager._format_task_instruction`:
-
-```
-<description.md>
-
-Note: Based on your understanding, solve the task all at once by yourself,
-don't ask for my opinions on anything.
-```
-
-Plus MCPMark's system prompt (`src/agents/mcpmark_agent.py:48`, `MAX_TURNS = 100`):
-
-> You are a helpful agent that uses tools iteratively to complete the user's task, and when
-> finished, provides the final answer or simply states "Task completed" without further
-> tool calls.
-
-**No templating, no randomisation, no timestamps.** All four trials send byte-identical
-text. Our similarity gate sees 1.0 against a 0.92 threshold, so replay arms on trials 2–4
-every time. Nothing in this plan depends on embedding luck.
-
-Corollary: `ANTHROPIC_API_KEY` is **not needed**. `doctor` reporting
-`derive=recorded sample values` is correct behaviour here, not a degradation — for an
-identical prompt the recorded values *are* the right ones.
-
----
-
-## 3. Topology
-
-```
-claude -p  (cwd = the run dir)
- ├── mcp__bir__run_scenario ─────────────▶ bir-scenario ──┐   direct-mode delivery
- │                                                        │   (§4.4 — needs a trusted
- └── mcp__postgres__*  ──▶ bir-proxy ──▶ postgres-mcp     │    workspace, or it is denied)
-                               │            (uvx,         │
-                      reports every call     mcp<2)       │
-                               ▼              │           ▼
-                        bir-hooks :53411      │     POST /scenario/run
-                               │              │           │
-                               ▼              ▼           │
-                     BaseIn service :8080  PostgreSQL 17 ◀┘
-                                           (native, :5432)
-                                                  ▲
-                                    verify.py ────┘
-                                (grades the real database)
-```
-
-The proxy holds the open connection, and that is what makes direct mode free: on
-`/scenario/run` all eight steps execute over it for zero model tokens. `bir-scenario` is
-how the results get back into the transcript as a genuine `tool_result` rather than an
-injected denial — which is why §4.4 matters as much as it does.
-
----
-
-## 4. Setup
-
-### 4.1 Postgres, seeded
-
-MCPMark runs `pgvector/pgvector:0.8.0-pg17-bookworm` in Docker. **This machine has no
-Docker, no Docker Desktop and no WSL**, so we install PostgreSQL 17 natively instead —
-the same major version. This task is plain SQL, so the missing `pgvector` extension is
-irrelevant; note the deviation in any published result.
-
-```powershell
-winget install --id PostgreSQL.PostgreSQL.17 -e `
-  --accept-package-agreements --accept-source-agreements `
-  --custom "--mode unattended --superpassword password --serverport 5432 --unattendedmodeui none"
-```
-
-Needs one UAC approval. The superuser password is set to `password` to match MCPMark's own
-`run-task.sh` default (`POSTGRES_PASSWORD:-password`) and the `DATABASE_URI` in §4.3.
-Binaries land in `C:\Program Files\PostgreSQL\17\bin`, which is where `psql` and
-`pg_restore` come from below.
-
-Then seed the template DB — `./seed.sh` in the run directory does this:
-
-```bash
-curl -o employees.backup https://storage.mcpmark.ai/postgres/employees.backup   # 33 MB
-psql -U postgres -d postgres -c "CREATE DATABASE employees;"
-pg_restore -U postgres -d employees --no-owner --no-privileges employees.backup
-```
-
-MCPMark's own `postgres_state_manager.py::_setup_database` does exactly this for five
-templates (`employees`, `chinook`, `dvdrental`, `sports`, `lego`). We only need one.
-
-### 4.2 A per-trial database
-
-MCPMark isolates each run with `CREATE DATABASE … WITH TEMPLATE …` and drops it afterwards.
-Reproduce that — it is what makes trials independent and `pass^4` meaningful. `./reset-db.sh`:
-
-```sql
-SELECT pg_terminate_backend(pid) FROM pg_stat_activity
-  WHERE datname = 'hys_trial' AND pid <> pg_backend_pid();
-DROP DATABASE IF EXISTS hys_trial;
-CREATE DATABASE hys_trial WITH TEMPLATE employees;
-```
-
-The `pg_terminate_backend` line is not optional: `postgres-mcp` holds a pooled connection,
-and `DROP DATABASE` fails while any session is attached. `reset-db.sh` then asserts
-`hiring_year_summary` is absent and exits non-zero if it is not.
-
-**Do not skip this.** Without a reset, trial 2 finds `employees.hiring_year_summary`
-already present and the task is no longer the task.
-
-### 4.3 The run directory
-
-```bash
-mkdir -p ~/Desktop/BasIns/mcpmark-hys && cd ~/Desktop/BasIns/mcpmark-hys
-```
-
-Write `.mcp.json` **unwrapped** — `bir install` wraps it:
-
-```json
-{
-  "mcpServers": {
-    "postgres": {
-      "command": "uvx",
-      "args": ["--with", "mcp<2", "postgres-mcp==0.3.0", "--access-mode=unrestricted"],
-      "env": {
-        "DATABASE_URI": "postgresql://postgres:password@localhost:5432/hys_trial"
-      }
-    }
-  }
-}
-```
-
-`--access-mode=unrestricted` is required — the task creates a table.
-
-**`--with "mcp<2"` is mandatory, and MCPMark's own invocation is broken without it.**
-`postgres-mcp==0.3.0` imports `mcp.server.fastmcp`, which no longer exists in the `mcp`
-2.x SDK (`FastMCP` was renamed to `MCPServer`). A bare
-`pipx run postgres-mcp==0.3.0` — MCPMark's canonical command at
-`src/agents/mcpmark_agent.py:1183` — resolves `mcp` 2.x today and dies at import:
-
-```
-ModuleNotFoundError: No module named 'mcp.server.fastmcp'. This is mcp 2.x, where
-FastMCP was renamed to MCPServer … or pin 'mcp<2' to keep running v1 code.
-```
-
-This is upstream drift, not our bug, but it means any MCPMark postgres result published
-today used a pinned resolution. `uvx` is used in place of `pipx` because this machine has
-no `pipx`; both are equivalent here.
-
-Then:
-
-```bash
-bir install --replay --local
-```
-
-That wraps `postgres` with `bir-proxy`, installs the hooks into `.claude/settings.json`, and
-adds the `bir` scenario server that direct mode delivers through. Note the port it reports —
-it picks a free one (**53411** here, not the 53455 the incident demo uses), and the hook
-URLs in `settings.json` are written for that port.
-
-### 4.4 Deny the built-ins — this is the whole ballgame
-
-Claude Code will reach for `Bash` + `psql` if you let it. One built-in in the recording and
-`modeFor` ([coverage.ts:52](../src/replay/coverage.ts#L52)) returns `steer`, and we already
-measured what steer mode is worth: nothing. On Windows `PowerShell` is a separate built-in
-from `Bash` and reaches `psql` just as easily, so it must be denied too. The subagent tool is
-`Agent` in current Claude Code; `Task` is its old name, kept for older builds. Add to
-`.claude/settings.json`:
-
-```json
-{
-  "permissions": {
-    "deny": ["Bash", "PowerShell", "Read", "Write", "Edit", "NotebookEdit", "Glob", "Grep",
-             "WebFetch", "WebSearch", "Agent", "Task"],
-    "allow": ["mcp__postgres__*", "mcp__bir__*"]
-  }
-}
-```
-
-**The workspace must also be trusted, or the `allow` half is silently discarded** and
-direct mode cannot work at all. Claude Code prints it once and then carries on:
-
-```
-Ignoring 2 permissions.allow entries from .claude/settings.json:
-this workspace has not been trusted.
-```
-
-This is not cosmetic, and it fails *asymmetrically* in the worst possible way:
-
-- `mcp__postgres__*` keeps working, because bir's own `PreToolUse` answers
-  `permissionDecision: "allow"` on the correlation path ([server.ts:859](../src/control/server.ts#L859)).
-- `mcp__bir__run_scenario` is the **one** tool bir deliberately passes through
-  ([controller.ts:301](../src/replay/controller.ts#L301) → `{kind:"passthrough"}`), so it gets
-  no hook-granted allow. With the allow list gone it is denied, and in `-p` mode a denial is
-  final:
-
-  ```
-  Claude requested permissions to use mcp__bir__run_scenario,
-  but you haven't granted it yet.
-  ```
-
-The plan then arms in `direct` mode, the model cannot reach the delivery tool, it does the
-task by hand, and the first manual call triggers divergence. Everything *looks* healthy —
-the task passes and the audit log says `mode=direct` — while the saving is negative. §11 is
-the measurement.
-
-Fix before the first replay, either by running `claude` interactively in the directory once
-and accepting the trust dialog, or by setting in `~/.claude.json`:
-
-```json
-"projects": { "C:/Users/Admin/Desktop/BasIns/mcpmark-hys": { "hasTrustDialogAccepted": true } }
-```
-
-### 4.5 Start the control server, in this directory
-
-```bash
-export BIR_AUTH_URL=http://127.0.0.1:8080
-export BIR_REPLAY=1
-bir-hooks
-```
-
-Then, in another shell **in the same directory**:
-
-```bash
-bir doctor
-```
-
-Required before going further:
-
-```
-Wrapped in config : postgres
-Registered proxies: postgres        ← needs claude running
-```
-
-`Registered proxies: (none)` means no proxy has bound. Launch `claude` in this directory
-first, then re-check. **Do not send the prompt until this line is right** — that is the exact
-mistake that made the incident-demo run cost more than the original.
-
----
-
-## 5. Run protocol
-
-The run directory is `C:\Users\Admin\Desktop\BasIns\mcpmark-hys`, and it carries four
-scripts and a prompt file so a trial is one command:
-
-| Script | Does |
-|---|---|
-| `seed.sh` | one-time: create the `employees` template DB and `pg_restore` the backup |
-| `reset-db.sh` | per-trial: recreate `hys_trial` from the template, assert the target table is gone |
-| `verify.sh` | MCPMark's unmodified `verify.py` plus the env it needs |
-| `trial.sh N` | reset → `claude -p "$(cat prompt.txt)"` → verify → verdict |
-| `prompt.txt` | the assembled prompt from §2, byte-exact |
-
-`trial.sh` drives the agent with `claude -p`, non-interactive, one call per trial. That is
-the same shape the τ²-bench adapter uses and it keeps the prompt byte-identical across
-trials without a human retyping it.
-
-### Trial 1 — record
-
-```bash
-./trial.sh 1
-```
-
-`trial.sh` ends by running `./verify.sh`, which is MCPMark's unmodified `verify.py` plus the
-environment it needs:
-
-```bash
-PYTHONIOENCODING=utf-8 \
-POSTGRES_HOST=localhost POSTGRES_PORT=5432 POSTGRES_DATABASE=hys_trial \
-POSTGRES_USERNAME=postgres POSTGRES_PASSWORD=password \
-uv run --with psycopg2-binary python verify.py
-```
-
-Exit 0 is a pass.
-
-**`PYTHONIOENCODING=utf-8` is required on this machine.** `verify.py` prints `✅`, `❌` and
-`🎉`, and this console's codepage is cp1255 — without it Python raises
-`UnicodeEncodeError: 'charmap' codec can't encode character '❌'` *while formatting its
-own result*, on both the PASS and FAIL paths. A run graded through a crashed reporter is a
-run with no grade.
-
-**A failed trial 1 is a stop, not a retry.** There is no point calculating a scenario from a
-run that did not solve the task.
-
-### Calculate the scenario
-
-```bash
-bir scenario list                 # find the runId
-bir scenario calc <runId>
-bir scenario show <runId>         # read the intent, params and steps before trusting it
-```
-
-Read the step logic. Specifically check that `tool_output_logic` navigates
-`data.content[0].text` and that the steps thread through `respParams` rather than every step
-carrying a hardcoded literal. In the incident scenario every step had a baked-in fallback
-(`?? "inventory-svc"`), which is why it looked healthy while threading nothing.
-
-### Trials 2–4 — replay
-
-Reset the DB, send the identical prompt, verify. Between each, confirm:
-
-```bash
-curl -s -H "Authorization: Bearer $(...)" http://127.0.0.1:53411/health
-```
-
-Wanted: `mode: "direct"`, `stepsPinned == stepsPlanned`, `outcome: "steered_full"`.
-
-You can also drive a replay without a session at all, which is the fastest way to debug the
-step logic:
-
-```bash
-bir replay --scenario <scnId> --prompt "<the prompt>" --dry   # recorded outputs, no real SQL
-bir replay --scenario <scnId> --prompt "<the prompt>"         # real SQL through the proxy
-```
-
-`--dry` books no execution and invents no saving.
-
----
-
-## 6. What to measure
-
-| Metric | Where from | Expectation |
-|---|---|---|
-| pass@1 | `verify.py` exit code, trial 1 | unchanged by us |
-| pass^4 | all four trials pass | this is the headline |
-| `session_cost_usd` per trial | `scenario_executions` | trials 2–4 ≪ trial 1 |
-| `saved_usd` | `scenario_executions` | **must be positive** |
-| `duration_ms` vs `baseline_ms` | same table | secondary |
-| `steps_pinned` / `steps_planned` | same table | must be 8/8-shaped, not 1/8 |
-
-**Baseline honesty.** `baseline_method` is `single` and only `not_steered`/`failed` outcomes
-feed new samples, so a scenario that keeps succeeding never widens its baseline. One sample
-with ±8% run variance cannot support a claim to two significant figures. Run trial 1 several
-times with `BIR_REPLAY=0` first and take a median, or state the sample size.
-
-**`steps_pinned` is currently wrong in steer mode** —
-[controller.ts:326](../src/replay/controller.ts#L326) uses `pinned.size`, which
-[controller.ts:383](../src/replay/controller.ts#L383) has already decremented, so sequential
-pins report 1. Direct mode uses a cumulative count and is fine. If this task lands in direct
-mode the number is trustworthy; if it lands in steer mode, do not read it.
-
----
-
-## 7. The honesty section
-
-`pass^4` exists to measure whether a *model* is consistent. Raising it by not re-deciding is
-real product value and a real user benefit — but it is not a model-capability result, and
-publishing it as one would be indefensible.
-
-Report it as **cost and latency at held-constant `pass^4`**, with replay disclosed as a
-system-level intervention and trial 1 shown separately as the recording cost. That is the
-same standard t-bench.md §3.2 already holds itself to, and it is the framing that survives
-review.
-
----
-
-## 8. Known risks
-
-| Risk | Mitigation |
-|---|---|
-| Model uses `Bash`+`psql` → steer mode → zero saving | §4.4 deny list; verify `mode: "direct"` on `/health` |
-| Recording made without the proxy → output logic authored against the hook's envelope, threads `{}` | `bir doctor` must show `Registered proxies: postgres` **before** trial 1 |
-| No DB reset between trials | §4.2, every trial |
-| Windows long paths — the repo will not check out | `git clone -c core.longpaths=true` |
-| `pipx` unavailable | `uvx postgres-mcp==0.3.0`, disclosed as a deviation |
-| Scenario expiry | `scenarios.expires_at` — re-`calc` if trials span days |
-
-## 9. Next, if this works
-
-Widen within `postgres` first — 31 prompts, same server, same reset mechanism, so the
-marginal cost per task is near zero. `tasks/postgres/standard/**` is where the paper's
-numbers come from.
-
-Hold `filesystem` (40 prompts) back: the model prefers built-in `Read`/`Edit` over the
-filesystem MCP server, so those tasks fight §4.4 rather than benefiting from it.
-
----
-
-## 10. Repo facts worth keeping
-
-- Layout: `tasks/<service>/<difficulty>/<initial-state>/<task-name>/{description.md,meta.json,verify.py}`
-- 177 prompts total: **127 `standard`** (the paper's number) + 50 `easy`
-- Per service: filesystem 40 · notion 38 · github 33 · playwright_webarena 31 · postgres 31 · playwright 4 · insforge/supabase 0
-- The repo has grown past the paper's five servers — `insforge`, `supabase` and `playwright_webarena` are new
-- `meta.json` carries an `"mcp": ["postgres"]` array — use it to select only tasks that can reach direct mode
-- Small upstream bug: the §2 suffix's docstring says "Notion-specific additions" but the
-  method is on `BaseTaskManager` with no override, so it is appended for every service
-
----
-
-## 11. First run: measured results (2026-09-04)
-
-Six trials on `hiring_year_summary`. Setup as in §4, on PostgreSQL 17 native.
-
-### Grading
-
-**6/6 PASS.** `verify.py` reported `✅ 16 records correct` every time, against ground truth
-it recomputes in SQL at verify time. `pass^6` on this task.
-
-Do **not** compare that to the paper's 33.86% `pass^4`. This is an `easy`-bucket task; the
-paper's number is over the 127 `standard` tasks. It says the harness works, not that we beat
-anybody.
-
-### Cost
-
-Baseline: **$0.22401525**, 25,705 ms, `baseline_samples=1`, method `single`.
-
-| trial | outcome | cost | saved | duration | steps |
-|---|---|---|---|---|---|
-| 1 | *recorded* | $0.22402 | — | 25,705 ms | 8 |
-| 2 | `diverged` | $0.23484 | **−$0.01083** | 36,734 ms | 8/8 |
-| 3 | `diverged` | $0.24438 | **−$0.02037** | 40,768 ms | 8/8 |
-| 4 | `diverged` | $0.24277 | **−$0.01876** | 36,243 ms | 8/8 |
-| 5 | `diverged` | $0.23937 | **−$0.01536** | 43,078 ms | 8/8 |
-| 6 | `diverged` | $0.24512 | **−$0.02110** | 35,810 ms | 8/8 |
-
-Mean saving **−$0.01728 (−7.7%)**, and every replay was *slower* than the recording.
-Five for five negative — this is not variance.
-
-### What worked
-
-- **Arming is correct.** `mode=direct`, `coverage=direct,direct,direct,direct,direct,direct,direct,direct`,
-  `similarity=1.000` on every trial. This is the piece the incident demo never reached.
-- **The recording is correct.** `wrapped=["postgres"]` at run start, outputs captured in the
-  proxy's `{"content":[...]}` envelope, 8/8 tool calls `mcp__postgres__*` and zero built-ins.
-- **State genuinely threads.** Steps 2–4 read `respParams.userSchemas` / `respParams.tableNames`;
-  the audit log shows `emitted=allSchemas,userSchemas`, `emitted=schemaObjects,tableNames`,
-  `emitted=objectDetails,employeeColumns`. Not the incident scenario's hardcoded fallbacks.
-- **Derivation is free.** `replay.derived params=5 costUsd=0.0000 source="recorded samples"`.
-  No `ANTHROPIC_API_KEY` needed for an identical prompt, as §2 predicted.
-- **`steps_pinned` is right here** — 8/8 on every trial. The `pinned.size` bug in §6 is
-  specific to steer mode; the direct and divergence paths use a cumulative count.
-
-### Why the saving is negative
-
-`mcp__bir__run_scenario` was **denied by the permission system** on every trial — see §4.4.
-The workspace was untrusted, so the `allow` list was discarded, and that tool is the only one
-bir passes through rather than auto-allowing. The model could not reach the plan, so it did
-the task by hand, and its first call diverged.
-
-**Divergence recovery then performed exactly as designed** — and this is worth stating
-plainly, because it is the safety net doing its job:
-
-```
-replay.compose remaining=8 executed=8 recorded=0 skipped=0 bytes=10259 costUsd=0.00
-replay.done   mode=direct steps=8/8 outcome=diverged ms=7591
-```
-
-All eight steps ran for real through the already-open proxy connection in ~370 ms of tool
-time, nothing fell back to a recorded output, nothing was skipped, and the fallback cost was
-zero. The task passed. But the model had already paid for the wasted `run_scenario` turn,
-then paid again to read a 10,259-byte injected bundle — so the session cost landed at
-roughly the cost of just doing the task, plus overhead.
-
-**The mechanism is sound; the delivery channel was blocked.** Nothing in these six trials
-measures direct mode's actual saving, because direct delivery never once happened.
-
-### Next
-
-1. Trust the workspace (§4.4). Without it every replay measures the divergence path.
-2. Re-record after trusting — trials 2–6 were `diverged`, which books a ledger row but
-   never contributes a baseline sample, so the baseline is still a lone measurement of one
-   run at `$0.22401525` with ±8% run-to-run variance. Take a median of 3–5 `BIR_REPLAY=0`
-   runs before quoting any saving.
-3. Then re-run and expect `replay.done mode=direct outcome=steered_full` with no
-   `replay.diverge` line. That is the only configuration in which a positive saving is
-   even possible.
-
----
-
-## 12. Second run: direct mode reached, and the real blocker
-
-After trusting the workspace (§4.4) and restarting `bir-hooks` with a clean session map,
-trial 7 finally reached direct delivery:
-
-```
-tool.pre    mcp__bir__run_scenario
-replay.step n=0..7  all ok=true          09:53:18.924 → 09:53:19.289   (435 ms total)
-replay.done  mode=direct steps=8/8 outcome=steered_full ms=4236
-tool.post   mcp__bir__run_scenario failed=false
-```
-
-No `replay.diverge`. Every step executed for real over the already-open proxy connection in
-435 ms of tool time, `fallbackCostUsd=0`, `deriveCostUsd=0`. `verify.py`: PASS.
-
-### And it was the most expensive run yet
-
-| # | outcome | cost | saved | duration |
-|---|---|---|---|---|
-| 1 | *recorded* | $0.22402 | — | 25,705 ms |
-| 2 | `diverged` | $0.23484 | −$0.01083 | 36,734 ms |
-| 3 | `diverged` | $0.24438 | −$0.02037 | 40,768 ms |
-| 4 | `diverged` | $0.24277 | −$0.01876 | 36,243 ms |
-| 5 | `diverged` | $0.23937 | −$0.01536 | 43,078 ms |
-| 6 | `diverged` | $0.24512 | −$0.02110 | 35,810 ms |
-| 7 | **`steered_full`** | **$0.31426** | **−$0.09025** | 35,921 ms |
-
-7/7 PASS. Direct mode cost **+40% over baseline** — roughly five times the mean loss of the
-divergence path it replaced.
-
-### Why: the model re-did the whole task
-
-The bundle arrived as a 10,751-byte `tool_result` on turn 3. The model read it — and then
-ran this:
-
-```sql
-DROP TABLE IF EXISTS employees.hiring_year_summary;
-CREATE TABLE employees.hiring_year_summary AS SELECT ...   -- rebuilt from scratch
-SELECT column_name, data_type FROM information_schema.columns WHERE ...
-SELECT * FROM employees.hiring_year_summary ORDER BY hire_year;
-```
-
-It **dropped the table the replay had just created and built it again**. So the session paid
-for the bundle *and* for doing the task, plus a longer final answer. The plan was already
-retired by then, so nothing logged a divergence — the audit log says `steered_full` and looks
-perfect.
-
-### The cause is one sentence of prompt text
-
-[bundle.ts:54](../src/replay/bundle.ts#L54):
-
-```ts
-"[BaseInstRunner calculated replay] A known-good tool sequence was previously " +
-"recorded for this request. Below are the tool calls and their results, in order. " +
-"Use them to answer the user's request now — do NOT call any tools."
-```
-
-That header is unconditional. It says **"previously recorded"** even when all eight steps
-executed live against the real database 400 ms earlier. `anyRecorded` correctly appends a
-staleness caveat when some step *did* fall back to a recorded output — but it never says the
-opposite when nothing did.
-
-For a read-only scenario that framing is harmless: the bundle contains the answer, and the
-model just answers. For a **write** scenario it is actively counterproductive. The model is
-told side effects are historical, cannot verify the live database without calling a tool, and
-so reasonably re-runs the work. Every CRUD task in MCPMark — which is to say the whole
-benchmark, its own README calls the tasks CRUD-heavy — lands in this case.
-
-**Proposed fix:** make the header state what actually happened. When `recorded === 0`, say
-the sequence *has just been executed against the live system and its side effects are already
-in place*; keep the current wording only for entries that really are replayed from a
-recording. This is the single highest-leverage change available to the product, and it is one
-string.
-
-*Revised by §13:* the scenario underneath these runs was broken, and that is the larger
-problem. This header fix drops to fourth in §13's list.
-
-Until it lands, `direct` mode's measured saving on a write task is **negative**, and no
-amount of correct plumbing changes that — trial 7 had perfect plumbing.
-
-### Second bug, independent of the above
-
-`onScenarioRun` ([server.ts:1109](../src/control/server.ts#L1109)) resolves the run with:
-
-```ts
-const run = [...this.sessions.values()].find((s) => s.run && !s.run.finished)?.run;
-```
-
-Insertion order. A `claude -p` session's run is not sealed when the process exits, so the
-*previous* trial's session is still first in the map, its plan retired. `runArmed` hits its
-`state.retired` guard and answers `no_plan`, and the live armed plan at index 1 is never
-reached. Caught on trial 4 by polling `/health` during the turn:
-
-```
-09:42:04  n=2  1a1b140c:direct,armed=false | 705f3cc5:direct,armed=true
-                └ picked, retired            └ the actual plan, ignored
-```
-
-Workaround used here: restart `bir-hooks` before every trial. Proper fix: select the session
-whose run carries an *armed* plan, not merely the first unfinished one.
-
-### Also latent
-
-`runToCompletion` calls `assembleBundle(entries, Number.MAX_SAFE_INTEGER)`
-([plan.ts:502](../src/replay/plan.ts#L502)) — the direct-mode bundle is **uncapped**, where
-the divergence path caps at `MAX_REPLAY_REASON` (60 KB). 10,751 bytes was fine here; a
-scenario with large tool outputs would push an unbounded payload into the model's context.
-
----
-
-## 13. The scenario was broken all along
-
-Trial 8 is the most informative run of the set, because it is the first one where the model
-*trusted* the bundle. It failed:
-
-```
-replay.done  mode=direct steps=8/8 outcome=steered_full ms=3629
-execution.reported outcome=steered_full session=0.0791 savedUsd=+0.14489850  measured=true
-verify.py: ❌ relation "employees.hiring_year_summary" does not exist  → FAIL
-```
-
-**A +$0.145 saving (−65% cost) booked on a run that produced nothing.**
-
-### Root cause: a prose sample value interpolated into SQL
-
-`params_object` for `scn_9832f19f`:
-
-| parameter | sampleValue |
-|---|---|
-| `schema_name` | `"employees"` |
-| `summary_table_name` | `"hiring_year_summary"` |
-| `active_to_date` | `"9999-01-01"` |
-| `group_by_dimension` | **`"hire_year (EXTRACT(YEAR FROM hire_date))"`** |
-
-Step 5's input logic:
-
-```js
-const dim = parameters.group_by_dimension || 'hire_date';
-... "EXTRACT(YEAR FROM e." + dim + ")::int AS hire_year," ...
-```
-
-which produces:
-
-```sql
-EXTRACT(YEAR FROM e.hire_year (EXTRACT(YEAR FROM hire_date)))::int AS hire_year,
-```
-
-A syntax error. `CREATE TABLE` never runs, step 6 returns `[]`, step 7 returns
-`Error: relation "employees.hiring_year_summary" does not exist`. The SQL template itself is
-correct — run it with `dim = 'hire_date'` and it produces the right 16 rows, verified
-directly against psql.
-
-The analyser wrote a **description** where a literal tool-input value was required. The
-declared `examples` carry the same string, so nothing downstream could have caught it.
-
-### Why six trials hid it
-
-| trial | outcome | replay's SQL | model's behaviour | grade |
-|---|---|---|---|---|
-| 2–6 | `diverged` | broken | saw the errors in the bundle, did the task itself | PASS |
-| 7 | `steered_full` | broken | `DROP TABLE` + rebuilt it correctly | PASS |
-| 8 | `steered_full` | broken | trusted the bundle, only *proposed* SQL | **FAIL** |
-
-**The replay has never once produced the task's result.** Every PASS came from the model
-working around a broken scenario — which is also the real reason costs were high in §11–12,
-and it means §12's "the model re-did the task" was the *symptom*, not the disease. The
-model re-did the task because the bundle it received was full of SQL errors.
-
-### Three things made this invisible from inside bir
-
-**1. A step's verdict does not mean its work succeeded.** `executeStep` rejects only when a
-tool could not be *run*; a tool that ran and returned an error resolves normally, so
-`replay.step ... ok=true` is logged for a failed `CREATE TABLE`. That is a deliberate and
-correct distinction ([controller.ts](../src/replay/controller.ts) — "a tool that ran and
-failed resolves with its failure as the response"), but it means the audit log cannot be read
-as evidence of success.
-
-**2. Step 5's output logic hardcodes success:**
-
-```js
-return { summaryTableCreated: true, summaryTableName: ... };
-```
-
-It never inspects `toolOutput`. So `emitted=summaryTableCreated` appears in the log whether
-or not the table exists. An output logic that asserts a fact it did not check is worse than
-no output logic, because it manufactures corroboration.
-
-**3. `saved_usd` is computed from cost alone.** `steered_full` means "the plan ran to
-completion", not "the task succeeded" — and bir cannot know the benchmark grade. So the
-cheapest possible outcome (a replay that does nothing) books the largest possible saving.
-The signed saving in the BaseIn service's `src/scenarios/repo.ts` (a separate repo)
-catches replays that cost *more*; nothing catches replays that cost less by doing less.
-
-### What to fix, in order
-
-1. **Sample values must be literal tool inputs.** The analyser prompt has to forbid prose in
-   `paramsObject.sampleValue` — it is interpolated into tool arguments verbatim. A validation
-   pass that re-evaluates each `toolInputLogic` against the recorded input and diffs the
-   result would have caught this at calculate time, for free.
-2. **Judge a step by its output, not its resolution.** At minimum, detect an MCP result whose
-   text begins `Error:` and mark the step `failed` rather than `ok`. Today a scenario can
-   report `steered_full` while every step errored.
-3. **Do not book a saving for a scenario whose steps errored.** Combined with (2), a step
-   marked `failed` should force `outcome: "failed"`, which already books a baseline sample
-   instead of a saving.
-4. Then the §12 bundle-header wording, which is a real problem but a smaller one.
-
-Until (1) is fixed, re-deriving is the immediate unblock:
-
-```bash
-BIR_AUTH_URL=http://127.0.0.1:8080 node <bir>/dist/bin/bir.js scenario calc <runId> --force
-```
-
-and then **check `params_object` before trusting the result** — `bir scenario show <runId>`,
-and read every `sampleValue` as though it were about to be pasted into a shell.
-
----
-
-## 14. Status (2026-09-04): every fix in §12–13 has landed
-
-| # | Finding | Fix |
-|---|---|---|
-| §13 (1) | prose `sampleValue` spliced into SQL | BaseIn `analysis.ts`: the parameter prompt now requires literal example values, and `repairMismatchedInputLogic` re-evaluates every `toolInputLogic` over the sample values at calculate time and replays the recorded input verbatim when the result differs |
-| §13 (2) | a step whose tool returned `Error:` logged `ok=true` | [tool-error.ts](../src/replay/tool-error.ts): an MCP result with `isError` or a leading `Error` is a `failed` step at stage `tool_call`, in direct, divergence and steer mode alike |
-| §13 (3) | `steered_full` booked a saving on errored steps | [controller.ts](../src/replay/controller.ts): any errored step forces `outcome: "failed"`, which books a baseline sample |
-| §12 header | bundle said "previously recorded" for live work | [bundle.ts](../src/replay/bundle.ts): a bundle with no recorded entry says the work was just executed and its side effects are in place |
-| §12 session | `/scenario/run` picked the first unfinished session | [server.ts](../src/control/server.ts): it picks the session whose plan is armed |
-| §12 latent | direct-mode bundle uncapped | [plan.ts](../src/replay/plan.ts): `runToCompletion` caps at `MAX_REPLAY_REASON` |
-| §6 | `steps_pinned` reported 1 for sequential pins | [controller.ts](../src/replay/controller.ts): counted as each pin threads, not from `pinned.size` |
-
-Not yet re-measured. Trials 9+ need a re-`calc` (the scenario must be regenerated for the
-validation pass to run) and then the §5 protocol again from a fresh baseline.
+# MCPMark — running one task against BaseInstRunner
+
+A concrete plan for one task, end to end:
+
+```
+tasks/postgres/easy/employees/hiring_year_summary/
+```
+
+Companion to [t-bench.md](t-bench.md). That document explains why τ²-bench was hard to
+integrate. This one exists because **MCPMark is structurally easier and scores our product
+more honestly**, and §1 is the reason.
+
+Source: [github.com/eval-sys/mcpmark](https://github.com/eval-sys/mcpmark) ·
+[paper](https://huggingface.co/papers/2509.24002) · [mcpmark.ai](https://mcpmark.ai/)
+
+---
+
+## 1. Why this benchmark, and why this task
+
+Three structural facts, each of which τ²-bench got wrong for us.
+
+**MCPMark grades the world, not the transcript.** `verify.py` opens a psycopg2 connection
+and queries the actual database. τ²-bench's `evaluator_env.py` builds a *fresh* environment
+and replays the trajectory's `(tool_call, tool_result)` pairs onto it — which is why direct
+mode was structurally penalised there (t-bench.md §3.1). Here, direct mode executes real
+SQL through a real connection, the table really exists, and `verify.py` passes. **Direct
+mode is fully scoreable on MCPMark.** This is the single most important line in this
+document.
+
+**The agent already speaks MCP.** MCPMark spawns `postgres-mcp` over stdio
+(`src/agents/mcpmark_agent.py:1183`). τ²-bench has no MCP interface at all, which is why it
+needed `_shadow_tools`, `mcp-stdio-bridge.mjs` and two copies of the domain. Here
+`bir-proxy` just wraps the server MCPMark was going to spawn anyway. **No bridge.**
+
+**Every task runs four times against a reset state.** That is our replay shape handed over
+for free: trial 1 records, trials 2–4 replay. And the metric MCPMark leads with, `pass^4`,
+collapses for every model (best is 52.56% pass@1 → 33.86% pass^4) purely from run-to-run
+inconsistency — which is exactly what a deterministic replay removes.
+
+### Why `hiring_year_summary` specifically
+
+| | |
+|---|---|
+| Service | `postgres` — pure MCP, no built-in substitute the model would prefer |
+| Difficulty | `L1` / `easy` bucket |
+| Template DB | `employees` |
+| Writes | one `CREATE TABLE` + populate — enough to be a real CRUD test, small enough to debug |
+| Verification | ground truth computed in SQL at verify time, compared row-by-row with 0.1 tolerance on decimals |
+| Prompt | 4 short sections, ~200 words |
+
+It is the smallest task in the repo that still exercises write operations and per-step
+parameter threading. Start here, then widen.
+
+---
+
+## 2. The prompt
+
+The user prompt is `description.md` **verbatim**, plus one fixed suffix appended by
+`BaseTaskManager._format_task_instruction`:
+
+```
+<description.md>
+
+Note: Based on your understanding, solve the task all at once by yourself,
+don't ask for my opinions on anything.
+```
+
+Plus MCPMark's system prompt (`src/agents/mcpmark_agent.py:48`, `MAX_TURNS = 100`):
+
+> You are a helpful agent that uses tools iteratively to complete the user's task, and when
+> finished, provides the final answer or simply states "Task completed" without further
+> tool calls.
+
+**No templating, no randomisation, no timestamps.** All four trials send byte-identical
+text. Our similarity gate sees 1.0 against a 0.92 threshold, so replay arms on trials 2–4
+every time. Nothing in this plan depends on embedding luck.
+
+Corollary: `ANTHROPIC_API_KEY` is **not needed**. `doctor` reporting
+`derive=recorded sample values` is correct behaviour here, not a degradation — for an
+identical prompt the recorded values *are* the right ones.
+
+---
+
+## 3. Topology
+
+```
+claude -p  (cwd = the run dir)
+ ├── mcp__bir__run_scenario ─────────────▶ bir-scenario ──┐   direct-mode delivery
+ │                                                        │   (§4.4 — needs a trusted
+ └── mcp__postgres__*  ──▶ bir-proxy ──▶ postgres-mcp     │    workspace, or it is denied)
+                               │            (uvx,         │
+                      reports every call     mcp<2)       │
+                               ▼              │           ▼
+                        bir-hooks :53411      │     POST /scenario/run
+                               │              │           │
+                               ▼              ▼           │
+                     BaseIn service :8080  PostgreSQL 17 ◀┘
+                                           (native, :5432)
+                                                  ▲
+                                    verify.py ────┘
+                                (grades the real database)
+```
+
+The proxy holds the open connection, and that is what makes direct mode free: on
+`/scenario/run` all eight steps execute over it for zero model tokens. `bir-scenario` is
+how the results get back into the transcript as a genuine `tool_result` rather than an
+injected denial — which is why §4.4 matters as much as it does.
+
+---
+
+## 4. Setup
+
+### 4.1 Postgres, seeded
+
+MCPMark runs `pgvector/pgvector:0.8.0-pg17-bookworm` in Docker. **This machine has no
+Docker, no Docker Desktop and no WSL**, so we install PostgreSQL 17 natively instead —
+the same major version. This task is plain SQL, so the missing `pgvector` extension is
+irrelevant; note the deviation in any published result.
+
+```powershell
+winget install --id PostgreSQL.PostgreSQL.17 -e `
+  --accept-package-agreements --accept-source-agreements `
+  --custom "--mode unattended --superpassword password --serverport 5432 --unattendedmodeui none"
+```
+
+Needs one UAC approval. The superuser password is set to `password` to match MCPMark's own
+`run-task.sh` default (`POSTGRES_PASSWORD:-password`) and the `DATABASE_URI` in §4.3.
+Binaries land in `C:\Program Files\PostgreSQL\17\bin`, which is where `psql` and
+`pg_restore` come from below.
+
+Then seed the template DB — `./seed.sh` in the run directory does this:
+
+```bash
+curl -o employees.backup https://storage.mcpmark.ai/postgres/employees.backup   # 33 MB
+psql -U postgres -d postgres -c "CREATE DATABASE employees;"
+pg_restore -U postgres -d employees --no-owner --no-privileges employees.backup
+```
+
+MCPMark's own `postgres_state_manager.py::_setup_database` does exactly this for five
+templates (`employees`, `chinook`, `dvdrental`, `sports`, `lego`). We only need one.
+
+### 4.2 A per-trial database
+
+MCPMark isolates each run with `CREATE DATABASE … WITH TEMPLATE …` and drops it afterwards.
+Reproduce that — it is what makes trials independent and `pass^4` meaningful. `./reset-db.sh`:
+
+```sql
+SELECT pg_terminate_backend(pid) FROM pg_stat_activity
+  WHERE datname = 'hys_trial' AND pid <> pg_backend_pid();
+DROP DATABASE IF EXISTS hys_trial;
+CREATE DATABASE hys_trial WITH TEMPLATE employees;
+```
+
+The `pg_terminate_backend` line is not optional: `postgres-mcp` holds a pooled connection,
+and `DROP DATABASE` fails while any session is attached. `reset-db.sh` then asserts
+`hiring_year_summary` is absent and exits non-zero if it is not.
+
+**Do not skip this.** Without a reset, trial 2 finds `employees.hiring_year_summary`
+already present and the task is no longer the task.
+
+### 4.3 The run directory
+
+```bash
+mkdir -p ~/Desktop/BasIns/mcpmark-hys && cd ~/Desktop/BasIns/mcpmark-hys
+```
+
+Write `.mcp.json` **unwrapped** — `bir install` wraps it:
+
+```json
+{
+  "mcpServers": {
+    "postgres": {
+      "command": "uvx",
+      "args": ["--with", "mcp<2", "postgres-mcp==0.3.0", "--access-mode=unrestricted"],
+      "env": {
+        "DATABASE_URI": "postgresql://postgres:password@localhost:5432/hys_trial"
+      }
+    }
+  }
+}
+```
+
+`--access-mode=unrestricted` is required — the task creates a table.
+
+**`--with "mcp<2"` is mandatory, and MCPMark's own invocation is broken without it.**
+`postgres-mcp==0.3.0` imports `mcp.server.fastmcp`, which no longer exists in the `mcp`
+2.x SDK (`FastMCP` was renamed to `MCPServer`). A bare
+`pipx run postgres-mcp==0.3.0` — MCPMark's canonical command at
+`src/agents/mcpmark_agent.py:1183` — resolves `mcp` 2.x today and dies at import:
+
+```
+ModuleNotFoundError: No module named 'mcp.server.fastmcp'. This is mcp 2.x, where
+FastMCP was renamed to MCPServer … or pin 'mcp<2' to keep running v1 code.
+```
+
+This is upstream drift, not our bug, but it means any MCPMark postgres result published
+today used a pinned resolution. `uvx` is used in place of `pipx` because this machine has
+no `pipx`; both are equivalent here.
+
+Then:
+
+```bash
+bir install --replay --local
+```
+
+That wraps `postgres` with `bir-proxy`, installs the hooks into `.claude/settings.json`, and
+adds the `bir` scenario server that direct mode delivers through. Note the port it reports —
+it picks a free one (**53411** here, not the 53455 the incident demo uses), and the hook
+URLs in `settings.json` are written for that port.
+
+### 4.4 Deny the built-ins — this is the whole ballgame
+
+Claude Code will reach for `Bash` + `psql` if you let it. One built-in in the recording and
+`modeFor` ([coverage.ts:52](../src/replay/coverage.ts#L52)) returns `steer`, and we already
+measured what steer mode is worth: nothing. On Windows `PowerShell` is a separate built-in
+from `Bash` and reaches `psql` just as easily, so it must be denied too. The subagent tool is
+`Agent` in current Claude Code; `Task` is its old name, kept for older builds. Add to
+`.claude/settings.json`:
+
+```json
+{
+  "permissions": {
+    "deny": ["Bash", "PowerShell", "Read", "Write", "Edit", "NotebookEdit", "Glob", "Grep",
+             "WebFetch", "WebSearch", "Agent", "Task"],
+    "allow": ["mcp__postgres__*", "mcp__bir__*"]
+  }
+}
+```
+
+**The workspace must also be trusted, or the `allow` half is silently discarded** and
+direct mode cannot work at all. Claude Code prints it once and then carries on:
+
+```
+Ignoring 2 permissions.allow entries from .claude/settings.json:
+this workspace has not been trusted.
+```
+
+This is not cosmetic, and it fails *asymmetrically* in the worst possible way:
+
+- `mcp__postgres__*` keeps working, because bir's own `PreToolUse` answers
+  `permissionDecision: "allow"` on the correlation path ([server.ts:859](../src/control/server.ts#L859)).
+- `mcp__bir__run_scenario` is the **one** tool bir deliberately passes through
+  ([controller.ts:301](../src/replay/controller.ts#L301) → `{kind:"passthrough"}`), so it gets
+  no hook-granted allow. With the allow list gone it is denied, and in `-p` mode a denial is
+  final:
+
+  ```
+  Claude requested permissions to use mcp__bir__run_scenario,
+  but you haven't granted it yet.
+  ```
+
+The plan then arms in `direct` mode, the model cannot reach the delivery tool, it does the
+task by hand, and the first manual call triggers divergence. Everything *looks* healthy —
+the task passes and the audit log says `mode=direct` — while the saving is negative. §11 is
+the measurement.
+
+Fix before the first replay, either by running `claude` interactively in the directory once
+and accepting the trust dialog, or by setting in `~/.claude.json`:
+
+```json
+"projects": { "C:/Users/Admin/Desktop/BasIns/mcpmark-hys": { "hasTrustDialogAccepted": true } }
+```
+
+### 4.5 Start the control server, in this directory
+
+```bash
+export BIR_AUTH_URL=http://127.0.0.1:8080
+export BIR_REPLAY=1
+bir-hooks
+```
+
+Then, in another shell **in the same directory**:
+
+```bash
+bir doctor
+```
+
+Required before going further:
+
+```
+Wrapped in config : postgres
+Registered proxies: postgres        ← needs claude running
+```
+
+`Registered proxies: (none)` means no proxy has bound. Launch `claude` in this directory
+first, then re-check. **Do not send the prompt until this line is right** — that is the exact
+mistake that made the incident-demo run cost more than the original.
+
+---
+
+## 5. Run protocol
+
+The run directory is `C:\Users\Admin\Desktop\BasIns\mcpmark-hys`, and it carries four
+scripts and a prompt file so a trial is one command:
+
+| Script | Does |
+|---|---|
+| `seed.sh` | one-time: create the `employees` template DB and `pg_restore` the backup |
+| `reset-db.sh` | per-trial: recreate `hys_trial` from the template, assert the target table is gone |
+| `verify.sh` | MCPMark's unmodified `verify.py` plus the env it needs |
+| `trial.sh N` | reset → `claude -p "$(cat prompt.txt)"` → verify → verdict |
+| `prompt.txt` | the assembled prompt from §2, byte-exact |
+
+`trial.sh` drives the agent with `claude -p`, non-interactive, one call per trial. That is
+the same shape the τ²-bench adapter uses and it keeps the prompt byte-identical across
+trials without a human retyping it.
+
+### Trial 1 — record
+
+```bash
+./trial.sh 1
+```
+
+`trial.sh` ends by running `./verify.sh`, which is MCPMark's unmodified `verify.py` plus the
+environment it needs:
+
+```bash
+PYTHONIOENCODING=utf-8 \
+POSTGRES_HOST=localhost POSTGRES_PORT=5432 POSTGRES_DATABASE=hys_trial \
+POSTGRES_USERNAME=postgres POSTGRES_PASSWORD=password \
+uv run --with psycopg2-binary python verify.py
+```
+
+Exit 0 is a pass.
+
+**`PYTHONIOENCODING=utf-8` is required on this machine.** `verify.py` prints `✅`, `❌` and
+`🎉`, and this console's codepage is cp1255 — without it Python raises
+`UnicodeEncodeError: 'charmap' codec can't encode character '❌'` *while formatting its
+own result*, on both the PASS and FAIL paths. A run graded through a crashed reporter is a
+run with no grade.
+
+**A failed trial 1 is a stop, not a retry.** There is no point calculating a scenario from a
+run that did not solve the task.
+
+### Calculate the scenario
+
+```bash
+bir scenario list                 # find the runId
+bir scenario calc <runId>
+bir scenario show <runId>         # read the intent, params and steps before trusting it
+```
+
+Read the step logic. Specifically check that `tool_output_logic` navigates
+`data.content[0].text` and that the steps thread through `respParams` rather than every step
+carrying a hardcoded literal. In the incident scenario every step had a baked-in fallback
+(`?? "inventory-svc"`), which is why it looked healthy while threading nothing.
+
+### Trials 2–4 — replay
+
+Reset the DB, send the identical prompt, verify. Between each, confirm:
+
+```bash
+curl -s -H "Authorization: Bearer $(...)" http://127.0.0.1:53411/health
+```
+
+Wanted: `mode: "direct"`, `stepsPinned == stepsPlanned`, `outcome: "steered_full"`.
+
+You can also drive a replay without a session at all, which is the fastest way to debug the
+step logic:
+
+```bash
+bir replay --scenario <scnId> --prompt "<the prompt>" --dry   # recorded outputs, no real SQL
+bir replay --scenario <scnId> --prompt "<the prompt>"         # real SQL through the proxy
+```
+
+`--dry` books no execution and invents no saving.
+
+---
+
+## 6. What to measure
+
+| Metric | Where from | Expectation |
+|---|---|---|
+| pass@1 | `verify.py` exit code, trial 1 | unchanged by us |
+| pass^4 | all four trials pass | this is the headline |
+| `session_cost_usd` per trial | `scenario_executions` | trials 2–4 ≪ trial 1 |
+| `saved_usd` | `scenario_executions` | **must be positive** |
+| `duration_ms` vs `baseline_ms` | same table | secondary |
+| `steps_pinned` / `steps_planned` | same table | must be 8/8-shaped, not 1/8 |
+
+**Baseline honesty.** `baseline_method` is `single` and only `not_steered`/`failed` outcomes
+feed new samples, so a scenario that keeps succeeding never widens its baseline. One sample
+with ±8% run variance cannot support a claim to two significant figures. Run trial 1 several
+times with `BIR_REPLAY=0` first and take a median, or state the sample size.
+
+**`steps_pinned` is currently wrong in steer mode** —
+[controller.ts:326](../src/replay/controller.ts#L326) uses `pinned.size`, which
+[controller.ts:383](../src/replay/controller.ts#L383) has already decremented, so sequential
+pins report 1. Direct mode uses a cumulative count and is fine. If this task lands in direct
+mode the number is trustworthy; if it lands in steer mode, do not read it.
+
+---
+
+## 7. The honesty section
+
+`pass^4` exists to measure whether a *model* is consistent. Raising it by not re-deciding is
+real product value and a real user benefit — but it is not a model-capability result, and
+publishing it as one would be indefensible.
+
+Report it as **cost and latency at held-constant `pass^4`**, with replay disclosed as a
+system-level intervention and trial 1 shown separately as the recording cost. That is the
+same standard t-bench.md §3.2 already holds itself to, and it is the framing that survives
+review.
+
+---
+
+## 8. Known risks
+
+| Risk | Mitigation |
+|---|---|
+| Model uses `Bash`+`psql` → steer mode → zero saving | §4.4 deny list; verify `mode: "direct"` on `/health` |
+| Recording made without the proxy → output logic authored against the hook's envelope, threads `{}` | `bir doctor` must show `Registered proxies: postgres` **before** trial 1 |
+| No DB reset between trials | §4.2, every trial |
+| Windows long paths — the repo will not check out | `git clone -c core.longpaths=true` |
+| `pipx` unavailable | `uvx postgres-mcp==0.3.0`, disclosed as a deviation |
+| Scenario expiry | `scenarios.expires_at` — re-`calc` if trials span days |
+
+## 9. Next, if this works
+
+Widen within `postgres` first — 31 prompts, same server, same reset mechanism, so the
+marginal cost per task is near zero. `tasks/postgres/standard/**` is where the paper's
+numbers come from.
+
+Hold `filesystem` (40 prompts) back: the model prefers built-in `Read`/`Edit` over the
+filesystem MCP server, so those tasks fight §4.4 rather than benefiting from it.
+
+---
+
+## 10. Repo facts worth keeping
+
+- Layout: `tasks/<service>/<difficulty>/<initial-state>/<task-name>/{description.md,meta.json,verify.py}`
+- 177 prompts total: **127 `standard`** (the paper's number) + 50 `easy`
+- Per service: filesystem 40 · notion 38 · github 33 · playwright_webarena 31 · postgres 31 · playwright 4 · insforge/supabase 0
+- The repo has grown past the paper's five servers — `insforge`, `supabase` and `playwright_webarena` are new
+- `meta.json` carries an `"mcp": ["postgres"]` array — use it to select only tasks that can reach direct mode
+- Small upstream bug: the §2 suffix's docstring says "Notion-specific additions" but the
+  method is on `BaseTaskManager` with no override, so it is appended for every service
+
+---
+
+## 11. First run: measured results (2026-09-04)
+
+Six trials on `hiring_year_summary`. Setup as in §4, on PostgreSQL 17 native.
+
+### Grading
+
+**6/6 PASS.** `verify.py` reported `✅ 16 records correct` every time, against ground truth
+it recomputes in SQL at verify time. `pass^6` on this task.
+
+Do **not** compare that to the paper's 33.86% `pass^4`. This is an `easy`-bucket task; the
+paper's number is over the 127 `standard` tasks. It says the harness works, not that we beat
+anybody.
+
+### Cost
+
+Baseline: **$0.22401525**, 25,705 ms, `baseline_samples=1`, method `single`.
+
+| trial | outcome | cost | saved | duration | steps |
+|---|---|---|---|---|---|
+| 1 | *recorded* | $0.22402 | — | 25,705 ms | 8 |
+| 2 | `diverged` | $0.23484 | **−$0.01083** | 36,734 ms | 8/8 |
+| 3 | `diverged` | $0.24438 | **−$0.02037** | 40,768 ms | 8/8 |
+| 4 | `diverged` | $0.24277 | **−$0.01876** | 36,243 ms | 8/8 |
+| 5 | `diverged` | $0.23937 | **−$0.01536** | 43,078 ms | 8/8 |
+| 6 | `diverged` | $0.24512 | **−$0.02110** | 35,810 ms | 8/8 |
+
+Mean saving **−$0.01728 (−7.7%)**, and every replay was *slower* than the recording.
+Five for five negative — this is not variance.
+
+### What worked
+
+- **Arming is correct.** `mode=direct`, `coverage=direct,direct,direct,direct,direct,direct,direct,direct`,
+  `similarity=1.000` on every trial. This is the piece the incident demo never reached.
+- **The recording is correct.** `wrapped=["postgres"]` at run start, outputs captured in the
+  proxy's `{"content":[...]}` envelope, 8/8 tool calls `mcp__postgres__*` and zero built-ins.
+- **State genuinely threads.** Steps 2–4 read `respParams.userSchemas` / `respParams.tableNames`;
+  the audit log shows `emitted=allSchemas,userSchemas`, `emitted=schemaObjects,tableNames`,
+  `emitted=objectDetails,employeeColumns`. Not the incident scenario's hardcoded fallbacks.
+- **Derivation is free.** `replay.derived params=5 costUsd=0.0000 source="recorded samples"`.
+  No `ANTHROPIC_API_KEY` needed for an identical prompt, as §2 predicted.
+- **`steps_pinned` is right here** — 8/8 on every trial. The `pinned.size` bug in §6 is
+  specific to steer mode; the direct and divergence paths use a cumulative count.
+
+### Why the saving is negative
+
+`mcp__bir__run_scenario` was **denied by the permission system** on every trial — see §4.4.
+The workspace was untrusted, so the `allow` list was discarded, and that tool is the only one
+bir passes through rather than auto-allowing. The model could not reach the plan, so it did
+the task by hand, and its first call diverged.
+
+**Divergence recovery then performed exactly as designed** — and this is worth stating
+plainly, because it is the safety net doing its job:
+
+```
+replay.compose remaining=8 executed=8 recorded=0 skipped=0 bytes=10259 costUsd=0.00
+replay.done   mode=direct steps=8/8 outcome=diverged ms=7591
+```
+
+All eight steps ran for real through the already-open proxy connection in ~370 ms of tool
+time, nothing fell back to a recorded output, nothing was skipped, and the fallback cost was
+zero. The task passed. But the model had already paid for the wasted `run_scenario` turn,
+then paid again to read a 10,259-byte injected bundle — so the session cost landed at
+roughly the cost of just doing the task, plus overhead.
+
+**The mechanism is sound; the delivery channel was blocked.** Nothing in these six trials
+measures direct mode's actual saving, because direct delivery never once happened.
+
+### Next
+
+1. Trust the workspace (§4.4). Without it every replay measures the divergence path.
+2. Re-record after trusting — trials 2–6 were `diverged`, which books a ledger row but
+   never contributes a baseline sample, so the baseline is still a lone measurement of one
+   run at `$0.22401525` with ±8% run-to-run variance. Take a median of 3–5 `BIR_REPLAY=0`
+   runs before quoting any saving.
+3. Then re-run and expect `replay.done mode=direct outcome=steered_full` with no
+   `replay.diverge` line. That is the only configuration in which a positive saving is
+   even possible.
+
+---
+
+## 12. Second run: direct mode reached, and the real blocker
+
+After trusting the workspace (§4.4) and restarting `bir-hooks` with a clean session map,
+trial 7 finally reached direct delivery:
+
+```
+tool.pre    mcp__bir__run_scenario
+replay.step n=0..7  all ok=true          09:53:18.924 → 09:53:19.289   (435 ms total)
+replay.done  mode=direct steps=8/8 outcome=steered_full ms=4236
+tool.post   mcp__bir__run_scenario failed=false
+```
+
+No `replay.diverge`. Every step executed for real over the already-open proxy connection in
+435 ms of tool time, `fallbackCostUsd=0`, `deriveCostUsd=0`. `verify.py`: PASS.
+
+### And it was the most expensive run yet
+
+| # | outcome | cost | saved | duration |
+|---|---|---|---|---|
+| 1 | *recorded* | $0.22402 | — | 25,705 ms |
+| 2 | `diverged` | $0.23484 | −$0.01083 | 36,734 ms |
+| 3 | `diverged` | $0.24438 | −$0.02037 | 40,768 ms |
+| 4 | `diverged` | $0.24277 | −$0.01876 | 36,243 ms |
+| 5 | `diverged` | $0.23937 | −$0.01536 | 43,078 ms |
+| 6 | `diverged` | $0.24512 | −$0.02110 | 35,810 ms |
+| 7 | **`steered_full`** | **$0.31426** | **−$0.09025** | 35,921 ms |
+
+7/7 PASS. Direct mode cost **+40% over baseline** — roughly five times the mean loss of the
+divergence path it replaced.
+
+### Why: the model re-did the whole task
+
+The bundle arrived as a 10,751-byte `tool_result` on turn 3. The model read it — and then
+ran this:
+
+```sql
+DROP TABLE IF EXISTS employees.hiring_year_summary;
+CREATE TABLE employees.hiring_year_summary AS SELECT ...   -- rebuilt from scratch
+SELECT column_name, data_type FROM information_schema.columns WHERE ...
+SELECT * FROM employees.hiring_year_summary ORDER BY hire_year;
+```
+
+It **dropped the table the replay had just created and built it again**. So the session paid
+for the bundle *and* for doing the task, plus a longer final answer. The plan was already
+retired by then, so nothing logged a divergence — the audit log says `steered_full` and looks
+perfect.
+
+### The cause is one sentence of prompt text
+
+[bundle.ts:54](../src/replay/bundle.ts#L54):
+
+```ts
+"[BaseInstRunner calculated replay] A known-good tool sequence was previously " +
+"recorded for this request. Below are the tool calls and their results, in order. " +
+"Use them to answer the user's request now — do NOT call any tools."
+```
+
+That header is unconditional. It says **"previously recorded"** even when all eight steps
+executed live against the real database 400 ms earlier. `anyRecorded` correctly appends a
+staleness caveat when some step *did* fall back to a recorded output — but it never says the
+opposite when nothing did.
+
+For a read-only scenario that framing is harmless: the bundle contains the answer, and the
+model just answers. For a **write** scenario it is actively counterproductive. The model is
+told side effects are historical, cannot verify the live database without calling a tool, and
+so reasonably re-runs the work. Every CRUD task in MCPMark — which is to say the whole
+benchmark, its own README calls the tasks CRUD-heavy — lands in this case.
+
+**Proposed fix:** make the header state what actually happened. When `recorded === 0`, say
+the sequence *has just been executed against the live system and its side effects are already
+in place*; keep the current wording only for entries that really are replayed from a
+recording. This is the single highest-leverage change available to the product, and it is one
+string.
+
+*Revised by §13:* the scenario underneath these runs was broken, and that is the larger
+problem. This header fix drops to fourth in §13's list.
+
+Until it lands, `direct` mode's measured saving on a write task is **negative**, and no
+amount of correct plumbing changes that — trial 7 had perfect plumbing.
+
+### Second bug, independent of the above
+
+`onScenarioRun` ([server.ts:1109](../src/control/server.ts#L1109)) resolves the run with:
+
+```ts
+const run = [...this.sessions.values()].find((s) => s.run && !s.run.finished)?.run;
+```
+
+Insertion order. A `claude -p` session's run is not sealed when the process exits, so the
+*previous* trial's session is still first in the map, its plan retired. `runArmed` hits its
+`state.retired` guard and answers `no_plan`, and the live armed plan at index 1 is never
+reached. Caught on trial 4 by polling `/health` during the turn:
+
+```
+09:42:04  n=2  1a1b140c:direct,armed=false | 705f3cc5:direct,armed=true
+                └ picked, retired            └ the actual plan, ignored
+```
+
+Workaround used here: restart `bir-hooks` before every trial. Proper fix: select the session
+whose run carries an *armed* plan, not merely the first unfinished one.
+
+### Also latent
+
+`runToCompletion` calls `assembleBundle(entries, Number.MAX_SAFE_INTEGER)`
+([plan.ts:502](../src/replay/plan.ts#L502)) — the direct-mode bundle is **uncapped**, where
+the divergence path caps at `MAX_REPLAY_REASON` (60 KB). 10,751 bytes was fine here; a
+scenario with large tool outputs would push an unbounded payload into the model's context.
+
+---
+
+## 13. The scenario was broken all along
+
+Trial 8 is the most informative run of the set, because it is the first one where the model
+*trusted* the bundle. It failed:
+
+```
+replay.done  mode=direct steps=8/8 outcome=steered_full ms=3629
+execution.reported outcome=steered_full session=0.0791 savedUsd=+0.14489850  measured=true
+verify.py: ❌ relation "employees.hiring_year_summary" does not exist  → FAIL
+```
+
+**A +$0.145 saving (−65% cost) booked on a run that produced nothing.**
+
+### Root cause: a prose sample value interpolated into SQL
+
+`params_object` for `scn_9832f19f`:
+
+| parameter | sampleValue |
+|---|---|
+| `schema_name` | `"employees"` |
+| `summary_table_name` | `"hiring_year_summary"` |
+| `active_to_date` | `"9999-01-01"` |
+| `group_by_dimension` | **`"hire_year (EXTRACT(YEAR FROM hire_date))"`** |
+
+Step 5's input logic:
+
+```js
+const dim = parameters.group_by_dimension || 'hire_date';
+... "EXTRACT(YEAR FROM e." + dim + ")::int AS hire_year," ...
+```
+
+which produces:
+
+```sql
+EXTRACT(YEAR FROM e.hire_year (EXTRACT(YEAR FROM hire_date)))::int AS hire_year,
+```
+
+A syntax error. `CREATE TABLE` never runs, step 6 returns `[]`, step 7 returns
+`Error: relation "employees.hiring_year_summary" does not exist`. The SQL template itself is
+correct — run it with `dim = 'hire_date'` and it produces the right 16 rows, verified
+directly against psql.
+
+The analyser wrote a **description** where a literal tool-input value was required. The
+declared `examples` carry the same string, so nothing downstream could have caught it.
+
+### Why six trials hid it
+
+| trial | outcome | replay's SQL | model's behaviour | grade |
+|---|---|---|---|---|
+| 2–6 | `diverged` | broken | saw the errors in the bundle, did the task itself | PASS |
+| 7 | `steered_full` | broken | `DROP TABLE` + rebuilt it correctly | PASS |
+| 8 | `steered_full` | broken | trusted the bundle, only *proposed* SQL | **FAIL** |
+
+**The replay has never once produced the task's result.** Every PASS came from the model
+working around a broken scenario — which is also the real reason costs were high in §11–12,
+and it means §12's "the model re-did the task" was the *symptom*, not the disease. The
+model re-did the task because the bundle it received was full of SQL errors.
+
+### Three things made this invisible from inside bir
+
+**1. A step's verdict does not mean its work succeeded.** `executeStep` rejects only when a
+tool could not be *run*; a tool that ran and returned an error resolves normally, so
+`replay.step ... ok=true` is logged for a failed `CREATE TABLE`. That is a deliberate and
+correct distinction ([controller.ts](../src/replay/controller.ts) — "a tool that ran and
+failed resolves with its failure as the response"), but it means the audit log cannot be read
+as evidence of success.
+
+**2. Step 5's output logic hardcodes success:**
+
+```js
+return { summaryTableCreated: true, summaryTableName: ... };
+```
+
+It never inspects `toolOutput`. So `emitted=summaryTableCreated` appears in the log whether
+or not the table exists. An output logic that asserts a fact it did not check is worse than
+no output logic, because it manufactures corroboration.
+
+**3. `saved_usd` is computed from cost alone.** `steered_full` means "the plan ran to
+completion", not "the task succeeded" — and bir cannot know the benchmark grade. So the
+cheapest possible outcome (a replay that does nothing) books the largest possible saving.
+The signed saving in the BaseIn service's `src/scenarios/repo.ts` (a separate repo)
+catches replays that cost *more*; nothing catches replays that cost less by doing less.
+
+### What to fix, in order
+
+1. **Sample values must be literal tool inputs.** The analyser prompt has to forbid prose in
+   `paramsObject.sampleValue` — it is interpolated into tool arguments verbatim. A validation
+   pass that re-evaluates each `toolInputLogic` against the recorded input and diffs the
+   result would have caught this at calculate time, for free.
+2. **Judge a step by its output, not its resolution.** At minimum, detect an MCP result whose
+   text begins `Error:` and mark the step `failed` rather than `ok`. Today a scenario can
+   report `steered_full` while every step errored.
+3. **Do not book a saving for a scenario whose steps errored.** Combined with (2), a step
+   marked `failed` should force `outcome: "failed"`, which already books a baseline sample
+   instead of a saving.
+4. Then the §12 bundle-header wording, which is a real problem but a smaller one.
+
+Until (1) is fixed, re-deriving is the immediate unblock:
+
+```bash
+BIR_AUTH_URL=http://127.0.0.1:8080 node <bir>/dist/bin/bir.js scenario calc <runId> --force
+```
+
+and then **check `params_object` before trusting the result** — `bir scenario show <runId>`,
+and read every `sampleValue` as though it were about to be pasted into a shell.
+
+---
+
+## 14. Status (2026-09-04): every fix in §12–13 has landed
+
+| # | Finding | Fix |
+|---|---|---|
+| §13 (1) | prose `sampleValue` spliced into SQL | BaseIn `analysis.ts`: the parameter prompt now requires literal example values, and `repairMismatchedInputLogic` re-evaluates every `toolInputLogic` over the sample values at calculate time and replays the recorded input verbatim when the result differs |
+| §13 (2) | a step whose tool returned `Error:` logged `ok=true` | [tool-error.ts](../src/replay/tool-error.ts): an MCP result with `isError: true` is a `failed` step at stage `tool_call`, in direct, divergence and steer mode alike. Since 2026-09-19 a leading `Error` text with `isError: false` no longer counts: replay judges a step by the same rule the recorder does |
+| §13 (3) | `steered_full` booked a saving on errored steps | [controller.ts](../src/replay/controller.ts): any errored step forces `outcome: "failed"`, which books a baseline sample |
+| §12 header | bundle said "previously recorded" for live work | [bundle.ts](../src/replay/bundle.ts): a bundle with no recorded entry says the work was just executed and its side effects are in place |
+| §12 session | `/scenario/run` picked the first unfinished session | [server.ts](../src/control/server.ts): it picks the session whose plan is armed |
+| §12 latent | direct-mode bundle uncapped | [plan.ts](../src/replay/plan.ts): `runToCompletion` caps at `MAX_REPLAY_REASON` |
+| §6 | `steps_pinned` reported 1 for sequential pins | [controller.ts](../src/replay/controller.ts): counted as each pin threads, not from `pinned.size` |
+
+Not yet re-measured. Trials 9+ need a re-`calc` (the scenario must be regenerated for the
+validation pass to run) and then the §5 protocol again from a fresh baseline.