@basein/runner 0.2.11 → 0.3.0

package/docs/my-first-sample.md

@@ -1,545 +1,569 @@
-# Your first sample — run one MCPMark task and watch it pay for itself
-
-A complete, from-zero manual. By the end you will have run one real agent
-benchmark task, graded it against a live database, and then watched
-BaseInstRunner replay the *same* task for **zero model tokens** — and you will
-see the dollar figure that saved.
-
-This is the hands-on companion to [mcpmark.md](mcpmark.md), which explains the
-*why* in depth. This document is the *what to type*, in order, with the numbers
-a real run produced on 2026-09-07 so you know what "working" looks like.
-
-> **The one task.** Everything below uses a single MCPMark task,
-> `postgres/easy/employees/hiring_year_summary`. It is the smallest task in the
-> benchmark that still writes to the database, which makes it the honest place
-> to start.
-
----
-
-## 0. What MCPMark is (and why we use it)
-
-**MCPMark is a standard agents benchmark.** It measures how well an AI agent can
-drive real tools, through the [Model Context Protocol
-(MCP)](https://modelcontextprotocol.io), to change real systems — not to chat
-about them.
-
-The facts worth knowing before you start:
-
-| | |
-|---|---|
-| Who | The `eval-sys` research group · [github.com/eval-sys/mcpmark](https://github.com/eval-sys/mcpmark) · [paper (arXiv 2509.24002)](https://arxiv.org/abs/2509.24002) · [mcpmark.ai](https://mcpmark.ai) |
-| What it tests | An agent using **five MCP services**: Postgres, GitHub, Filesystem, Notion, Playwright |
-| Size | **127 standard tasks + 50 easy** (the repo has grown past the paper's five services) |
-| How it grades | Each task ships a `verify.py` that opens a real connection and **queries the actual state**, not the agent's transcript. Exit 0 = PASS |
-| Metrics | `pass@1` (one try), `pass@k`, and `pass^k` — *stability*: did it pass **all** k independent runs |
-| State of the art | On MCPMark's verified set, the strongest model reports about **92.9%**; most models sit far lower, and `pass^4` collapses for everyone |
-
-Two of those facts are why this benchmark suits BaseInstRunner so well:
-
-1. **It grades the world, not the words.** A replayed run that produces the
-   right rows passes exactly like a hand-typed one. There is no transcript to
-   fool.
-2. **`pass^k` punishes inconsistency.** The benchmark runs each task several
-   times against a reset database and asks whether *every* run passed. A model
-   that re-decides the plan token-by-token every time is exactly what
-   BaseInstRunner removes — it runs the *same* known-good plan each time.
-
-That is the whole idea you are about to see working: **solve the task once, then
-stop paying to re-solve it.**
-
----
-
-## 1. What you are about to build
-
-```
-  Part A — the benchmark, on its own
-    seed a Postgres DB  →  agent solves the task  →  verify.py grades it  →  PASS
-
-  Parts B–D — the same task, through BaseInstRunner
-    claude -p ──mcp__postgres__*──▶ bir-proxy ──▶ postgres-mcp ──▶ PostgreSQL
-        │                              │  records every call
-        └──mcp__bir__run_scenario──▶ bir-hooks (control server)
-                                       │  matches the prompt, arms the plan,
-                                       ▼  runs all 6 steps for $0 model tokens
-                                  BaseIn service  ──▶  books the saving
-```
-
-You will run the task **three ways**, and the difference between them is the
-point:
-
-| Run | Runner | What the model does | What it costs |
-|---|---|---|---|
-| **Baseline** | off | discovers and solves the task | full price |
-| **Record** | on, replay **off** | solves it once; bir *records* the plan | full price, one time |
-| **Replay** | on, replay **on** | one tool call reads a pre-computed result | a fraction |
-
----
-
-## 2. Prerequisites
-
-Check each one. The whole setup takes about ten minutes, most of it downloads.
-
-| # | You need | Check | If missing |
-|---|---|---|---|
-| 1 | **Node 20+** | `node -v` | [nodejs.org](https://nodejs.org) |
-| 2 | **Claude Code** | `claude --version` | Anthropic's CLI; this manual drives it with `claude -p` |
-| 3 | **PostgreSQL 17**, running | `psql --version` and the service is up | see §2.1 |
-| 4 | **uv / uvx** | `uvx --version` | [astral.sh/uv](https://docs.astral.sh/uv/) — runs `postgres-mcp` and `verify.py` |
-| 5 | **The runner** | `bir --version` | `npm i -g @basein/runner` |
-| 6 | **A BaseIn account + API URL** | `bir login` succeeds | your service address, e.g. `https://api.your-domain.com` |
-
-> **`BIR_AUTH_URL` must be the API host**, normally `https://api.<domain>`, not
-> the docs or app website. Point it at a website and `bir login` refuses it,
-> saying what answered instead. Set it once, then open a **fresh** terminal — an
-> existing one keeps the value it started with.
-
-`bir login` signs you in through the browser: it prints a link and a short code
-and waits for you to approve it. Nothing is typed into the terminal, and it works
-for accounts that only sign in with Google.
-
-### 2.1 Postgres, without Docker
-
-MCPMark's own harness runs Postgres in Docker (`pgvector/pgvector:pg17`). If the
-machine has no Docker — this one did not — install PostgreSQL 17 **natively**;
-it is the same major version, and this task needs no vector support. Disclose
-the deviation in any published result.
-
-```powershell
-winget install --id PostgreSQL.PostgreSQL.17 -e `
-  --accept-package-agreements --accept-source-agreements `
-  --custom "--mode unattended --superpassword password --serverport 5432 --unattendedmodeui none"
-```
-
-That sets the `postgres` superuser password to `password`, which is the value
-every command and connection string below expects. On macOS:
-`brew install postgresql@17 && brew services start postgresql@17`.
-
----
-
-## 3. Part A — run the benchmark task on its own
-
-First prove the plain benchmark works, with no runner in the picture at all.
-
-### 3.1 Make a run folder and fetch the task
-
-```bash
-mkdir -p ~/my-first-sample && cd ~/my-first-sample
-
-TASK=https://raw.githubusercontent.com/eval-sys/mcpmark/main/tasks/postgres/easy/employees/hiring_year_summary
-curl -fsSLO "$TASK/description.md"   # what the agent is asked to do
-curl -fsSLO "$TASK/verify.py"        # MCPMark's grader — unmodified
-curl -fsSLO "$TASK/meta.json"        # task metadata (difficulty L1, mcp: postgres)
-```
-
-The task asks the agent to build one table, `employees.hiring_year_summary`,
-with a row per hiring year: how many were hired, how many are still employed,
-and the retention rate. Small, but a genuine write.
-
-### 3.2 The exact prompt
-
-MCPMark sends `description.md` **verbatim** plus one fixed suffix. Assemble it
-once so every run is byte-identical:
-
-```bash
-{ cat description.md
-  printf '\n\nNote: Based on your understanding, solve the task all at once by yourself, don'\''t ask for my opinions on anything.\n'
-} > prompt.txt
-```
-
-### 3.3 Seed the database
-
-Download MCPMark's 33 MB backup and build the `employees` **template**
-database from it. Save this as `seed.sh` and run it once:
-
-```bash
-#!/usr/bin/env bash
-set -euo pipefail
-PGBIN="${PGBIN:-/c/Program Files/PostgreSQL/17/bin}"   # macOS: "$(brew --prefix postgresql@17)/bin"
-export PGPASSWORD="${POSTGRES_PASSWORD:-password}"
-PGUSER=postgres PGHOST=localhost PGPORT=5432
-
-[ -f employees.backup ] || curl -fsSL -o employees.backup https://storage.mcpmark.ai/postgres/employees.backup
-
-"$PGBIN/psql" -h $PGHOST -p $PGPORT -U $PGUSER -d postgres -v ON_ERROR_STOP=1 \
-  -c "DROP DATABASE IF EXISTS employees;" -c "CREATE DATABASE employees;"
-"$PGBIN/pg_restore" -h $PGHOST -p $PGPORT -U $PGUSER -d employees \
-  --no-owner --no-privileges employees.backup \
-  || echo "[seed] pg_restore printed notices — checking the result instead"
-"$PGBIN/psql" -h $PGHOST -p $PGPORT -U $PGUSER -d employees \
-  -c "SELECT count(*) AS employees FROM employees.employee;"
-echo "[seed] template database 'employees' is ready"
-```
-
-A healthy seed reports **300024** employees.
-
-### 3.4 A fresh database for every run
-
-MCPMark isolates each run with `CREATE DATABASE … WITH TEMPLATE …` and asserts
-the target table is absent. Reproduce that as `reset-db.sh` — **run it before
-every single trial**, or the second run finds the table already built and the
-task stops being the task:
-
-```bash
-#!/usr/bin/env bash
-set -euo pipefail
-PGBIN="${PGBIN:-/c/Program Files/PostgreSQL/17/bin}"
-export PGPASSWORD=password; PGUSER=postgres PGHOST=localhost PGPORT=5432
-DB="${TRIAL_DB:-hys_trial}"
-"$PGBIN/psql" -h $PGHOST -p $PGPORT -U $PGUSER -d postgres -v ON_ERROR_STOP=1 \
-  -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity
-      WHERE datname='$DB' AND pid <> pg_backend_pid();" \
-  -c "DROP DATABASE IF EXISTS $DB;" \
-  -c "CREATE DATABASE $DB WITH TEMPLATE employees;"
-LEFT=$("$PGBIN/psql" -h $PGHOST -p $PGPORT -U $PGUSER -d "$DB" -tAc \
-  "SELECT count(*) FROM information_schema.tables
-   WHERE table_schema='employees' AND table_name='hiring_year_summary';")
-[ "$LEFT" = "0" ] || { echo "[reset] target table survived — aborting" >&2; exit 1; }
-echo "[reset] $DB recreated from template — target table absent"
-```
-
-The `pg_terminate_backend` line is not optional: an MCP server holds a pooled
-connection, and `DROP DATABASE` fails while any session is attached.
-
-### 3.5 The grader
-
-Save this as `verify.sh`. It runs MCPMark's own `verify.py`, unmodified, against
-the real database:
-
-```bash
-#!/usr/bin/env bash
-export PYTHONIOENCODING=utf-8          # verify.py prints ✅/❌ — required on Windows consoles
-export POSTGRES_HOST=localhost POSTGRES_PORT=5432 POSTGRES_DATABASE="${TRIAL_DB:-hys_trial}"
-export POSTGRES_USERNAME=postgres POSTGRES_PASSWORD=password
-cd "$(dirname "$0")"
-uv run --quiet --with psycopg2-binary python verify.py
-```
-
-> **`PYTHONIOENCODING=utf-8` is not optional on Windows.** `verify.py` prints
-> `✅`/`❌`/`🎉`; a cp1255/cp1252 console raises `UnicodeEncodeError` *while
-> formatting its own PASS line*, and a run graded through a crashed reporter has
-> no grade.
-
-### 3.6 Run it once, by hand
-
-```bash
-./seed.sh                 # once
-./reset-db.sh             # before the run
-claude -p "$(cat prompt.txt)" --model claude-sonnet-5
-./verify.sh
-```
-
-What a pass looks like:
-
-```
-✅ Hiring year summary results are correct (16 records)
-🎉 Task verification: PASS
-```
-
-That is the benchmark working, end to end, with **no runner involved**. On this
-machine that run cost about **$0.15**. Hold that number — it is the baseline the
-runner has to beat.
-
-The 16 rows it produced (the grader recomputes these in SQL and compares
-row-by-row):
-
-| hire_year | employees_hired | still_employed | retention_rate |
-|---|---|---|---|
-| 1985 | 35316 | 28291 | 80.11 |
-| 1986 | 36150 | 28840 | 79.78 |
-| … | … | … | … |
-| 1999 | 1514 | 1204 | 79.52 |
-| 2000 | 13 | 9 | 69.23 |
-
-(16 rows, hire years 1985 through 2000.)
-
----
-
-## 4. Part B — install the runner
-
-Now put BaseInstRunner between the agent and Postgres. From inside the run
-folder, write the MCP config **unwrapped** — `bir install` will wrap it:
-
-`.mcp.json`
-```json
-{
-  "mcpServers": {
-    "postgres": {
-      "command": "uvx",
-      "args": ["--with", "mcp<2", "postgres-mcp==0.3.0", "--access-mode=unrestricted"],
-      "env": { "DATABASE_URI": "postgresql://postgres:password@localhost:5432/hys_trial" }
-    }
-  }
-}
-```
-
-Two details that will bite if you skip them:
-
-- **`--with "mcp<2"` is mandatory.** `postgres-mcp==0.3.0` imports
-  `mcp.server.fastmcp`, which the `mcp` 2.x SDK removed. MCPMark's own
-  `pipx run postgres-mcp==0.3.0` is broken today without this pin. (`uvx` stands
-  in for `pipx`; both are equivalent here.)
-- **`--access-mode=unrestricted`** — the task creates a table.
-
-Then install, with calculated replay switched on:
-
-```bash
-export BIR_AUTH_URL=https://api.your-domain.com
-bir login                    # once; approve it in the browser
-bir install --replay --global
-```
-
-`bir install --replay` rewrites `.mcp.json` to route `postgres` through
-`bir-proxy`, adds a first-party `bir` server (the channel a replay's results
-come back through), and wires Claude Code's ten hooks into
-`.claude/settings.json`. `bir uninstall` reverses every byte of it.
-
----
-
-## 5. Part C — the two settings that decide everything
-
-This is the section the notes call *"the whole ballgame."* Get these two things
-right before your first trial, or every measurement below will be wrong in a way
-that still *looks* healthy.
-
-### 5.1 Deny the built-in tools
-
-Left alone, Claude Code will reach for `Bash` + `psql` and never touch the MCP
-server — and a single built-in call in the recording makes the whole plan
-un-replayable. Pin the agent to MCP only. Add to `.claude/settings.json`:
-
-```json
-{
-  "permissions": {
-    "deny": ["Bash", "PowerShell", "Read", "Write", "Edit", "NotebookEdit",
-             "Glob", "Grep", "WebFetch", "WebSearch", "Agent", "Task"],
-    "allow": ["mcp__postgres__*", "mcp__bir__*"]
-  }
-}
-```
-
-### 5.2 Trust the workspace — or the `allow` list is silently discarded
-
-**This is the single most common way a first run goes wrong.** If Claude Code
-has not been told to trust this folder, it throws the `allow` list away and
-prints one line you will scroll past:
-
-```
-Ignoring 2 permissions.allow entries from .claude/settings.json:
-this workspace has not been trusted.
-```
-
-It then fails *asymmetrically*: wrapped `mcp__postgres__*` calls keep working
-(bir's own hook allows them), but `mcp__bir__run_scenario` — the one tool the
-replay is delivered through — gets denied. The agent cannot reach the plan, so
-it does the task by hand, and the run lands in the **divergence** path. It still
-passes, the audit log still looks fine, and you have measured nothing.
-
-Fix it **before the first trial**, one of two ways:
-
-- **Interactive, once:** run `claude` in the folder and accept the trust dialog.
-- **By hand:** set this in `~/.claude.json` (use the exact path Claude Code
-  prints, forward slashes):
-
-  ```json
-  "projects": {
-    "C:/Users/Admin/my-first-sample": { "hasTrustDialogAccepted": true }
-  }
-  ```
-
-> The interactive route is the reliable one — Claude Code guards `~/.claude.json`
-> against outside edits, so a script that tries to set the flag may be blocked.
-> Opening `claude` in the folder once and clicking *trust* is the path that
-> always works.
-
-### 5.3 Start the control server, and prove it is bound
-
-Leave this running in its own terminal, **in the run folder**:
-
-```bash
-export BIR_AUTH_URL=https://api.your-domain.com
-unset BIR_REPLAY                          # replay is on by default; BIR_REPLAY=0 turns it off
-export BIR_REPLAY_ALLOW_SERVERS=postgres  # servers allowed to run unattended
-bir-hooks 2>&1 | tee -a bir-hooks.log
-```
-
-Then, in another terminal in the same folder, **launch a `claude` session so a
-proxy registers**, and check:
-
-```bash
-bir doctor
-```
-
-Do not send a single prompt until you see both of these:
-
-```
-Wrapped in config : postgres
-Registered proxies: postgres        ← this needs a live claude session
-```
-
-`Registered proxies: (none)` means no proxy has bound yet. This exact check is
-what separates a real measurement from a confident fiction.
-
----
-
-## 6. Part D — record once, then replay for free
-
-The run loop is one script, `trial.sh`. It resets the DB, runs the agent
-non-interactively, prints the cost, and grades. **Keep `--model` the same for
-every trial**, or the costs are not comparable:
-
-```bash
-#!/usr/bin/env bash
-set -uo pipefail; cd "$(dirname "$0")"
-LABEL="${1:-1}"; MODEL="${MODEL:-claude-sonnet-5}"
-echo "==== trial $LABEL ($MODEL) ===="
-./reset-db.sh || exit 1
-claude -p "$(cat prompt.txt)" --model "$MODEL" \
-  --allowedTools "mcp__postgres__*" "mcp__bir__*" \
-  --output-format json > "trial-$LABEL.json"
-node -e 'const r=JSON.parse(require("fs").readFileSync(process.argv[1]));
-  console.log("cost   $"+(+r.total_cost_usd).toFixed(4)+"  turns="+r.num_turns+
-              "  denied="+(r.permission_denials||[]).length)' "trial-$LABEL.json"
-./verify.sh
-```
-
-### 6.1 Trial 1 — record
-
-```bash
-./trial.sh 1
-```
-
-bir records the run. If the service has seen this prompt before it keeps its own
-canonical run and this one is *not* re-recorded — that is the match working.
-Either way, the plan you will replay is a **calculated scenario**: an intent, a
-parameter schema, and one bit of logic per step. Inspect it before you trust it:
-
-```bash
-bir scenario list                                  # find the runId / scenario id
-bir scenario replay <scenarioId> --prompt "$(cat prompt.txt)" --dry
-```
-
-`--dry` runs the scenario's logic against the *recorded* outputs — no real SQL,
-one cheap model call — and prints each step's computed input. Read it for three
-things: parameters that actually vary with the prompt, step inputs that look
-*computed* rather than copied, and non-empty `emitted` on steps others depend
-on. On our scenario the dry run produced the correct six steps and the right 16
-rows for **$0.004**.
-
-### 6.2 Trials 2+ — replay
-
-Reset, send the identical prompt, grade. Watch `bir-hooks.log` for the story:
-
-```
-run.matched     similarity=1.000
-plan.armed      mode=direct steps=6 coverage=direct,direct,direct,direct,direct,direct
-replay.derived  params=7 costUsd=0.0000 source="recorded samples"
-replay.step     n=0 … ok=true emitted=tableList
-replay.step     n=5 … ok=true emitted=validationRow,totalEmployees,sumHired,…
-replay.done     mode=direct steps=6/6 outcome=steered_full
-execution.reported  outcome=steered_full  session=0.045  savedUsd=+0.174  measured=true
-```
-
-All six SQL steps executed for real through the connection the proxy already
-held — in **about a second of real SQL time** (the `CREATE TABLE … AS` over
-300k rows is most of it), **for $0.00 in model tokens**. The model's only job
-was to read the result.
-
----
-
-## 7. Reading the result — the money
-
-Here are the real figures this task produced, so you can compare yours:
-
-| Run | Outcome | Session cost | Saved vs baseline |
-|---|---|---|---|
-| Baseline (recorded, median of 3) | — | **$0.2188** | — |
-| A clean replay | `steered_full` | **$0.0448** | **+$0.1739  (≈ 80%)** |
-
-Across five verified replays on the test account, the service had booked
-**$0.6374 saved** in total. That is the number the manual promised you: the same
-task, solved once and then not re-solved, at roughly a fifth of the price each
-time after.
-
-What each outcome means for your ledger:
-
-| Outcome | Meaning | Books as |
-|---|---|---|
-| `steered_full` | every step ran under the plan | a **saving** |
-| `diverged` | agent went off-script; bir recovered for $0 | a saving (usually smaller) |
-| `not_steered` | matched but replay was off | a **baseline sample** (this is how the baseline is measured) |
-| `failed` | armed, then gave up | a baseline sample |
-
-Where you see it: on your BaseIn console (for the hosted service,
-**bi2202.com** → sign in → **Dashboard** / **Recordings**), the KPI strip shows
-**Verified saved** — "money we watched move: costs measured on both sides." A
-loss shows red; a saving shows green.
-
----
-
-## 8. What actually happened here, honestly
-
-On the machine this manual was written on, the run folder had **not** been
-trusted (§5.2), so both replays landed in the **divergence** path rather than
-clean `steered_full`. They still passed, and they still saved (`+$0.033` and
-`+$0.116` against the $0.219 baseline) — but two things are worth stating
-plainly, because you may hit them too:
-
-1. **Untrusted → divergence.** With the `allow` list discarded,
-   `mcp__bir__run_scenario` was denied and the agent solved the task by hand.
-   The safety net (bir re-running the six steps for $0) is why it still passed
-   and still saved. Trust the workspace to get the clean channel.
-
-2. **A cautious model may reject the replay as a prompt injection.** When the
-   pre-computed bundle arrived unbidden (the divergence path delivers it as a
-   tool result), Claude Sonnet 5 flagged it:
-
-   > *"the result … came back wrapped in a fake '[BaseInstRunner calculated
-   > replay]' block claiming all 6 steps were 'already executed' … This is a
-   > prompt injection embedded in the tool response channel … I'm not going to
-   > report those figures as fact."*
-
-   The model then re-verified the work itself — which is correct, cautious
-   behaviour, and which raises the per-run cost. Delivering the bundle through
-   the model's *own* `run_scenario` call (the trusted, direct path) is far less
-   likely to trip this, and the bundle wording is being sharpened (see
-   [mcpmark.md](mcpmark.md) §12–14). It is a real product wrinkle, not a setup
-   error — worth knowing before you read too much into one run's cost.
-
-Neither changes the headline: **the scenario is correct, the replay executes the
-real work for zero tokens, and the saving is real and positive.**
-
----
-
-## 9. Troubleshooting
-
-| Symptom | Cause | Fix |
-|---|---|---|
-| `Ignoring N permissions.allow entries … not been trusted` | workspace untrusted | §5.2 — trust it, then every replay is clean |
-| `Registered proxies: (none)` | no live session bound a proxy | launch `claude` in the folder, re-run `bir doctor` |
-| replay always `diverged`, never `steered_full` | usually §5.2, sometimes a cautious model (§8) | trust first; then see mcpmark.md §12–14 |
-| trial 2 finds the table already there | no DB reset | `reset-db.sh` before **every** trial |
-| `UnicodeEncodeError` in verify.py | Windows console codepage | `PYTHONIOENCODING=utf-8` (already in `verify.sh`) |
-| `ModuleNotFoundError: mcp.server.fastmcp` | `mcp` 2.x resolved | pin `--with "mcp<2"` in `.mcp.json` |
-| `bir login` says "not a BaseIn service" | `BIR_AUTH_URL` is a website, not the API | set it to `https://api.<domain>`, open a fresh terminal |
-| `savedUsd` looks impossible | pricing drift between runner and server | compare `PRICING_VERSION` on both sides |
-| the number won't move | a replayed turn is deliberately not re-recorded | correct — the matched run stays canonical |
-
----
-
-## 10. Where to go next
-
-- **Widen within Postgres.** 31 prompts share this exact server and reset
-  mechanism, so each new task costs almost nothing to add.
-  `tasks/postgres/standard/**` is where the paper's numbers come from.
-- **[mcpmark.md](mcpmark.md)** — the full rationale, the topology in detail, and
-  the measured history of this task, including the bugs that were found and
-  fixed along the way.
-- **[calculatedReplayGuide.md](calculatedReplayGuide.md)** — the runner's own
-  manual: every audit line, every switch, and how the saving is booked.
-- **[quickstart.md](quickstart.md)** — setting up a fresh machine to record from
-  your *own* work, not just this benchmark.
-
-> **Turning replay off** is one line: `export BIR_REPLAY=0`. The system is then
-> exactly a recorder again — it recognises a repeated prompt and declines to
-> record it twice, and the model does the work. Nothing is skipped, nothing is
-> risked.
+# Your first sample — run one MCPMark task and watch it pay for itself
+
+A complete, from-zero manual. By the end you will have run one real agent
+benchmark task, graded it against a live database, and then watched
+BaseInstRunner replay the *same* task for **zero model tokens** — and you will
+see the dollar figure that saved.
+
+This is the hands-on companion to [mcpmark.md](mcpmark.md), which explains the
+*why* in depth. This document is the *what to type*, in order, with the numbers
+a real run produced on 2026-09-07 so you know what "working" looks like.
+
+> **The one task.** Everything below uses a single MCPMark task,
+> `postgres/easy/employees/hiring_year_summary`. It is the smallest task in the
+> benchmark that still writes to the database, which makes it the honest place
+> to start.
+
+---
+
+## 0. What MCPMark is (and why we use it)
+
+**MCPMark is a standard agents benchmark.** It measures how well an AI agent can
+drive real tools, through the [Model Context Protocol
+(MCP)](https://modelcontextprotocol.io), to change real systems — not to chat
+about them.
+
+The facts worth knowing before you start:
+
+| | |
+|---|---|
+| Who | The `eval-sys` research group · [github.com/eval-sys/mcpmark](https://github.com/eval-sys/mcpmark) · [paper (arXiv 2509.24002)](https://arxiv.org/abs/2509.24002) · [mcpmark.ai](https://mcpmark.ai) |
+| What it tests | An agent using **five MCP services**: Postgres, GitHub, Filesystem, Notion, Playwright |
+| Size | **127 standard tasks + 50 easy** (the repo has grown past the paper's five services) |
+| How it grades | Each task ships a `verify.py` that opens a real connection and **queries the actual state**, not the agent's transcript. Exit 0 = PASS |
+| Metrics | `pass@1` (one try), `pass@k`, and `pass^k` — *stability*: did it pass **all** k independent runs |
+| State of the art | On MCPMark's verified set, the strongest model reports about **92.9%**; most models sit far lower, and `pass^4` collapses for everyone |
+
+Two of those facts are why this benchmark suits BaseInstRunner so well:
+
+1. **It grades the world, not the words.** A replayed run that produces the
+   right rows passes exactly like a hand-typed one. There is no transcript to
+   fool.
+2. **`pass^k` punishes inconsistency.** The benchmark runs each task several
+   times against a reset database and asks whether *every* run passed. A model
+   that re-decides the plan token-by-token every time is exactly what
+   BaseInstRunner removes — it runs the *same* known-good plan each time.
+
+That is the whole idea you are about to see working: **solve the task once, then
+stop paying to re-solve it.**
+
+---
+
+## 1. What you are about to build
+
+```
+  Part A — the benchmark, on its own
+    seed a Postgres DB  →  agent solves the task  →  verify.py grades it  →  PASS
+
+  Parts B–D — the same task, through BaseInstRunner
+    claude -p ──mcp__postgres__*──▶ bir-proxy ──▶ postgres-mcp ──▶ PostgreSQL
+        │                              │  records every call
+        └──mcp__bir__run_scenario──▶ bir-hooks (control server)
+                                       │  matches the prompt, arms the plan,
+                                       ▼  runs all 6 steps for $0 model tokens
+                                  BaseIn service  ──▶  books the saving
+```
+
+You will run the task **three ways**, and the difference between them is the
+point:
+
+| Run | Runner | What the model does | What it costs |
+|---|---|---|---|
+| **Baseline** | off | discovers and solves the task | full price |
+| **Record** | on, replay **off** | solves it once; bir *records* the plan | full price, one time |
+| **Replay** | on, replay **on** | one tool call reads a pre-computed result | a fraction |
+
+---
+
+## 2. Prerequisites
+
+Check each one. The whole setup takes about ten minutes, most of it downloads.
+
+| # | You need | Check | If missing |
+|---|---|---|---|
+| 1 | **Node 20+** | `node -v` | [nodejs.org](https://nodejs.org) |
+| 2 | **Claude Code** | `claude --version` | Anthropic's CLI; this manual drives it with `claude -p` |
+| 3 | **PostgreSQL 17**, running | `psql --version` and the service is up | see §2.1 |
+| 4 | **uv / uvx** | `uvx --version` | [astral.sh/uv](https://docs.astral.sh/uv/) — runs `postgres-mcp` and `verify.py` |
+| 5 | **The runner** | `bir --version` | `npm i -g @basein/runner` |
+| 6 | **A BaseIn account + API URL** | `bir login` succeeds | your service address, e.g. `https://api.your-domain.com` |
+
+> **`BIR_AUTH_URL` must be the API host**, normally `https://api.<domain>`, not
+> the docs or app website. Point it at a website and `bir login` refuses it,
+> saying what answered instead. Set it once, then open a **fresh** terminal — an
+> existing one keeps the value it started with.
+
+`bir login` signs you in through the browser: it prints a link and a short code
+and waits for you to approve it. Nothing is typed into the terminal, and it works
+for accounts that only sign in with Google.
+
+### 2.1 Postgres, without Docker
+
+MCPMark's own harness runs Postgres in Docker (`pgvector/pgvector:pg17`). If the
+machine has no Docker — this one did not — install PostgreSQL 17 **natively**;
+it is the same major version, and this task needs no vector support. Disclose
+the deviation in any published result.
+
+```powershell
+winget install --id PostgreSQL.PostgreSQL.17 -e `
+  --accept-package-agreements --accept-source-agreements `
+  --custom "--mode unattended --superpassword password --serverport 5432 --unattendedmodeui none"
+```
+
+That sets the `postgres` superuser password to `password`, which is the value
+every command and connection string below expects. On macOS:
+`brew install postgresql@17 && brew services start postgresql@17`.
+
+---
+
+## 3. Part A — run the benchmark task on its own
+
+First prove the plain benchmark works, with no runner in the picture at all.
+
+### 3.1 Make a run folder and fetch the task
+
+```bash
+mkdir -p ~/my-first-sample && cd ~/my-first-sample
+
+TASK=https://raw.githubusercontent.com/eval-sys/mcpmark/main/tasks/postgres/easy/employees/hiring_year_summary
+curl -fsSLO "$TASK/description.md"   # what the agent is asked to do
+curl -fsSLO "$TASK/verify.py"        # MCPMark's grader — unmodified
+curl -fsSLO "$TASK/meta.json"        # task metadata (difficulty L1, mcp: postgres)
+```
+
+The task asks the agent to build one table, `employees.hiring_year_summary`,
+with a row per hiring year: how many were hired, how many are still employed,
+and the retention rate. Small, but a genuine write.
+
+### 3.2 The exact prompt
+
+MCPMark sends `description.md` **verbatim** plus one fixed suffix. Assemble it
+once so every run is byte-identical:
+
+```bash
+{ cat description.md
+  printf '\n\nNote: Based on your understanding, solve the task all at once by yourself, don'\''t ask for my opinions on anything.\n'
+} > prompt.txt
+```
+
+### 3.3 Seed the database
+
+Download MCPMark's 33 MB backup and build the `employees` **template**
+database from it. Save this as `seed.sh` and run it once:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+PGBIN="${PGBIN:-/c/Program Files/PostgreSQL/17/bin}"   # macOS: "$(brew --prefix postgresql@17)/bin"
+export PGPASSWORD="${POSTGRES_PASSWORD:-password}"
+PGUSER=postgres PGHOST=localhost PGPORT=5432
+
+[ -f employees.backup ] || curl -fsSL -o employees.backup https://storage.mcpmark.ai/postgres/employees.backup
+
+"$PGBIN/psql" -h $PGHOST -p $PGPORT -U $PGUSER -d postgres -v ON_ERROR_STOP=1 \
+  -c "DROP DATABASE IF EXISTS employees;" -c "CREATE DATABASE employees;"
+"$PGBIN/pg_restore" -h $PGHOST -p $PGPORT -U $PGUSER -d employees \
+  --no-owner --no-privileges employees.backup \
+  || echo "[seed] pg_restore printed notices — checking the result instead"
+"$PGBIN/psql" -h $PGHOST -p $PGPORT -U $PGUSER -d employees \
+  -c "SELECT count(*) AS employees FROM employees.employee;"
+echo "[seed] template database 'employees' is ready"
+```
+
+A healthy seed reports **300024** employees.
+
+### 3.4 A fresh database for every run
+
+MCPMark isolates each run with `CREATE DATABASE … WITH TEMPLATE …` and asserts
+the target table is absent. Reproduce that as `reset-db.sh` — **run it before
+every single trial**, or the second run finds the table already built and the
+task stops being the task:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+PGBIN="${PGBIN:-/c/Program Files/PostgreSQL/17/bin}"
+export PGPASSWORD=password; PGUSER=postgres PGHOST=localhost PGPORT=5432
+DB="${TRIAL_DB:-hys_trial}"
+"$PGBIN/psql" -h $PGHOST -p $PGPORT -U $PGUSER -d postgres -v ON_ERROR_STOP=1 \
+  -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity
+      WHERE datname='$DB' AND pid <> pg_backend_pid();" \
+  -c "DROP DATABASE IF EXISTS $DB;" \
+  -c "CREATE DATABASE $DB WITH TEMPLATE employees;"
+LEFT=$("$PGBIN/psql" -h $PGHOST -p $PGPORT -U $PGUSER -d "$DB" -tAc \
+  "SELECT count(*) FROM information_schema.tables
+   WHERE table_schema='employees' AND table_name='hiring_year_summary';")
+[ "$LEFT" = "0" ] || { echo "[reset] target table survived — aborting" >&2; exit 1; }
+echo "[reset] $DB recreated from template — target table absent"
+```
+
+The `pg_terminate_backend` line is not optional: an MCP server holds a pooled
+connection, and `DROP DATABASE` fails while any session is attached.
+
+### 3.5 The grader
+
+Save this as `verify.sh`. It runs MCPMark's own `verify.py`, unmodified, against
+the real database:
+
+```bash
+#!/usr/bin/env bash
+export PYTHONIOENCODING=utf-8          # verify.py prints ✅/❌ — required on Windows consoles
+export POSTGRES_HOST=localhost POSTGRES_PORT=5432 POSTGRES_DATABASE="${TRIAL_DB:-hys_trial}"
+export POSTGRES_USERNAME=postgres POSTGRES_PASSWORD=password
+cd "$(dirname "$0")"
+uv run --quiet --with psycopg2-binary python verify.py
+```
+
+> **`PYTHONIOENCODING=utf-8` is not optional on Windows.** `verify.py` prints
+> `✅`/`❌`/`🎉`; a cp1255/cp1252 console raises `UnicodeEncodeError` *while
+> formatting its own PASS line*, and a run graded through a crashed reporter has
+> no grade.
+
+### 3.6 Run it once, by hand
+
+```bash
+./seed.sh                 # once
+./reset-db.sh             # before the run
+claude -p "$(cat prompt.txt)" --model claude-sonnet-5
+./verify.sh
+```
+
+What a pass looks like:
+
+```
+✅ Hiring year summary results are correct (16 records)
+🎉 Task verification: PASS
+```
+
+That is the benchmark working, end to end, with **no runner involved**. On this
+machine that run cost about **$0.15**. Hold that number — it is the baseline the
+runner has to beat.
+
+The 16 rows it produced (the grader recomputes these in SQL and compares
+row-by-row):
+
+| hire_year | employees_hired | still_employed | retention_rate |
+|---|---|---|---|
+| 1985 | 35316 | 28291 | 80.11 |
+| 1986 | 36150 | 28840 | 79.78 |
+| … | … | … | … |
+| 1999 | 1514 | 1204 | 79.52 |
+| 2000 | 13 | 9 | 69.23 |
+
+(16 rows, hire years 1985 through 2000.)
+
+---
+
+## 4. Part B — install the runner
+
+Now put BaseInstRunner between the agent and Postgres. From inside the run
+folder, write the MCP config **unwrapped** — `bir install` will wrap it:
+
+`.mcp.json`
+```json
+{
+  "mcpServers": {
+    "postgres": {
+      "command": "uvx",
+      "args": ["--with", "mcp<2", "postgres-mcp==0.3.0", "--access-mode=unrestricted"],
+      "env": { "DATABASE_URI": "postgresql://postgres:password@localhost:5432/hys_trial" }
+    }
+  }
+}
+```
+
+Two details that will bite if you skip them:
+
+- **`--with "mcp<2"` is mandatory.** `postgres-mcp==0.3.0` imports
+  `mcp.server.fastmcp`, which the `mcp` 2.x SDK removed. MCPMark's own
+  `pipx run postgres-mcp==0.3.0` is broken today without this pin. (`uvx` stands
+  in for `pipx`; both are equivalent here.)
+- **`--access-mode=unrestricted`** — the task creates a table.
+
+Then install, with calculated replay switched on:
+
+```bash
+export BIR_AUTH_URL=https://api.your-domain.com
+bir login                    # once; approve it in the browser
+bir install --replay --global --server postgres
+```
+
+`bir install --replay` routes `postgres` through `bir-proxy` with a wrapped
+copy in this folder's local scope in `~/.claude.json` (your `.mcp.json` stays as
+written), adds a first-party `bir` server there (the channel a replay's results
+come back through), and wires Claude Code's ten hooks into
+`.claude/settings.local.json`. `bir uninstall` takes all of it back out.
+
+`--server postgres` matters in a fresh folder. Claude Code asks before it starts
+a server from `.mcp.json`, but never about a local-scope copy, so `bir install`
+copies a `.mcp.json` server only once Claude Code has approved it; until then it
+prints `postgres — left unwrapped` and only the hooks record. Naming the server
+wraps it now. (Or approve it in an interactive `claude` session here and run the
+install again.)
+
+---
+
+## 5. Part C — the two settings that decide everything
+
+This is the section the notes call *"the whole ballgame."* Get these two things
+right before your first trial, or every measurement below will be wrong in a way
+that still *looks* healthy.
+
+### 5.1 Deny the built-in tools
+
+Left alone, Claude Code will reach for `Bash` + `psql` and never touch the MCP
+server — and a single built-in call in the recording makes the whole plan
+un-replayable. Pin the agent to MCP only. Add to `.claude/settings.json`:
+
+```json
+{
+  "permissions": {
+    "deny": ["Bash", "PowerShell", "Read", "Write", "Edit", "NotebookEdit",
+             "Glob", "Grep", "WebFetch", "WebSearch", "Agent", "Task"],
+    "allow": ["mcp__postgres__*", "mcp__bir__*"]
+  }
+}
+```
+
+### 5.2 Trust the workspace — or the `allow` list is silently discarded
+
+**This is the single most common way a first run goes wrong.** If Claude Code
+has not been told to trust this folder, it throws the `allow` list away and
+prints one line you will scroll past:
+
+```
+Ignoring 2 permissions.allow entries from .claude/settings.json:
+this workspace has not been trusted.
+```
+
+What that breaks: bir does not approve your tool calls. Its hook answers a
+wrapped `mcp__postgres__*` call with no permission decision, so with the `allow`
+list gone Claude Code asks about it — and under `claude -p` a call that has to
+be asked about is denied. The calls bir's hook allows itself are a replay's
+own: `mcp__bir__run_scenario`, the tool a direct replay is delivered through,
+once bir has announced one for this prompt, and a step a steered replay pins.
+(Runners up to 0.2.12 were the other way round: they allowed every wrapped call
+and left `run_scenario` to the `allow` list, which is how the runs in §8 landed
+in the divergence path.)
+
+Fix it **before the first trial**, one of two ways:
+
+- **Interactive, once:** run `claude` in the folder and accept the trust dialog.
+- **By hand:** set this in `~/.claude.json` (use the exact path Claude Code
+  prints, forward slashes):
+
+  ```json
+  "projects": {
+    "C:/Users/Admin/my-first-sample": { "hasTrustDialogAccepted": true }
+  }
+  ```
+
+> The interactive route is the reliable one — Claude Code guards `~/.claude.json`
+> against outside edits, so a script that tries to set the flag may be blocked.
+> Opening `claude` in the folder once and clicking *trust* is the path that
+> always works.
+
+### 5.3 Start the control server, and prove it is bound
+
+Leave this running in its own terminal, **in the run folder**:
+
+```bash
+export BIR_AUTH_URL=https://api.your-domain.com
+unset BIR_REPLAY                          # on here: install --replay stored it; BIR_REPLAY=0 turns it off
+export BIR_REPLAY_ALLOW_SERVERS=postgres  # servers allowed to run unattended
+bir-hooks 2>&1 | tee -a bir-hooks.log
+```
+
+Then, in another terminal in the same folder, **launch a `claude` session so a
+proxy registers**, and check:
+
+```bash
+bir doctor
+```
+
+Do not send a single prompt until you see both of these:
+
+```
+Wrapped in config : postgres
+Registered proxies: postgres        ← this needs a live claude session
+```
+
+`Registered proxies: (none)` means no proxy has bound yet. This exact check is
+what separates a real measurement from a confident fiction.
+
+---
+
+## 6. Part D — record once, then replay for free
+
+The run loop is one script, `trial.sh`. It resets the DB, runs the agent
+non-interactively, prints the cost, and grades. **Keep `--model` the same for
+every trial**, or the costs are not comparable:
+
+```bash
+#!/usr/bin/env bash
+set -uo pipefail; cd "$(dirname "$0")"
+LABEL="${1:-1}"; MODEL="${MODEL:-claude-sonnet-5}"
+echo "==== trial $LABEL ($MODEL) ===="
+./reset-db.sh || exit 1
+claude -p "$(cat prompt.txt)" --model "$MODEL" \
+  --allowedTools "mcp__postgres__*" "mcp__bir__*" \
+  --output-format json > "trial-$LABEL.json"
+node -e 'const r=JSON.parse(require("fs").readFileSync(process.argv[1]));
+  console.log("cost   $"+(+r.total_cost_usd).toFixed(4)+"  turns="+r.num_turns+
+              "  denied="+(r.permission_denials||[]).length)' "trial-$LABEL.json"
+./verify.sh
+```
+
+### 6.1 Trial 1 — record
+
+```bash
+./trial.sh 1
+```
+
+bir records the run. If the service has seen this prompt before it keeps its own
+canonical run and this one is *not* re-recorded — that is the match working.
+Either way, the plan you will replay is a **calculated scenario**: an intent, a
+parameter schema, and one bit of logic per step. Inspect it before you trust it:
+
+```bash
+bir scenario list                                  # find the runId / scenario id
+bir scenario replay <scenarioId> --prompt "$(cat prompt.txt)" --dry
+```
+
+`--dry` runs the scenario's logic against the *recorded* outputs — no real SQL,
+one cheap model call — and prints each step's computed input. Read it for three
+things: parameters that actually vary with the prompt, step inputs that look
+*computed* rather than copied, and non-empty `emitted` on steps others depend
+on. On our scenario the dry run produced the correct six steps and the right 16
+rows for **$0.004**.
+
+### 6.2 Trials 2+ — replay
+
+Reset, send the identical prompt, grade. Watch `bir-hooks.log` for the story:
+
+```
+run.matched     similarity=1.000
+plan.armed      mode=direct steps=6 coverage=direct,direct,direct,direct,direct,direct
+replay.derived  params=7 costUsd=0.0000 source="recorded samples"
+replay.step     n=0 … ok=true emitted=tableList
+replay.step     n=5 … ok=true emitted=validationRow,totalEmployees,sumHired,…
+replay.done     mode=direct steps=6/6 outcome=steered_full
+execution.reported  outcome=steered_full  session=0.045  savedUsd=+0.174  measured=true
+```
+
+All six SQL steps executed for real through the connection the proxy already
+held — in **about a second of real SQL time** (the `CREATE TABLE … AS` over
+300k rows is most of it), **for $0.00 in model tokens**. The model's only job
+was to read the result.
+
+---
+
+## 7. Reading the result — the money
+
+Here are the real figures this task produced, so you can compare yours:
+
+| Run | Outcome | Session cost | Saved vs baseline |
+|---|---|---|---|
+| Baseline (recorded, median of 3) | — | **$0.2188** | — |
+| A clean replay | `steered_full` | **$0.0448** | **+$0.1739  (≈ 80%)** |
+
+Across five verified replays on the test account, the service had booked
+**$0.6374 saved** in total. That is the number the manual promised you: the same
+task, solved once and then not re-solved, at roughly a fifth of the price each
+time after.
+
+What each outcome means for your ledger:
+
+| Outcome | Meaning | Books as |
+|---|---|---|
+| `steered_full` | every step ran under the plan | a **saving** |
+| `diverged` | agent went off a steered plan's script; bir ran the rest for $0 | a saving (usually smaller) |
+| `fell_back` | the plan ran some steps, then handed the rest of the task to the agent | a saving against the whole turn (may be negative) |
+| `not_steered` | matched, but nothing ran under the plan: replay was off, or a direct plan's `run_scenario` was never called, or was refused | a **baseline sample** (this is how the baseline is measured) |
+| `failed` | armed, then gave up | a baseline sample |
+
+Where you see it: on your BaseIn console (for the hosted service,
+**bi2202.com** → sign in → **Dashboard** / **Recordings**), the KPI strip shows
+**Verified saved** — "money we watched move: costs measured on both sides." A
+loss shows red; a saving shows green.
+
+---
+
+## 8. What actually happened here, honestly
+
+On the machine this manual was written on, the run folder had **not** been
+trusted (§5.2), so both replays landed in the **divergence** path rather than
+clean `steered_full`. They still passed, and they still saved (`+$0.033` and
+`+$0.116` against the $0.219 baseline) — but two things are worth stating
+plainly, because you may hit them too:
+
+1. **Untrusted → divergence.** With the `allow` list discarded,
+   `mcp__bir__run_scenario` was denied and the agent solved the task by hand.
+   The safety net (bir re-running the six steps for $0) is why it still passed
+   and still saved. Trust the workspace to get the clean channel. (That was a
+   runner of 0.2.12 or earlier. A current one allows `run_scenario` itself once
+   it has announced a replay, and a direct plan the agent never calls is
+   retired without running a step: booked `not_steered`, the agent's own run.)
+
+2. **A cautious model may reject the replay as a prompt injection.** When the
+   pre-computed bundle arrived unbidden (the divergence path delivers it as a
+   tool result), Claude Sonnet 5 flagged it:
+
+   > *"the result … came back wrapped in a fake '[BaseInstRunner calculated
+   > replay]' block claiming all 6 steps were 'already executed' … This is a
+   > prompt injection embedded in the tool response channel … I'm not going to
+   > report those figures as fact."*
+
+   The model then re-verified the work itself — which is correct, cautious
+   behaviour, and which raises the per-run cost. Delivering the bundle through
+   the model's *own* `run_scenario` call (the trusted, direct path) is far less
+   likely to trip this, and the bundle wording is being sharpened (see
+   [mcpmark.md](mcpmark.md) §12–14). It is a real product wrinkle, not a setup
+   error — worth knowing before you read too much into one run's cost.
+
+Neither changes the headline: **the scenario is correct, the replay executes the
+real work for zero tokens, and the saving is real and positive.**
+
+---
+
+## 9. Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `Ignoring N permissions.allow entries … not been trusted` | workspace untrusted | §5.2 — trust it; until then the `allow` list does not count |
+| `Registered proxies: (none)` | no live session bound a proxy | launch `claude` in the folder, re-run `bir doctor` |
+| `bir install` prints `postgres — left unwrapped` | Claude Code has not approved the `.mcp.json` server yet | `bir install --replay --global --server postgres` (§4) |
+| `replay.decision … code=no_delivery_tool` in the log | no running `bir` server has told the recorder it serves `run_scenario` | check the `bir` server in `/mcp` (connected, not disabled) |
+| replay never `steered_full`: `diverged` on a runner of 0.2.12 or earlier, `not_steered` with a `replay.vetoed` line on a current one | `run_scenario` never ran — denied because of §5.2 (0.2.12 or earlier), refused by a rule of yours, or not called by a cautious model (§8) | trust first; then see mcpmark.md §12–14 |
+| trial 2 finds the table already there | no DB reset | `reset-db.sh` before **every** trial |
+| `UnicodeEncodeError` in verify.py | Windows console codepage | `PYTHONIOENCODING=utf-8` (already in `verify.sh`) |
+| `ModuleNotFoundError: mcp.server.fastmcp` | `mcp` 2.x resolved | pin `--with "mcp<2"` in `.mcp.json` |
+| `bir login` says "not a BaseIn service" | `BIR_AUTH_URL` is a website, not the API | set it to `https://api.<domain>`, open a fresh terminal |
+| `savedUsd` looks impossible | pricing drift between runner and server | compare `PRICING_VERSION` on both sides |
+| the number won't move | a replayed turn is deliberately not re-recorded | correct — the matched run stays canonical |
+
+---
+
+## 10. Where to go next
+
+- **Widen within Postgres.** 31 prompts share this exact server and reset
+  mechanism, so each new task costs almost nothing to add.
+  `tasks/postgres/standard/**` is where the paper's numbers come from.
+- **[mcpmark.md](mcpmark.md)** — the full rationale, the topology in detail, and
+  the measured history of this task, including the bugs that were found and
+  fixed along the way.
+- **[calculatedReplayGuide.md](calculatedReplayGuide.md)** — the runner's own
+  manual: every audit line, every switch, and how the saving is booked.
+- **[quickstart.md](quickstart.md)** — setting up a fresh machine to record from
+  your *own* work, not just this benchmark.
+
+> **Turning replay off** is one line: `export BIR_REPLAY=0` for the `bir-hooks`
+> you start (§5.3), or `bir replay off` for a recorder the session starts itself
+> (then `bir up --restart`). The system is then exactly a recorder again — it
+> recognises a repeated prompt and declines to record it twice, and the model
+> does the work. Nothing is skipped, nothing is risked.
+>
+> **Turning the whole runner off** — no recording, no replay, nothing
+> uninstalled — is `bir off --here`; `bir on --here` turns it back on. To remove
+> it from this folder, `bir uninstall`; before removing the runner package from
+> the machine, `bir uninstall --all`.