@basein/runner 0.1.0

package/docs/calculatedReplayGuide.md

@@ -0,0 +1,448 @@
+# Calculated Replay — instructions
+
+> The operator's half of [calculatedReplay.md](calculatedReplay.md). That document says *why*; this
+> one says *what to type*, in order, from an empty machine to a prompt that answers itself.
+>
+> **Read this first.** Replay executes tool calls with arguments a scenario computed, and it does so
+> **without the permission prompts you would normally see** (design §13.2). It is off by default and
+> it should stay off until you have read §5. Nothing in §1–§4 changes any behaviour.
+
+---
+
+## 0. The shape of it, in thirty seconds
+
+```
+1. Record          you work normally; bir records the run
+2. Calculate       BaseIn analyses that run into a reusable scenario
+3. Match           next time you ask the same thing in different words,
+                   the server recognises it and hands the scenario back
+4. Replay          bir runs the scenario's tools instead of the model
+                   rediscovering the plan
+```
+
+All four are implemented — see §10 for the map from design phase to code, and for how to run the
+end-to-end smoke test.
+
+---
+
+## 1. Prerequisites
+
+| # | You need | Check it with |
+|---|---|---|
+| 1 | A BaseIn service you can reach | `curl -fsS "$BIR_AUTH_URL/health"` |
+| 2 | Anthropic configured **on the server** — calculation and derivation both need it | a `503 anthropic_not_configured` from `/calculate` means it is not |
+| 3 | `SIMILARITY_DETECTION_ENABLED=true` on the server (default) | otherwise no prompt ever matches |
+| 4 | BaseInstRunnerMCP built and installed in your project | `bir status` |
+| 5 | Tier 1 — the hooks wired and `bir-hooks` running | `bir doctor` |
+| 6 | *(replay only)* `ANTHROPIC_API_KEY` in the `bir-hooks` environment | optional — see §5.3 |
+
+Tier 1 is not optional for replay. A match is a match on **the prompt**, and a standalone proxy never
+sees one (design §1.1). If `bir doctor` says `tier: standalone`, replay cannot arm, and that is the
+honest ceiling rather than a bug.
+
+```bash
+export BIR_AUTH_URL=https://your-basein-service
+npm install && npm run build
+node dist/bin/bir.js install        # or `bir install` once linked/published
+bir login
+bir-hooks                           # leave running in its own terminal
+```
+
+---
+
+## 2. Record a run worth reusing
+
+Nothing special — work normally. Two server-side facts decide whether the run becomes reusable, and
+both bite quietly:
+
+**A run needs at least `RECORDING_MIN_ACTIONS` tool actions (default 4).** Below that it is not
+embedded, does not enter the similarity pool, does not appear in the list, and is eventually pruned.
+A two-tool run will never match anything, no matter how many times you repeat it.
+
+**The prompt is what gets embedded.** `findSimilarRun` embeds `run.input` — the text of your
+`UserPromptSubmit` — not the transcript, not the answer. So the prompt that *starts* the turn is the
+one future prompts are compared against. Write the first prompt of a reusable task as if someone else
+will have to phrase it again.
+
+Watch it land:
+
+```
+[bir] …  run.start   run=run_… sess=birsess_… tier=bound
+[bir] …  run.finish  run=run_… steps=14 durationMs=48210 lossy=false recorded=true
+```
+
+`recorded=false` means the service declined the run — usually because it matched something you
+already have. That is §4 working.
+
+List what you have:
+
+```bash
+bir scenario list                    # or: GET /recordings/runs
+```
+
+```
+RUN            ITER  STEPS  SCENARIO   STATE       TITLE
+run_5f3a…         3     14  scn_9c1b…  ready       Fleet risk sweep
+run_a812…         1      9  —          —           Rewrite the billing README
+run_c004…         1      6  scn_44d0…  calculating Check open PRs for stale reviews
+```
+
+---
+
+## 3. Calculate the scenario
+
+A recorded run is a transcript. A **scenario** is what BaseIn derives from it: an intent, a parameter
+schema, and one JavaScript logic body per step that recomputes that step's tool arguments from new
+parameters. Until a run has one, a match can only stop recording — there is nothing to run.
+
+```bash
+bir scenario calc run_5f3a…            # POST /recordings/runs/:runId/calculate → 202
+bir scenario show run_5f3a…            # GET  /recordings/runs/:runId/scenario  (poll)
+```
+
+Raw, if you prefer — the token lives in `~/.baseinstrunner/credentials.json`:
+
+```bash
+TOKEN=$(node -p "require(require('os').homedir()+'/.baseinstrunner/credentials.json').accessToken")
+curl -fsS -X POST "$BIR_AUTH_URL/recordings/runs/run_5f3a…/calculate" \
+     -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' -d '{}'
+```
+
+```powershell
+$t = (Get-Content "$HOME\.baseinstrunner\credentials.json" | ConvertFrom-Json).accessToken
+Invoke-RestMethod -Method Post -Uri "$env:BIR_AUTH_URL/recordings/runs/run_5f3a…/calculate" `
+  -Headers @{ authorization = "Bearer $t" } -ContentType 'application/json' -Body '{}'
+```
+
+Calculation is a background job. Poll `show` until `state` leaves `calculating`:
+
+| `state` | Meaning | Next |
+|---|---|---|
+| `calculating` | job running | poll again in a few seconds |
+| `ready` | usable | §4 |
+| `failed` | see the `error` field | fix and recalculate with `--force` |
+
+`409 scenario_exists` means this run already has one. Re-derive it in place — keeping the id and its
+replay history — with `bir scenario calc run_… --force`.
+
+### 3.1 Check the scenario before you trust it
+
+**Dry replay is the cheapest thing you will do all day. Do it every time.** It derives parameters
+from a fresh prompt and runs the stored logic against the *original run's recorded outputs* — no real
+tools, no side effects, one Haiku call:
+
+```bash
+bir scenario replay scn_9c1b… --prompt "which devices in the fleet look risky right now?" --dry
+```
+
+```
+params      { "fleetId": "eu-west-1", "riskThreshold": 0.8 }
+step 0      mcp__chrome-devtools__navigate_page  { "url": "https://fleet.internal/eu-west-1" }
+step 1      mcp__chrome-devtools__take_snapshot  {}
+              → emitted: devices[24]
+step 2      mcp__fleet-api__device_detail        { "id": "dev_88f1" }
+              → emitted: highestRiskDeviceId, riskScore
+response    { "fleet": "eu-west-1", "atRisk": 3, "worst": "dev_88f1" }
+cost        $0.0012
+```
+
+Read it for three things:
+
+1. **Did the parameters come out right** for a prompt phrased differently from the original?
+2. **Do the step inputs look computed**, or are they the original run's constants copied through? A
+   scenario whose inputs never vary with the parameters is a recording, not a scenario.
+3. **Is `emitted` non-empty** for steps that later steps depend on? An empty `emitted` means
+   `toolOutputLogic` derived nothing, and every downstream input reading it will be wrong.
+
+A dry replay is recorded as `outcome: 'dry'` and is excluded from savings by construction — it costs
+you a Haiku call and nothing else.
+
+---
+
+## 4. Confirm the match fires
+
+Now ask the same thing in different words, in a new session.
+
+```
+you:  can you check the eu-west fleet and tell me which boxes are in trouble?
+```
+
+```
+[bir] …  run.matched  run=run_… matchedRun=run_5f3a… similarity=0.94
+         why="similar prompt — the service kept its own run; not recording this one"
+```
+
+That line is the whole feature working up to the point of replay. If it does not appear, the prompt
+did not clear the server's `SIMILARITY_THRESHOLD` (default 0.9) — see §8.
+
+**With replay off (the default), this is where it stops.** The server keeps its canonical run, bumps
+its `iterations`, this turn is not recorded, and the model answers the ordinary way. Nothing is lost
+and nothing is skipped.
+
+---
+
+## 5. Turn replay on
+
+> Read §5.1 before running anything in this section.
+
+### 5.1 What you are agreeing to
+
+A calculated scenario is, in effect, **a pre-approved list of tool calls with computed arguments.**
+When replay arms:
+
+- **Steered steps are auto-approved.** `PreToolUse` is answered `allow`, so a pinned `Bash` or `Edit`
+  runs without the prompt you would normally see.
+- **Direct steps are not visible to the permission system at all.** They are dispatched straight to a
+  `bir-proxy`, which calls the upstream on a connection it already holds. Claude Code never sees them.
+
+That is what makes replay fast, and it is the whole risk. Two things follow:
+
+1. Only enable it in a project whose scenarios you have dry-replayed and read (§3.1).
+2. Set `BIR_REPLAY_ALLOW_SERVERS` (§5.3). An allowlist of the servers you are comfortable having
+   called unattended is cheap insurance, and it is the difference between "replay reads a dashboard"
+   and "replay can do anything any wrapped server can do".
+
+### 5.2 Enable it
+
+```bash
+bir install --replay          # adds the `bir` MCP server; raises the prompt-hook timeout
+export BIR_REPLAY=1           # in the terminal that runs bir-hooks
+bir-hooks                     # restart it
+```
+
+`bir install --replay` does exactly two things beyond a normal install, and `bir uninstall` reverses
+both:
+
+- registers a first-party MCP server under the key **`bir`**, exposing one tool,
+  `mcp__bir__run_scenario` — the channel a fully-wrapped scenario's results come back through;
+- raises `UserPromptSubmit`'s hook timeout from 5 s to 15 s, so the match round trip fits.
+
+The `bir` tool is visible to the model in every session in this project. That is a real cost — one
+more tool in the list, a little context per turn — and it is why replay is a separate flag rather
+than part of `bir install`.
+
+Confirm:
+
+```bash
+bir doctor
+```
+
+```
+replay          ON      servers=chrome-devtools,fleet-api   minSimilarity=0.92
+                derive=claude-haiku-4-5-20251001 (ANTHROPIC_API_KEY present)
+control         http://127.0.0.1:53411   sess=birsess_…
+wrapped         chrome-devtools ✓ proxy pid 41822   fleet-api ✓ proxy pid 41823
+recording       yes  (https://your-basein-service)
+```
+
+### 5.3 Environment
+
+**Set these in one place: the terminal that runs `bir-hooks`.** The proxies need no replay
+configuration of their own — each one is told whether to open its work loop when it registers, so
+there is exactly one switch and it cannot get out of step with itself.
+
+| Variable | Default | What it does |
+|---|---|---|
+| `BIR_REPLAY` | *(unset)* | `1` enables replay. Nothing below matters until it is set |
+| `BIR_REPLAY_ALLOW_SERVERS` | *(all wrapped)* | Comma-separated server keys eligible for **direct** execution. **Set this** |
+| `BIR_MIN_STEER_SIMILARITY` | `0.92` | Below this a match is detected but not replayed (§8) |
+| `ANTHROPIC_API_KEY` | *(unset)* | Enables parameter derivation. Without it, replay uses the scenario's **recorded sample values** — free, and often still correct |
+| `BIR_DERIVE_MODEL` | `claude-haiku-4-5-20251001` | The derivation model |
+| `BIR_MATCH_BUDGET_MS` | `2500` | How long the prompt hook waits for a match before giving up |
+| `BIR_DERIVE_BUDGET_MS` | `8000` | How long the first `PreToolUse` waits for parameters |
+| `BIR_REPLAY_BUDGET_MS` | `120000` | Whole-plan ceiling for direct execution |
+| `BIR_STEP_TIMEOUT_MS` | `60000` | Per-step ceiling for one direct `tools/call` |
+| `BIR_VERBOSE` | *(unset)* | `1` adds per-step `replay.*` detail lines |
+
+**No API key is a supported configuration, not a broken one.** Derivation is skipped and every
+parameter takes the value it had in the recorded run. For a scenario whose parameters rarely change
+— a dashboard sweep, a fixed report — that is a complete, free replay. For one that keys off the
+prompt ("check *eu-west*"), you want the key.
+
+---
+
+## 6. Run it, and read what happened
+
+Ask the matching question again, with replay on.
+
+### 6.1 A fully-wrapped scenario — direct
+
+```
+[bir] …  run.matched   run=run_… matchedRun=run_5f3a… similarity=0.94
+[bir] …  plan.armed    scenario=scn_9c1b… mode=direct steps=3
+         tools="mcp__chrome-devtools__navigate_page,mcp__chrome-devtools__take_snapshot,mcp__fleet-api__device_detail"
+[bir] …  replay.derived  params=2 costUsd=0.0011 ms=840
+[bir] …  replay.step     n=0 tool=navigate_page server=chrome-devtools ms=612 ok=true
+[bir] …  replay.step     n=1 tool=take_snapshot  server=chrome-devtools ms=1104 ok=true emitted=devices
+[bir] …  replay.step     n=2 tool=device_detail  server=fleet-api ms=210 ok=true emitted=highestRiskDeviceId,riskScore
+[bir] …  replay.done     scenario=scn_9c1b… mode=direct steps=3/3 outcome=steered_full ms=2766
+[bir] …  execution.reported  scenario=scn_9c1b… outcome=steered_full derive=0.0011 session=0.0083 fallback=0 savedUsd=0.114
+```
+
+In the session you will see one tool call, `mcp__bir__run_scenario`, returning the composed results.
+The model reads them and answers. It never emitted the three underlying calls.
+
+### 6.2 A mixed scenario — steer
+
+The model emits each call itself; `bir` overrides the arguments.
+
+```
+[bir] …  plan.armed    scenario=scn_44d0… mode=steer steps=4
+[bir] …  replay.pin    n=0 tool=mcp__github__list_pull_requests
+[bir] …  replay.thread n=0 via=proxy emitted=prNumbers
+[bir] …  replay.pin    n=1 tool=Read
+[bir] …  replay.thread n=1 via=hook emitted=reviewers
+[bir] …  replay.done   scenario=scn_44d0… mode=steer steps=4/4 outcome=steered_full
+```
+
+`via=proxy` versus `via=hook` matters: a wrapped MCP step's output is threaded from the **proxy's**
+report, because that is the shape the scenario's logic was written against (design §7.2). A
+`via=hook fallback=true` line means the proxy did not report in time and the hook's differently-shaped
+view was used instead — expect derived values to be empty, and treat repeats as a bug.
+
+### 6.3 The model goes off-script
+
+```
+[bir] …  replay.diverge  scenario=scn_44d0… expected=Read called=Bash step=2/4
+[bir] …  replay.compose  remaining=2 direct=1 recorded=1 skipped=0 bytes=8412 costUsd=0
+[bir] …  replay.done     outcome=diverged steps=4/4
+```
+
+`costUsd=0` on the compose line is the point of the whole design: the remaining steps ran through
+proxies that were already connected, so recovering from a divergence cost no tokens at all.
+
+### 6.4 The audit vocabulary
+
+Everything replay decides is one line. `grep` for these:
+
+| Line | Means |
+|---|---|
+| `run.matched` | The server recognised the prompt |
+| `replay.decision … verdict=no-steer` | Matched, and declined. `why=` says which gate |
+| `plan.armed` | A plan exists. Carries mode, step count and the ordered tool names — **before** anything runs |
+| `replay.derived` / `replay.derive_failed` | Parameters resolved, or fell back to sample values |
+| `replay.pin` / `replay.thread` | One steered step's input pinned / output threaded |
+| `replay.step` | One direct step executed |
+| `replay.diverge` / `replay.compose` | Off-script, and the recovery |
+| `replay.done` | Final outcome, steps completed, elapsed |
+| `execution.reported` | The ticket was redeemed and the saving booked |
+
+Keep them: `bir-hooks 2>&1 | tee -a ~/.baseinstrunner/audit.log`.
+
+---
+
+## 7. What each outcome means for your numbers
+
+Reported once per matched turn, to `POST /scenarios/:id/executions`, redeeming the match's
+`executionTicket`.
+
+| Outcome | When | Where it lands |
+|---|---|---|
+| `steered_full` | Every planned step ran under the scenario's control | savings ledger |
+| `diverged` | Off-script; the composed bundle answered | savings ledger |
+| `not_steered` | Matched, declined at a gate — the agent ran the task normally | **baseline sample** |
+| `failed` | Armed, then gave up — the agent ran the task normally | **baseline sample** |
+
+`not_steered` is not an error, and it is not wasted. A saving is `baseline − actual`, and the baseline
+is *what this task costs when the agent does it*. Every declined match is a fresh measurement of
+exactly that. A deployment that never declines has no baseline to measure against — which is a good
+reason to leave `BIR_MIN_STEER_SIMILARITY` where it is rather than driving it to zero.
+
+An **unredeemed** ticket books nothing at all. Silence loses a saving; it never invents one.
+
+---
+
+## 8. Tuning
+
+Only one knob matters in practice.
+
+**`BIR_MIN_STEER_SIMILARITY` (default 0.92).** The server detects matches at 0.9 and uses that to
+decide *"don't record this again"* — cheap: the worst case is a lost recording. Steering makes a much
+stronger claim, *"don't think about this again"*, and a 0.90-similar prompt usually covers only part
+of the live task. Steering it produces a confident, thin, wrong answer.
+
+Tune it from evidence, not taste. Run for a week and read the declines:
+
+```bash
+grep 'replay.decision' audit.log | grep 'why="similarity below threshold"'
+```
+
+- Prompts you *would* have wanted replayed, sitting at 0.90–0.92 → lower it, in steps of 0.01.
+- Replays that answered the wrong question → raise it, and re-read the scenario's `intent`.
+
+If a specific pair of prompts *should* match and does not, the fix is usually upstream of the
+threshold: the two prompts genuinely mean different things to the embedding model. Re-record with a
+first prompt that states the task plainly.
+
+---
+
+## 9. Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| No `run.matched`, ever | Prompt below the server's `SIMILARITY_THRESHOLD` | Rephrase closer, or lower it server-side. Confirm the original run is in the list — a sub-`RECORDING_MIN_ACTIONS` run is never embedded |
+| `run.matched` but no `plan.armed` | A gate declined | Read the `replay.decision` line's `why=`. Ranked by frequency: `BIR_REPLAY` unset · scenario not `ready` · similarity below threshold · no step is executable |
+| `why="no step is executable"` | None of the scenario's tools is a wrapped MCP server and none is a built-in reachable in this session | Wrap the servers the scenario uses (`bir install --server …`) and recalculate |
+| `plan.armed mode=steer` where you expected `direct` | At least one step is a built-in, an unwrapped MCP server, or `claude-in-chrome` | `bir status` shows what is wrapped. `claude-in-chrome` is `scope: "dynamic"` and can never be wrapped |
+| Model ignores the directive and diverges every turn | Steering is advisory — the model chooses; `bir` only pins arguments | Expected occasionally. Persistent divergence usually means the scenario's tools do not fit the live task; check `intent` |
+| `replay.thread … emitted=` empty | `toolOutputLogic` derived nothing — usually the output shape changed since recording | Dry-replay (§3.1). If the dry run emits and the live run does not, the tool's output shape has drifted; recalculate |
+| `via=hook fallback=true` on a wrapped step | The proxy did not report within the grace window | Check that proxy is alive (`bir doctor`). Repeats mean a slow or dying upstream |
+| `run_scenario` returns "no scenario is armed" | The model called it on a turn with no plan | Harmless. It happens when the model remembers the tool from an earlier turn |
+| `execution.reported … 409 invalid_ticket` | The ticket was already redeemed, or belongs to another scenario | Harmless if `duplicate: true`. Otherwise a rollover raced `SessionEnd`; the server books once |
+| `savedUsd` looks impossible | Pricing drift between the runner's table and the server's | Compare `PRICING_VERSION` on both sides. Design §11.3 documents a known live drift between two existing copies |
+| `sessionCostUsd` grows every prompt in a session | The per-run usage watermark is missing or not taken | Design §11.4. Reports without a mark must carry `measured: false` |
+| Everything works, nothing is saved | A replayed turn is deliberately not recorded (design §12) | Correct. The matched run stays canonical and its `iterations` is bumped |
+
+---
+
+## 10. What is built
+
+All of it. Phases R0–R7 of [calculatedReplay.md](calculatedReplay.md) §17 are implemented and
+tested; everything this guide describes is code you can run.
+
+| Phase | What landed | Where |
+|---|---|---|
+| **R0** | Plan, logic shims, coverage, bundle | [src/replay/](../src/replay/) — `plan.ts`, `logic.ts`, `coverage.ts`, `bundle.ts` |
+| **R1** | Derivation over raw `fetch`; pricing | [derive.ts](../src/replay/derive.ts), [pricing.ts](../src/replay/pricing.ts) |
+| **R2** | Gate ladder, mode choice, arming, directive | [controller.ts](../src/replay/controller.ts), [control/server.ts](../src/control/server.ts) |
+| **R3** | Pin, thread, retire | `onToolPre` / `onToolPost` / `onProxyStep` in [control/server.ts](../src/control/server.ts) |
+| **R4** | Long-poll work queue; direct execution | [executor.ts](../src/replay/executor.ts), [proxy/session.ts](../src/proxy/session.ts) |
+| **R5** | The `bir` MCP server, `/scenario/run`, `--replay` | [bin/bir-scenario.ts](../src/bin/bir-scenario.ts), [bin/bir.ts](../src/bin/bir.ts) |
+| **R6** | Divergence, source-run fallback, `bir replay` | `diverge()` in [controller.ts](../src/replay/controller.ts), [source-run.ts](../src/replay/source-run.ts) |
+| **R7** | Usage watermark, reporting, ticket, doctor | [control/transcript.ts](../src/control/transcript.ts), [remote-recorder.ts](../src/record/remote-recorder.ts) |
+
+```bash
+npm test              # 195 tests, incl. test/replay.test.ts and test/replay-server.test.ts
+npm run build && node test/smoke-replay.mjs
+```
+
+The smoke test is the one worth running after any change to the seams. It starts a stand-in BaseIn
+service, a real `bir-hooks`, a real `bir-proxy` wrapping a real MCP server, and a real `bir-scenario`,
+then drives a two-step scenario end to end through actual stdio — proving that a pinned argument
+reaches the real tool, that step 1's input was computed from step 0's real output, and that the turn
+books `steered_full` with `fallbackCostUsd: 0`.
+
+Three things the implementation settled that the design left open:
+
+- **`SerializedScenarioStep.recordedOutput` really is never sent.** Recorded outputs come from a lazy
+  `GET /recordings/runs/:runId` ([source-run.ts](../src/replay/source-run.ts)), as design §8.1 anticipated.
+- **The work queue must hold work for a proxy that is mid-round-trip.** Failing fast when no poller is
+  *parked* lost every step after the first in a multi-step direct plan, because the proxy is between
+  polls exactly then. It now fails fast only for a server no proxy has served in the last minute.
+- **The proxies need no replay configuration.** `/proxy/register` tells each one whether to open its
+  work loop, so `BIR_REPLAY` is a single switch on the control server.
+
+---
+
+## 11. Turning it off
+
+Three levels, least to most.
+
+```bash
+unset BIR_REPLAY        # stop arming. Matches still detected; recording still stops on a match
+bir uninstall --replay  # remove the `bir` MCP server, restore the prompt-hook timeout
+bir uninstall           # remove everything: proxies, hooks, files restored byte-for-byte
+```
+
+With `BIR_REPLAY` unset the system is exactly v1 again: it recognises a repeated prompt, declines to
+record it a second time, and lets the model do the work.