@basein/runner 0.2.11 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (144) hide show
  1. package/README.md +540 -350
  2. package/dist/auth/client.d.ts +65 -3
  3. package/dist/auth/client.js +305 -53
  4. package/dist/bin/bir-hooks.d.ts +72 -31
  5. package/dist/bin/bir-hooks.js +176 -284
  6. package/dist/bin/bir-proxy.d.ts +17 -2
  7. package/dist/bin/bir-proxy.js +222 -28
  8. package/dist/bin/bir-scenario.d.ts +61 -10
  9. package/dist/bin/bir-scenario.js +704 -135
  10. package/dist/bin/bir.d.ts +14 -8
  11. package/dist/bin/bir.js +2937 -488
  12. package/dist/bin/investigate.js +41 -16
  13. package/dist/bin/scenario-edit.d.ts +43 -10
  14. package/dist/bin/scenario-edit.js +281 -80
  15. package/dist/bin/scenario-liveness.d.ts +194 -0
  16. package/dist/bin/scenario-liveness.js +459 -0
  17. package/dist/bin/scenario-stderr.d.ts +43 -0
  18. package/dist/bin/scenario-stderr.js +76 -0
  19. package/dist/bin/service-read.d.ts +88 -0
  20. package/dist/bin/service-read.js +128 -0
  21. package/dist/bin/setup.d.ts +19 -0
  22. package/dist/bin/setup.js +146 -20
  23. package/dist/config/adapters/claude-code.d.ts +92 -14
  24. package/dist/config/adapters/claude-code.js +291 -101
  25. package/dist/config/adapters/generic.d.ts +4 -1
  26. package/dist/config/adapters/generic.js +13 -14
  27. package/dist/config/generate.d.ts +392 -14
  28. package/dist/config/generate.js +666 -19
  29. package/dist/config/mcp-approval.d.ts +76 -0
  30. package/dist/config/mcp-approval.js +210 -0
  31. package/dist/config/node-path.d.ts +42 -0
  32. package/dist/config/node-path.js +102 -0
  33. package/dist/config/overrides.d.ts +62 -0
  34. package/dist/config/overrides.js +136 -0
  35. package/dist/config/resolve.d.ts +70 -8
  36. package/dist/config/resolve.js +186 -15
  37. package/dist/control/client.d.ts +22 -0
  38. package/dist/control/client.js +26 -7
  39. package/dist/control/correlation.d.ts +23 -5
  40. package/dist/control/correlation.js +0 -0
  41. package/dist/control/daemon.d.ts +400 -18
  42. package/dist/control/daemon.js +1105 -102
  43. package/dist/control/discovery.d.ts +77 -5
  44. package/dist/control/discovery.js +186 -11
  45. package/dist/control/ensure-hook.d.ts +61 -0
  46. package/dist/control/ensure-hook.js +160 -8
  47. package/dist/control/paths.d.ts +26 -0
  48. package/dist/control/paths.js +34 -0
  49. package/dist/control/pin-guard.d.ts +83 -0
  50. package/dist/control/pin-guard.js +534 -0
  51. package/dist/control/recorder-keeper.d.ts +229 -0
  52. package/dist/control/recorder-keeper.js +721 -0
  53. package/dist/control/recorder-main.d.ts +73 -0
  54. package/dist/control/recorder-main.js +778 -0
  55. package/dist/control/server.d.ts +707 -31
  56. package/dist/control/server.js +2803 -306
  57. package/dist/control/transcript.d.ts +21 -1
  58. package/dist/control/transcript.js +378 -112
  59. package/dist/index.d.ts +5 -1
  60. package/dist/index.js +5 -1
  61. package/dist/jsonrpc/framing.d.ts +76 -12
  62. package/dist/jsonrpc/framing.js +216 -59
  63. package/dist/proxy/intercept.d.ts +49 -5
  64. package/dist/proxy/intercept.js +188 -59
  65. package/dist/proxy/relay.d.ts +16 -0
  66. package/dist/proxy/relay.js +77 -5
  67. package/dist/proxy/revive.d.ts +122 -0
  68. package/dist/proxy/revive.js +251 -0
  69. package/dist/proxy/session.d.ts +222 -14
  70. package/dist/proxy/session.js +708 -99
  71. package/dist/record/housekeeping.d.ts +71 -0
  72. package/dist/record/housekeeping.js +417 -0
  73. package/dist/record/recorder.d.ts +51 -3
  74. package/dist/record/recorder.js +3 -0
  75. package/dist/record/redact.d.ts +103 -9
  76. package/dist/record/redact.js +659 -30
  77. package/dist/record/remote-recorder.d.ts +327 -32
  78. package/dist/record/remote-recorder.js +1517 -187
  79. package/dist/record/switchable-recorder.d.ts +126 -0
  80. package/dist/record/switchable-recorder.js +351 -0
  81. package/dist/record/truncate.d.ts +40 -1
  82. package/dist/record/truncate.js +111 -4
  83. package/dist/replay/bash-channel.d.ts +35 -0
  84. package/dist/replay/bash-channel.js +133 -0
  85. package/dist/replay/bundle.d.ts +10 -0
  86. package/dist/replay/bundle.js +10 -0
  87. package/dist/replay/controller.d.ts +371 -20
  88. package/dist/replay/controller.js +1185 -111
  89. package/dist/replay/derive.d.ts +1 -0
  90. package/dist/replay/derive.js +81 -24
  91. package/dist/replay/executor.d.ts +51 -16
  92. package/dist/replay/executor.js +114 -41
  93. package/dist/replay/handover.d.ts +54 -1
  94. package/dist/replay/handover.js +55 -5
  95. package/dist/replay/logic.d.ts +62 -10
  96. package/dist/replay/logic.js +140 -22
  97. package/dist/replay/plan.d.ts +201 -27
  98. package/dist/replay/plan.js +581 -141
  99. package/dist/replay/safety.d.ts +72 -0
  100. package/dist/replay/safety.js +303 -0
  101. package/dist/replay/source-run.d.ts +10 -0
  102. package/dist/replay/source-run.js +80 -35
  103. package/dist/replay/targets.d.ts +15 -1
  104. package/dist/replay/targets.js +62 -4
  105. package/dist/replay/tool-error.d.ts +15 -1
  106. package/dist/replay/tool-error.js +33 -1
  107. package/dist/replay/types.d.ts +7 -4
  108. package/dist/upstream/client.d.ts +56 -3
  109. package/dist/upstream/client.js +61 -4
  110. package/dist/upstream/headers.d.ts +43 -0
  111. package/dist/upstream/headers.js +90 -0
  112. package/dist/upstream/http-client.d.ts +213 -22
  113. package/dist/upstream/http-client.js +1034 -168
  114. package/dist/upstream/http-transport.d.ts +143 -0
  115. package/dist/upstream/http-transport.js +391 -0
  116. package/dist/upstream/lazy-client.d.ts +8 -0
  117. package/dist/upstream/lazy-client.js +29 -1
  118. package/dist/upstream/sse.d.ts +78 -0
  119. package/dist/upstream/sse.js +189 -0
  120. package/dist/upstream/stdio-client.d.ts +61 -1
  121. package/dist/upstream/stdio-client.js +356 -33
  122. package/dist/util/atomic-write.d.ts +29 -0
  123. package/dist/util/atomic-write.js +96 -0
  124. package/dist/util/journal.d.ts +15 -0
  125. package/dist/util/journal.js +48 -3
  126. package/dist/util/log.d.ts +10 -0
  127. package/dist/util/log.js +28 -1
  128. package/dist/util/off-switch.d.ts +46 -0
  129. package/dist/util/off-switch.js +91 -0
  130. package/dist/util/service-health.d.ts +133 -0
  131. package/dist/util/service-health.js +217 -0
  132. package/docs/BaseInstRunner.md +694 -621
  133. package/docs/calculatedReplay.md +1625 -1244
  134. package/docs/calculatedReplayGuide.md +1212 -879
  135. package/docs/installRun.md +1013 -772
  136. package/docs/loginWeb.md +627 -607
  137. package/docs/mcpmark.md +49 -16
  138. package/docs/my-first-sample.md +569 -545
  139. package/docs/quickstart.md +384 -271
  140. package/docs/t-bench.md +406 -394
  141. package/package.json +58 -57
  142. package/production.md +1371 -0
  143. package/scripts/install.ps1 +97 -6
  144. package/scripts/install.sh +104 -7
@@ -1,1244 +1,1625 @@
1
- # Calculated Replay — running a scenario on a similar-meaning match
2
-
3
- > Design doc for BaseInstRunnerMCP **v2** ([BaseInstRunner.md](BaseInstRunner.md) §12), the
4
- > milestone the v1 design deferred: *"Serve recorded results; port RRepeat's divergence handling."*
5
- >
6
- > **What it adds.** When the user's prompt means the same thing as a prompt they have run before,
7
- > the BaseIn service says so and hands back the *calculated scenario* derived from that earlier run.
8
- > This document specifies what BaseInstRunnerMCP does with it: derive the new prompt's parameters,
9
- > run the scenario's tool sequence against live data, and let the model answer from genuine results —
10
- > instead of rediscovering the same plan token by token.
11
- >
12
- > **Status:** implemented. Phases R0–R7 (§17) are in `src/replay/`, wired through the control server
13
- > and the proxies, covered by `test/replay.test.ts` and `test/replay-server.test.ts`, and proved end
14
- > to end with real processes by `test/smoke-replay.mjs`. Three notes below are marked **[built]**
15
- > where the implementation settled something this design left open.
16
- > **Reference implementation:** RRepeat's [`docs/replay.md`](../../../BasInstRunner/RRepeat/docs/replay.md)
17
- > §3, §12, §13 and [`docs/savingCalcRunner.md`](../../../BasInstRunner/RRepeat/docs/savingCalcRunner.md).
18
- > **Server contract:** BaseIn `src/recordings/routes.ts`, `src/scenarios/{routes,repo,replay,logic}.ts`.
19
-
20
- ---
21
-
22
- ## 0. The one fact that makes this different from RRepeat
23
-
24
- RRepeat and BaseInstRunnerMCP are solving the same problem from opposite sides of the tool call.
25
-
26
- RRepeat **observes** a Claude Code session through hooks. It has no execution path of its own, so
27
- when a replay diverges it must spawn a *second* Claude Agent SDK session to run the remaining tools
28
- for real (`sdk-tool-session.ts`, `replay.md` §13). Its own savings doc names the consequence:
29
-
30
- > *"the **dominant** cost of today's calculated run is a full second agent session, and it is the one
31
- > cost nobody counts."* — `savingCalcRunner.md` §4
32
-
33
- BaseInstRunnerMCP **is** the execution path for MCP. Every wrapped server is reached through a
34
- `bir-proxy` process that already holds an initialized, live `UpstreamClient` to it
35
- ([`src/upstream/client.ts`](../src/upstream/client.ts)). Issuing a `tools/call` on that connection
36
- costs **zero tokens** and requires no second agent, no subprocess, and no model in the loop.
37
-
38
- Everything below follows from that. Where RRepeat has one replay mode and an expensive fallback,
39
- this design has a ladder whose cheapest rung is free — and the more of a stack sits behind
40
- `bir-proxy`, the further down the ladder a scenario lands.
41
-
42
- | | RRepeat | BaseInstRunnerMCP |
43
- |---|---|---|
44
- | Runs MCP tools itself | No (`replay.md` §12: *"rrepeat never executes tools — not even MCP"*) | **Yes**, on the connection it already owns |
45
- | Divergence fallback | A second Agent SDK session — the dominant cost | Direct `tools/call` through the proxies — **$0** |
46
- | Reaches built-ins (`Bash`, `Read`) | Only by steering the live session | Same — only by steering the live session |
47
- | Needs `@anthropic-ai/*` at runtime | Yes | No (§16.3) |
48
-
49
- ---
50
-
51
- ## 1. Decisions locked
52
-
53
- Continuing v1's D1–D8.
54
-
55
- | # | Decision | Choice | Consequence |
56
- |---|---|---|---|
57
- | **D9** | What a match buys | **Execute the scenario**, do not merely stop recording | v1's `watchForMatch` already detects the hit and stops; this replaces the no-op with work |
58
- | **D10** | Who decides the mode | **Tool coverage**, computed per scenario at arm time (§5) | A fully-wrapped scenario replays for free; a mixed one steers the live session |
59
- | **D11** | Execution ownership for MCP steps | **The proxy that owns the upstream** | No second connection, no double-spawned browser, no Agent SDK dependency |
60
- | **D12** | Control → proxy channel | **Long-poll**, proxy dials out | No new listeners, no new ports, no new tokens (§13.3, §19.2) |
61
- | **D13** | Result delivery in direct mode | A **first-party MCP server** (`bir`), one tool | The model reads a genuine `tool_result`, never a `deny` reason it distrusts (§19.1) |
62
- | **D14** | Default state | **On** once `bir install --replay` has run; `BIR_REPLAY=0` turns it off | Replay bypasses permission prompts (§13.2); the install flag is the opt-in, the env var is the kill switch |
63
- | **D15** | Parameter derivation | One Haiku call over raw `fetch`; **no API key ⇒ recorded sample values** | Preserves v1's zero-runtime-dependency property, and degrades to a free replay |
64
- | **D16** | Savings reporting | Every armed match reports to `POST /scenarios/:id/executions`, including declines | A decline is a *baseline sample*, not silence — it is what keeps the ledger honest |
65
-
66
- ### 1.1 Non-goals
67
-
68
- - **Replay in Tier 2.** A match is a match *on the prompt*, and a standalone proxy never sees one
69
- (§0.1 of the v1 design). Tier 2 gets the manual `bir replay` command (§12.1) and nothing automatic.
70
- - **Chaining scenarios.** RRepeat's `next`/`getScenarioChain` has no counterpart in BaseIn's
71
- `serializeScenario` payload, which serializes exactly one scenario. Out of scope.
72
- - **Editing tool descriptions to steer selection.** Still a non-goal, unchanged. §6's `bir` server
73
- adds one *first-party* tool and touches no upstream schema — and it costs context in every session,
74
- which is why D14 makes it opt-in.
75
- - **Recording the replayed turn.** The service created no run for it, by design. §12.
76
-
77
- ---
78
-
79
- ## 2. The match, and where it already works
80
-
81
- Nothing in this section is new. It is already implemented and already runs on every prompt.
82
-
83
- ```
84
- UserPromptSubmit
85
- └─ ControlServer.onPrompt src/control/server.ts
86
- └─ ensureRun → recorder.startRun(input, metadata)
87
- └─ RemoteRecorder.startRun POST /recordings/runs { id, framework, input, metadata }
88
- └─ BaseIn recordings/routes.ts:
89
- findSimilarRun(userId, input) ← embed + cosine over the user's runs
90
- ├─ miss → 201 { id, matched: null }
91
- └─ hit → 200 { id: null, matched: {...} } ← no run is created
92
- ```
93
-
94
- On a hit the server answers with the whole thing — there is no follow-up fetch:
95
-
96
- ```jsonc
97
- {
98
- "id": null,
99
- "matched": {
100
- "runId": "run_…", // the canonical earlier run
101
- "scenarioId": "scn_…", // its calculated scenario, when one is ready
102
- "similarity": 0.94, // cosine, ≥ SIMILARITY_THRESHOLD (server default 0.9)
103
- "executionTicket": "tkt_…", // redeem exactly once at POST /scenarios/:id/executions
104
- "scenario": { /* serializeScenario(): intent, paramsObject, paramsLogic,
105
- responseParamsLogic, steps[{ stepIndex, toolName,
106
- toolInputLogic, toolOutputLogic, reasoning }] */ }
107
- }
108
- }
109
- ```
110
-
111
- [`RemoteRecorder.getMatch(runId)`](../src/record/remote-recorder.ts) already exposes it, and
112
- [`ControlServer.watchForMatch`](../src/control/server.ts) already consumes it — to do exactly one
113
- thing:
114
-
115
- ```ts
116
- run.recording = false;
117
- logLine("run.matched", { …, why: "similar prompt — the service kept its own run; not recording this one" });
118
- ```
119
-
120
- **That is the whole of v1's replay support: a correct stop.** This design replaces the stop with a
121
- plan, and keeps the stop.
122
-
123
- ### 2.1 What "similar meaning" means, precisely
124
-
125
- Server-side, and worth stating because the threshold in §4 is a judgement about it:
126
-
127
- - The embedding is of the run's **`input`** — the user's prompt — computed by
128
- `Xenova/all-MiniLM-L6-v2` (`EMBEDDING_MODEL`), mean-pooled and normalized
129
- (`recordings/embeddings.ts`).
130
- - Candidates are **the caller's own canonical runs only** (`listUserRunEmbeddings(userId)`);
131
- duplicates and sub-threshold runs are excluded from the pool at record time.
132
- - The match is the single best cosine ≥ `SIMILARITY_THRESHOLD` (default **0.9**).
133
- - A matched run only carries a `scenarioId` when its scenario exists **and** is `state: 'ready'`.
134
-
135
- So "similar meaning" is: *the same user asked for something that embeds within 0.1 cosine of this,
136
- and we finished calculating a reusable plan for it.* It is not "the same words", and it is not
137
- cross-tenant.
138
-
139
- ---
140
-
141
- ## 3. Architecture
142
-
143
- ```mermaid
144
- graph TB
145
- subgraph host["Claude Code session"]
146
- M[model loop]
147
- H[hooks]
148
- end
149
-
150
- subgraph bir["BaseInstRunnerMCP"]
151
- C["bir-hooks — control server<br/>ScenarioReplayPlan lives here"]
152
- S["bir-scenario<br/>MCP server: run_scenario"]
153
- P1["bir-proxy — chrome-devtools"]
154
- P2["bir-proxy — github"]
155
- end
156
-
157
- U1[chrome-devtools-mcp]
158
- U2[github mcp]
159
- R[("BaseIn /recordings/* /scenarios/*")]
160
-
161
- H -->|"UserPromptSubmit → directive"| C
162
- H -->|"PreToolUse → pinned updatedInput"| C
163
- H -->|"PostToolUse → real output"| C
164
- M -->|"mcp__bir__run_scenario"| S
165
- S -->|"POST /scenario/run"| C
166
- C -.->|"work: tools/call"| P1
167
- C -.->|"work: tools/call"| P2
168
- P1 -->|"bir: id request"| U1
169
- P2 -->|"bir: id request"| U2
170
- C -->|"match + execution report"| R
171
- ```
172
-
173
- Three additions to the v1 topology, and nothing else moves:
174
-
175
- | Component | New? | Role in replay |
176
- |---|---|---|
177
- | `bir-hooks` control server | extended | Holds the `ScenarioReplayPlan` on `RunState`. Arms it, pins inputs, threads outputs, dispatches direct steps, redeems the ticket |
178
- | `bir-proxy` | extended | Long-polls for work; executes a `tools/call` on the upstream it already owns |
179
- | `bir-scenario` | **new**, ~120 lines | A stdio MCP server exposing one tool, `run_scenario`. The delivery channel for direct mode (§6) |
180
-
181
- ---
182
-
183
- ## 4. The gate ladder
184
-
185
- Ported from RRepeat's `applyMatch`, with one gate this repo needs and RRepeat cannot have.
186
- Every gate that declines logs **one** `replay.decision` line carrying the reason — because
187
- *"a scenario existed and was not used"* is a fact an auditor needs as much as *"a scenario ran"*.
188
-
189
- | # | Gate | Declines when | Outcome floor |
190
- |---|---|---|---|
191
- | 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched) |
192
- | 2 | **Replay enabled** | `BIR_REPLAY === "0"` | `not_steered` |
193
- | 3 | **Scenario ready** | `scenario == null`, `state !== "ready"`, or `steps.length === 0` | `not_steered` |
194
- | 4 | **Similarity** | `similarity < BIR_MIN_STEER_SIMILARITY` (default **0.92**) | `not_steered` |
195
- | 5 | **Tool coverage** | No step is executable anywhere (§5 → `mode: "none"`) | `not_steered` |
196
-
197
- Gate 4 deserves its own note. The server's own detection threshold is 0.9, and it uses it to decide
198
- *"do not record this again"* — a cheap claim: the worst case is a lost recording. Steering makes a
199
- much stronger claim: *"do not think about this again."* A prompt that is 0.90-similar usually covers
200
- only part of the live task, and steering it produces a confident, thin, wrong answer. So the steer
201
- threshold sits **above** the detection threshold by default, and `BIR_MIN_STEER_SIMILARITY` is the
202
- single knob an operator turns after watching `replay.decision` lines for a week.
203
-
204
- `not_steered` as the floor is not a failure code. It is the **control group**: the prompt matched,
205
- the agent ran the task the ordinary way, and the cost of doing so is a fresh measurement of what this
206
- task costs — which is exactly what a baseline is made of. BaseIn's
207
- `POST /scenarios/:id/executions` routes `not_steered` to `addBaselineSample` rather than the
208
- savings ledger, on purpose (`scenarios/routes.ts`). §11.
209
-
210
- ---
211
-
212
- ## 5. Tool coverage decides the mode
213
-
214
- The classification is a pure function of the scenario's steps and the set of server keys registered
215
- with this control server (`ControlServer.wrapped`, grown by `/proxy/register`).
216
-
217
- ```ts
218
- // src/replay/coverage.ts
219
- export type StepReach = "direct" | "live";
220
- export type ReplayMode = "direct" | "steer" | "none";
221
-
222
- export function reachOf(toolName: string, wrapped: ReadonlySet<string>): StepReach {
223
- const mcp = parseQualifiedName(toolName); // src/control/correlation.ts
224
- return mcp && wrapped.has(mcp.serverName) ? "direct" : "live";
225
- }
226
-
227
- export function modeFor(steps: SerializedScenarioStep[], wrapped: ReadonlySet<string>): ReplayMode {
228
- if (steps.length === 0) return "none";
229
- return steps.every((s) => reachOf(s.toolName, wrapped) === "direct") ? "direct" : "steer";
230
- }
231
- ```
232
-
233
- | Every step | Mode | Where the tools run | Model spend |
234
- |---|---|---|---|
235
- | a wrapped MCP tool | **`direct`** (§6) | The proxies, on connections already open | one Haiku derivation + one turn that reads the results |
236
- | anything else, or mixed | **`steer`** (§7) | The live session, inputs pinned | one Haiku derivation + the live turn |
237
- | — | `none` | nowhere | declines at gate 5 |
238
-
239
- `live` covers three genuinely different cases and treats them identically, correctly: a built-in
240
- (`Bash`, `Read`, `Edit`, `Grep`, `Task`, `WebFetch`), an MCP server the user chose not to wrap, and
241
- `claude-in-chrome` — which is `scope: "dynamic"`, appears in no config file, and therefore can never
242
- be wrapped (v1 design §1). None of them can be reached except through the session.
243
-
244
- A **mixed** scenario in `steer` mode still gets direct execution for its wrapped steps once it
245
- diverges (§8), so coverage is not all-or-nothing at the divergence boundary — only at the arm-time
246
- mode choice.
247
-
248
- ---
249
-
250
- ## 6. Mode A — direct execution
251
-
252
- The cheap rung. Every step is a wrapped MCP tool, so the model never has to emit a single tool call.
253
-
254
- ### 6.1 Sequence
255
-
256
- ```mermaid
257
- sequenceDiagram
258
- participant CC as Claude Code (model)
259
- participant CS as bir-hooks (control server)
260
- participant SC as bir-scenario (MCP)
261
- participant PX as bir-proxy
262
- participant UP as upstream MCP server
263
-
264
- CC->>CS: UserPromptSubmit
265
- CS->>CS: gates pass, mode=direct → arm plan; start derivation (not awaited)
266
- CS-->>CC: additionalContext: "call mcp__bir__run_scenario once"
267
- CC->>SC: tools/call run_scenario {}
268
- SC->>CS: POST /scenario/run
269
- CS->>CS: await plan.ready() (derivation lands)
270
- loop each step
271
- CS->>CS: toolInputForCurrentStep()
272
- CS-->>PX: work { workId, toolName, arguments } (answers an open long-poll)
273
- PX->>UP: tools/call (bir: id, on the open connection)
274
- UP-->>PX: CallToolResult
275
- PX->>CS: POST /proxy/result { workId, result }
276
- CS->>CS: applyOutput(JSON.stringify(result)) → respParams
277
- end
278
- CS->>CS: responseParamsLogic → response model; assembleBundle()
279
- CS-->>SC: { text, responseModel, steps }
280
- SC-->>CC: tool_result (genuine)
281
- CC->>CC: answers from the results
282
- CC->>CS: Stop → seal, report execution (ticket)
283
- ```
284
-
285
- ### 6.2 Why a first-party MCP server, and not `additionalContext`
286
-
287
- Executing inside `UserPromptSubmit` and returning the results as `additionalContext` is the obvious
288
- shortcut, and it is wrong for three separate reasons:
289
-
290
- 1. **Budget.** `UserPromptSubmit` has a hook timeout (§10). N live MCP round trips do not fit inside
291
- one, and if it expires Claude Code proceeds *without* the context — so the work is burned and the
292
- user waits for nothing.
293
- 2. **Trust.** RRepeat learned this expensively. Its `deny`-with-a-reason delivery is *"inherently
294
- read as a denial by the model"* (`hook-server.ts`), which is why it grew a Bash-rewrite path and
295
- then a whole MCP bridge (`mcp-bridge.ts`) to hand the same bytes back as a genuine `tool_result`.
296
- A first-party MCP tool is that bridge, without the shadowing.
297
- 3. **Truth.** A `tool_result` in the transcript is a real record of a real call. `additionalContext`
298
- asserting that tools ran is a claim the transcript cannot corroborate.
299
-
300
- The cost is honest and worth naming: one extra tool in the model's tool list, in **every** session
301
- where `bir install --replay` ran. That is why it is a separate install flag and why `bir status`
302
- prints it.
303
-
304
- ### 6.3 `run_scenario`
305
-
306
- ```jsonc
307
- // tools/list, from bir-scenario
308
- {
309
- "name": "run_scenario",
310
- "description":
311
- "Run the pre-calculated tool sequence BaseInstRunner has armed for the current request, and return its results. Call once, with no arguments, when instructed to.",
312
- "inputSchema": { "type": "object", "properties": {}, "additionalProperties": false }
313
- }
314
- ```
315
-
316
- No arguments **by design**: the armed plan is server-side state keyed by the control server's open
317
- run, so there is nothing for the model to get wrong and nothing it can point at a different scenario.
318
- When no plan is armed the tool returns `isError: true` with `"no scenario is armed for this turn"`,
319
- which is a tool failure the model reacts to normally.
320
-
321
- `bir-scenario` is a client of the control server, discovered the same way a proxy discovers it —
322
- `~/.baseinstrunner/control/<key>.json`, mode `0600`, bearer token
323
- ([`src/control/discovery.ts`](../src/control/discovery.ts)). It holds no state, no credentials and
324
- no upstream connections.
325
-
326
- ### 6.4 The steering directive (direct)
327
-
328
- ```
329
- [BaseInstRunner] A known-good tool sequence was recorded for this request
330
- (intent: <scenario.intent>). Call mcp__bir__run_scenario once, with no
331
- arguments, before any other tool. It runs the sequence and returns the
332
- results. Answer the user from those results.
333
- ```
334
-
335
- If the model calls something else first, the turn falls to §8 — the plan is still armed and the
336
- divergence path still executes it.
337
-
338
- ---
339
-
340
- ## 7. Mode B — model-steered live replay
341
-
342
- The mixed rung. Ported from RRepeat's `replay.md` §3/§12, which is the right design and is already
343
- proven: the model chooses *which* tool and *when*; the control server overrides the *arguments* and
344
- threads state between steps. The session never leaves the model's hands, so every
345
- `tool_use`/`tool_result` in the transcript is genuine and continuation is automatic.
346
-
347
- **Arm** at `UserPromptSubmit` — plan on `run.plan`, `run.planToolUse = new Set()`, directive via
348
- `additionalContext`, **no `decision`** so the prompt still reaches the model:
349
-
350
- ```
351
- [BaseInstRunner calculated replay] A known-good tool sequence was previously
352
- recorded for this request. Call exactly these tools, in order, one at a time:
353
-
354
- 1. mcp__chrome-devtools__navigate_page — open the fleet dashboard
355
- 2. mcp__chrome-devtools__take_snapshot — read the device table
356
- 3. Read — load the alert thresholds
357
-
358
- The system supplies the exact arguments for each call — you do not need to
359
- compute them. Do not call any other tools until this sequence is complete,
360
- then answer the user's request from the tool results.
361
- ```
362
-
363
- **Pin** at `PreToolUse`, while a plan is active and the call is not `mcp__bir__run_scenario`:
364
-
365
- ```ts
366
- if (toolName === plan.expectedTool()) {
367
- const input = plan.toolInputForCurrentStep(); // may throw → §8
368
- run.planToolUse.add(toolUseId);
369
- return { hookSpecificOutput: { hookEventName: "PreToolUse",
370
- permissionDecision: "allow",
371
- updatedInput: withCallId(input) } }; // §7.1
372
- }
373
- return divergence(…); // §8
374
- ```
375
-
376
- **Thread** at `PostToolUse`, for a `tool_use_id ∈ run.planToolUse` — but *from the right source*
377
- (§7.2). When `plan.isDone()`, retire the plan; the model then writes its answer from the real tool
378
- history.
379
-
380
- ### 7.1 Steering and correlation both write `updatedInput` — they must merge
381
-
382
- This collision does not exist in RRepeat and is easy to miss here.
383
-
384
- `onToolPre` already returns an `updatedInput` for every **wrapped MCP** call: it is how the
385
- correlation id rides along to the proxy ([`src/control/correlation.ts`](../src/control/correlation.ts)).
386
- A pinned step whose tool is a wrapped MCP tool therefore needs *both* payloads in one object, and the
387
- order is not arbitrary:
388
-
389
- ```ts
390
- /** Pinned arguments plus, for a wrapped MCP tool, the correlation id. */
391
- function withCallId(pinned: Record<string, unknown>): Record<string, unknown> {
392
- const mcp = parseQualifiedName(toolName);
393
- if (!mcp || !this.wrapped.has(mcp.serverName) || this.opts.noCorrelation) return pinned;
394
- const correlation = mintCorrelation(run, mcp, pinned, context); // hookArgs = the PINNED args
395
- return { ...pinned, [BIR_CALL_ID]: correlation.callId };
396
- }
397
- ```
398
-
399
- Three consequences, all load-bearing:
400
-
401
- - **The correlation's `hookArgs` must be the pinned arguments, not the model's.** `settle()` records
402
- `correlation.hookArgs ?? report.args`; recording the arguments the model *proposed* rather than the
403
- ones that *ran* would put a lie in the corpus. (In a replayed turn nothing is recorded — §12 — but
404
- the same code path serves recorded turns, and the invariant must hold there.)
405
- - **Schema relaxation must already be active**, or the host may reject the extra key. It is, whenever
406
- Tier 1 correlation is on (`ProxySession.correlationActive`). Under `--no-correlation`, `withCallId`
407
- is a no-op and pinning still works — only the *result* join degrades to fingerprints.
408
- - **`permissionDecision: "allow"` auto-approves.** Already true for every correlated call today
409
- (`BIR_CORRELATION_DECISION`), and now true for pinned built-ins too. §13.2.
410
-
411
- ### 7.2 Thread from the same source that recorded
412
-
413
- `toolOutputLogic` was authored by BaseIn's `analyzeRun` against the exact bytes stored in that step's
414
- `tool_output`. Feeding it a differently-shaped rendering of the same result does not fail loudly — it
415
- silently derives nothing, and every later step reading `respParams` produces a wrong input.
416
-
417
- In this repo those bytes have **two different origins**:
418
-
419
- | Step's tool | What was recorded | Therefore thread from |
420
- |---|---|---|
421
- | Wrapped MCP | `serializeCapped(redact(report.result))` — the proxy's whole `CallToolResult`, via `settle()` → `record()` | the **proxy's `/proxy/step` report** |
422
- | Built-in / unwrapped MCP | `serializeCapped(redact(payload.tool_response))` — Claude Code's rendering, via `onToolPost` | the **hook's `tool_response`** |
423
-
424
- So `/tool/post` must not thread unconditionally. It threads for a `live` step, and for a `direct`
425
- step it does what it already does for a correlated call: **arm the existing
426
- `PROXY_REPORT_GRACE_MS` fallback**, and let `onProxyStep` thread when the report lands.
427
-
428
- ```ts
429
- // onToolPost, for a pinned step
430
- if (reachOf(step.toolName, this.wrapped) === "live") {
431
- plan.applyOutput(serializeCapped(payload.tool_response ?? {}));
432
- } else {
433
- correlation.threadOnReport = true; // onProxyStep calls plan.applyOutput(report.result)
434
- correlation.fallback = setTimeout(() => { // …unless the proxy never reports
435
- logLine("replay.thread_fallback", { why: "no proxy report — threading the hook's view; shapes may differ" });
436
- plan.applyOutput(serializeCapped(payload.tool_response ?? {}));
437
- }, PROXY_REPORT_GRACE_MS);
438
- }
439
- ```
440
-
441
- This reuses machinery that exists and is already correct. It also means the §7.3 ordering constraint
442
- binds a little harder: the proxy report must land before the next `PreToolUse`, and
443
- `PROXY_REPORT_GRACE_MS` (1500 ms) is the bound on how long that can take before we degrade.
444
-
445
- ### 7.3 Synchronous `PostToolUse` is a requirement, not a preference
446
-
447
- Step *N*'s `/tool/post` must finish updating `respParams` before step *N+1*'s `/tool/pre` reads it.
448
- In [`HOOK_ROUTES`](../src/config/adapters/claude-code.ts) neither `PreToolUse` nor `PostToolUse`
449
- carries `async: true`, so this already holds — **and must never be relaxed**. Add it to the comment
450
- block there, next to the `SessionEnd` note that records the same class of lesson.
451
-
452
- ---
453
-
454
- ## 8. Divergence
455
-
456
- The model called a tool the plan did not expect, or `toolInputLogic` threw, or the tool had no
457
- `tool_use_id`. Three tiers, tried in order. Here is where the §0 asymmetry pays.
458
-
459
- **Tier 1 — direct-execute the remainder (new; the common case).**
460
- Take every not-yet-run step, execute it, and hand the composed result back. Steps with
461
- `reach === "direct"` run through their proxy. Steps with `reach === "live"` cannot run here at all —
462
- a hook process is not the session — and fall back to the step's **recorded output** (§8.1), or are
463
- skipped with a logged warning. RRepeat's rule, and its reasoning holds verbatim: a *skipped* step
464
- also stops threading `respParams`, so every later step reading from it fails too, and a five-step
465
- scenario degrades to nothing. A recorded result is marked as such in the bundle, so the model is
466
- never told a stale result is fresh.
467
-
468
- Delivery, best channel first:
469
-
470
- 1. **The `bir` tool** — if the divergent call *is* `mcp__bir__run_scenario`, or the model can be
471
- nudged to it. Genuine `tool_result`.
472
- 2. **Bash-clean** — if the divergent call is `Bash`, rewrite the command to `cat` a heredoc of the
473
- bundle. The model sees ordinary command output. (RRepeat's `bashCleanPath`; neutralize the
474
- delimiter first.)
475
- 3. **Deny-inject** — `permissionDecision: "deny"`, bundle as `permissionDecisionReason`, capped at
476
- `MAX_REPLAY_REASON` (60 000 chars) with the same fair-share truncation
477
- (`assembleBundle`/`truncate`, portable verbatim from `payload-executor.ts`). Works for any tool,
478
- and is read as adversarial — hence last.
479
-
480
- **Tier 2 — abort.** Composition itself threw. Retire the plan, return `{}`, continue as an ordinary
481
- turn. Outcome `failed`.
482
-
483
- **The cost line.** RRepeat's Tier 1 is an Agent SDK session, and its own doc calls that the dominant
484
- cost of a calculated run. Here Tier 1 is N `tools/call`s on connections that are already open:
485
- **`fallbackCostUsd` is 0.00 by construction**, for every step the proxies can reach.
486
-
487
- > **Do not port the unconditional-divergence line.** RRepeat's `steerPreTool` ends with a deliberate
488
- > `return await injectBundle(...)` that bypasses the pin branch on *every* armed plan
489
- > (`savingCalcRunner.md` §4 documents the consequence: `stepsPinned` is 0 on every row). That was a
490
- > choice made when the bundle was the only path that ran real tools. It is not this design's choice:
491
- > pinning is tried first, always, and divergence is what its name says.
492
-
493
- ### 8.1 Where a recorded output actually comes from
494
-
495
- RRepeat reads it from `SerializedScenarioStep.recordedOutput`, whose own comment says the field is
496
- optional because *"it is the server's payload that decides whether to send it, and older payloads do
497
- not."* Verify this before relying on it: **no payload does.** BaseIn's `scenario_steps` table has
498
- columns `tool_input_logic`, `tool_output_logic`, `reasoning`, `reasoning_vector` — and no
499
- `recorded_output`; `serializeScenario` emits no such field. RRepeat's fallback is unreachable in
500
- production today.
501
-
502
- There is a path that needs no server change, and the match payload already carries its key:
503
-
504
- ```
505
- GET /recordings/runs/:runId → { run, steps } // steps carry tool_name + tool_output
506
- ↑ matched.runId — the canonical source run this scenario was calculated from
507
- ```
508
-
509
- So: fetch the source run **lazily**, on the first `live` step of the first divergence, and index its
510
- `tool_response` steps by tool name into the same FIFO queue BaseIn's own `dryReplay` builds
511
- (`recordedOutputsByTool` in `scenarios/replay.ts` — a `Map<toolName, string[]>` drawn in order, with
512
- a per-tool cursor). One authenticated GET, cached for the turn, and the queue semantics match the
513
- server's dry replay exactly — which matters, because a scenario that behaves one way under
514
- `bir scenario replay --dry` and another way here would be untestable.
515
-
516
- Adding `recorded_output` to `scenario_steps` would make this a field read instead of a fetch, and is
517
- a reasonable server-side follow-up. It is not a prerequisite.
518
-
519
- > **[built]** [`src/replay/source-run.ts`](../src/replay/source-run.ts). The type still carries the
520
- > optional `recordedOutput` field and prefers it when present, so a server that starts sending it
521
- > needs no change here — but nothing depends on it.
522
-
523
- ---
524
-
525
- ## 9. The plan, and state threading across hook calls
526
-
527
- `ScenarioReplayPlan` ports from RRepeat's [`payload-executor.ts`](../../../BasInstRunner/RRepeat/src/server/payload-executor.ts)
528
- with three changes.
529
-
530
- ```ts
531
- // src/replay/plan.ts
532
- export class ScenarioReplayPlan {
533
- readonly scenarioId: string;
534
- readonly mode: ReplayMode;
535
- readonly stepCount: number;
536
-
537
- /** Resolves when derivation has landed and paramsLogic has been applied. */
538
- ready(): Promise<void>;
539
-
540
- currentStepIndex: number;
541
- expectedTool(): string | undefined;
542
- isDone(): boolean;
543
-
544
- /** Pinned input for the current step. Throws → divergence. */
545
- toolInputForCurrentStep(): Record<string, unknown>;
546
- /** Thread a real output into respParams and advance. Throws → abort. */
547
- applyOutput(realOutput: string): void;
548
-
549
- steeringDirective(): string;
550
- /** Execute every remaining step through `execute`, and lay the results out under `maxChars`. */
551
- composeBundle(maxChars: number, execute: ExecuteStep): Promise<BundleResult>;
552
- /** responseParamsLogic over the accumulated respParams. */
553
- responseModel(): Record<string, unknown>;
554
- }
555
- ```
556
-
557
- **Change 1 — lifetime is a run, not a session.** The plan hangs off `RunState`, not `SessionState`.
558
- In this codebase a run *is* a turn (`UserPromptSubmit → Stop`, `server.ts` §onPrompt), which is
559
- exactly the plan's lifetime. `sealRun` retires it; a second prompt in the same session cannot inherit
560
- a stale plan, which in RRepeat requires care.
561
-
562
- **Change 2 — `params` are a promise.** The constructor takes `Promise<Record<string, unknown>>`, not
563
- a value, so arming does not wait for the Haiku call (§10). `paramsLogic` is applied once, inside
564
- `ready()`, the first time it resolves.
565
-
566
- **Change 3 — `execute` is a step, not a tool name.** `composeBundle` takes
567
- `(step, input) => Promise<string>` rather than `(toolName, input)`, because the executor must know
568
- the step's `reach` to decide between the proxy, the recorded output, and skipping.
569
-
570
- The four `new Function` shims (`evalParamsLogic`, `evalToolInputLogic`, `evalToolOutputLogic`,
571
- `evalResponseParamsLogic`) go in `src/replay/logic.ts`, ported from BaseIn's
572
- [`src/scenarios/logic.ts`](../../../BaseIntSever/BaseIn/src/scenarios/logic.ts) — which is the same
573
- code with better comments and an `assertPlainObject` guard the other copies lack. **Port BaseIn's
574
- version**, not RRepeat's: it is the one the server evaluates against, so a divergence between them is
575
- a divergence between what a scenario does in a dry replay and what it does here.
576
-
577
- ### 9.1 What flows between steps
578
-
579
- - **`params`** — computed once (derivation → `paramsLogic`), read by every `toolInputLogic`.
580
- - **`respParams`** — starts `{}`; each step's `toolOutputLogic` merges keys in
581
- (`respParams = { ...respParams, ...derived }`); later steps' `toolInputLogic` read the accumulation.
582
- This is how step 1's `highestRiskDeviceId` reaches step 2's input.
583
-
584
- In direct mode both live inside one `for` loop. In steer mode they are split across *separate HTTP
585
- requests* — which is what §7.3's ordering rule protects.
586
-
587
- ---
588
-
589
- ## 10. Timing: what may block, and for how long
590
-
591
- The hook is on the user's critical path. Every await here is a stall the user watches.
592
-
593
- | Moment | Route | Hook timeout | What may be awaited | Budget |
594
- |---|---|---|---|---|
595
- | `UserPromptSubmit` | `/session/prompt` | **5 s → raise to 15 s** | `getMatch()` only | `BIR_MATCH_BUDGET_MS` = **2500** |
596
- | `PreToolUse` | `/tool/pre` | 30 s (unchanged) | `plan.ready()` (derivation) | `BIR_DERIVE_BUDGET_MS` = **8000** |
597
- | `run_scenario` | `/scenario/run` | MCP client's own | `ready()` + N steps | `BIR_REPLAY_BUDGET_MS` = **120000**, `BIR_STEP_TIMEOUT_MS` = **60000** |
598
- | `Stop` | `/session/stop` | 10 s | nothing (report is queued) | — |
599
-
600
- **Derivation is deliberately not awaited at the prompt.** The steering directive needs only the
601
- scenario's ordered tool names and reasonings — it does not need parameter values. So the prompt hook
602
- returns the directive the moment the match lands, and the Haiku call runs un-awaited; the *first*
603
- `PreToolUse` awaits it, inside a 30-second budget instead of a 15-second one. On a mode-`direct` turn
604
- the wait lands inside `run_scenario` instead, where there is no hook timeout at all.
605
-
606
- **Budget expiry is never an error.** Match budget expired → no plan, ordinary turn, one
607
- `replay.decision` line. Derivation budget expired → fall back to the scenario's recorded
608
- `sampleValue`s (D15) and keep going; a replay with stale parameters is worth more than no replay, and
609
- the audit line says which happened.
610
-
611
- **The one hook-config change** is `UserPromptSubmit`'s timeout, `5` → `15`, in
612
- [`HOOK_ROUTES`](../src/config/adapters/claude-code.ts). It must stay non-`async`: an async hook is
613
- fire-and-forget and cannot return `additionalContext` at all.
614
-
615
- ---
616
-
617
- ## 11. Reporting: outcome, ticket, cost
618
-
619
- This is the half that makes replay *provable* rather than merely fast, and v1 does none of it.
620
-
621
- ### 11.1 The outcome ladder
622
-
623
- ```ts
624
- // Upgrade-only. `not_steered` is the floor and the safe answer if the turn dies mid-flight.
625
- const OUTCOME_RANK = { not_steered: 0, failed: 1, diverged: 2, steered_full: 3 } as const;
626
- ```
627
-
628
- | Outcome | Means | Server-side effect |
629
- |---|---|---|
630
- | `not_steered` | Matched, declined at a gate; the agent ran the task | **baseline sample** (`addBaselineSample`) |
631
- | `failed` | Armed, then the machinery gave up; the agent ran the task | **baseline sample** |
632
- | `diverged` | The model went off-script; the bundle answered | ledger row, `saved_usd` computed |
633
- | `steered_full` | Every planned step ran with scenario-derived inputs | ledger row, `saved_usd` computed |
634
-
635
- Direct mode reports `steered_full`. BaseIn's zod enum
636
- (`['steered_full','diverged','not_steered','failed']`) has no `direct` member, and `steered_full`
637
- means exactly what happened: every planned step executed under the scenario's control and the
638
- response model was produced. A `direct` discriminator would be a *server* change and is §20's
639
- question 2, not a blocker.
640
-
641
- ### 11.2 The ticket
642
-
643
- `matched.executionTicket` is the server's claim token, and **it is the execution row's id** — which
644
- is what makes a doubled report book once (`scenarios/routes.ts`: a `claimed` ticket returns
645
- `200 { duplicate: true }`, not an error). Redeem it exactly once, in `sealRun`, pushed onto
646
- `pendingSeals` alongside the answer resolution so `Stop` never blocks on the network:
647
-
648
- ```
649
- POST /scenarios/:scenarioId/executions
650
- { ticket, outcome, deriveCostUsd, sessionCostUsd, fallbackCostUsd,
651
- durationMs, stepsPlanned, stepsPinned, measured, pricingVersion, prompt,
652
- steps, error, errorStage, errorStepIndex, errorToolName }
653
- ```
654
-
655
- An **unredeemed** ticket leaves no execution row at all. That asymmetry is intentional and worth
656
- preserving: silence loses a saving; it never invents one.
657
-
658
- ### 11.2a What each step did (the service's errorshandling.md)
659
-
660
- The last five fields are the diagnosis, and none of the money depends on them. The service's
661
- recording page draws the scenario's chain step by step; without these, a scenario whose step 2
662
- throws on every replay renders as four healthy-looking steps beside a saving that never grows. The
663
- error exists — in a log line, on the operator's machine. The person who owns the scenario is looking
664
- at a web page.
665
-
666
- ```ts
667
- steps: Array<{
668
- stepIndex: number; // the scenario's own index, not a position in this array
669
- toolName: string;
670
- status: "ok" | "recorded" | "skipped" | "failed";
671
- stage?: ExecutionStage; // where in the step, for `failed` / `skipped`
672
- error?: string; // message only — this is rendered in a web page
673
- durationMs?: number;
674
- }>
675
- ```
676
-
677
- `ok` and `recorded` are passes (`recorded` with an asterisk: the values came from the source run).
678
- `skipped` did not run. **`failed` is the only one that means the scenario is broken** rather than
679
- the environment — it is the scenario's own logic throwing, and it is what `Recalculate` fixes.
680
-
681
- Three rules this must keep:
682
-
683
- 1. **A step the turn never reached is absent**, not `skipped`. The page shows those as "not run",
684
- and a turn that dies at step 2 of 5 must not paint steps 3-5 red.
685
- 2. **Every field is optional on the wire.** An older service ignores them; the costs still book.
686
- 3. **The headline and the steps are one statement.** `error*` is lifted from the first `failed`
687
- step (`blameStep`), never assembled independently, so the page's card cannot contradict the step
688
- list underneath it.
689
-
690
- The verdicts come from `plan.ts`'s `onStep`, which has always known them — with one addition: a
691
- logic failure is announced *before* the throw propagates (`reportingStage`), because after it
692
- nothing knows which step the stack trace belonged to. In steer mode there is no loop to observe, so
693
- the controller records the verdict at both hooks: `PreToolUse` for an input-logic throw, `PostToolUse`
694
- for the pass or the output-logic throw. `ReplayState.stepResults` is keyed by step index so the
695
- second word on a step replaces the first.
696
-
697
- One consequence for the rule that a costless decline is not worth sending (`buildReport`, and the
698
- service's matching `202 { recorded: false }`): it now *is* worth sending when a step broke. No cost
699
- **and** no error remains nothing to report.
700
-
701
- ### 11.3 Cost, and a live pricing hazard
702
-
703
- | Component | Source | Notes |
704
- |---|---|---|
705
- | `deriveCostUsd` | our own Haiku call's `usage`, priced locally | the only tokens replay itself spends |
706
- | `sessionCostUsd` | transcript usage **delta** for this run (§11.4) | the live turn: prompt, tool calls, answer |
707
- | `fallbackCostUsd` | **0.00** | direct execution spends no tokens (§8) |
708
-
709
- > ### ⚠ Verify the pricing table before copying it
710
- >
711
- > A saving is `baseline − actual`, where the baseline is priced by BaseIn's
712
- > [`src/scenarios/cost.ts`](../../../BaseIntSever/BaseIn/src/scenarios/cost.ts) and the actual by the
713
- > runner's copy. Both currently declare `PRICING_VERSION = '2026-08-30'`, and **the two tables
714
- > already disagree**: RRepeat prices `claude-opus-4-6` at `3.0 / 15.0`, BaseIn at `5.0 / 25.0`. Equal
715
- > versions are supposed to mean *"these are comparable"*. Right now, for that row, they are not.
716
- >
717
- > So: **copy BaseIn's table, not RRepeat's** — the server owns the baseline, and the baseline is the
718
- > number a saving is measured *against*. Copy `PRICING_VERSION` with it. Treat any price change as a
719
- > three-repository change, and file the drift above against RRepeat rather than reproducing it.
720
- >
721
- > Practical exposure here is small — derivation runs on `claude-haiku-4-5-20251001`, which both
722
- > tables price identically at `1.0 / 5.0` — but the invariant is what matters, not today's blast
723
- > radius.
724
-
725
- ### 11.4 The per-run usage watermark
726
-
727
- `savingCalcRunner.md` §3 documents this trap in RRepeat, and **BaseInstRunnerMCP has it too**, for
728
- the same structural reason: Claude Code writes one transcript per *session*, and `onPrompt` rolls a
729
- new run on every prompt. Summing the whole file gives run 3 the cost of runs 1+2+3.
730
-
731
- Every one of those numbers is a **baseline** on the server. Do not ship the whole-file version.
732
-
733
- ```ts
734
- // src/control/transcript.ts — two new exports beside contextForToolUse / settledLastAssistantText
735
- /** Per-model cumulative usage, captured when a run starts. */
736
- export function markTranscriptUsage(path: string | undefined): UsageMark;
737
- /** Per-model delta since `mark`, clamped at 0 per field. */
738
- export function usageSince(path: string | undefined, mark: UsageMark): ModelUsage[];
739
- ```
740
-
741
- Take the mark in `ensureRun`, beside `startedAtMs`, so the cost window and the duration window are
742
- the same window. Clamp per field, so a compacted or rotated transcript degrades to *"no usage"*
743
- rather than to a negative cost. A run with **no mark** (one opened lazily by a proxy step before any
744
- prompt) reports `measured: false` — better to say "unmeasured" than to hand the ledger an inflated
745
- baseline.
746
-
747
- `measured` is `true` only when a mark existed **and** `pricingVersion` matches the server's. The
748
- server enforces the second half itself and stores a mismatched report as unmeasured rather than
749
- trusting it.
750
-
751
- ---
752
-
753
- ## 12. What the replayed turn records
754
-
755
- **Nothing, and that is correct.** The service answered `{ id: null }`; no run exists; every step post
756
- would 404. `run.recording = false` already handles it — `record()` early-returns and `sealRun` skips
757
- both the final answer and `finishRun`.
758
-
759
- Three consequences to keep straight:
760
-
761
- 1. **Step indices are still allocated.** `ordering` runs regardless, correlations are still minted
762
- and settled, and `bir doctor` still shows a live step count. Only the *sending* stops.
763
- 2. **The matched run is the canonical one.** The server bumped its `iterations` at match time. The
764
- corpus does not grow on a replay, by design — that is what stops one popular task from flooding
765
- the similarity pool with near-duplicates.
766
- 3. **Direct-mode calls bypass the recorder entirely.** They are issued via `UpstreamClient.request`,
767
- which allocates a `bir:` id and consumes its own response in `accept()` before the relay's
768
- handlers ever see it ([`src/upstream/client.ts`](../src/upstream/client.ts)). So
769
- `RecordingInterceptor` never observes them and cannot double-record. That interface comment says
770
- `request()` is *"Never used by the relay"* — it was built for `bir doctor` and tests, and this is
771
- its second, load-bearing user.
772
-
773
- ### 12.1 `bir replay` — the Tier 2 and debugging path
774
-
775
- ```bash
776
- bir replay --scenario scn_… --prompt "check the fleet for high-risk devices" # run it
777
- bir replay --scenario scn_… --prompt "…" --dry # derive + plan only
778
- ```
779
-
780
- Runs the same plan through the same executor, against the same live proxies, printing the per-step
781
- input/output and the response model. It is how you test a scenario without a session, how a Tier 2
782
- or non-Claude-Code client gets any replay at all, and the first thing to reach for when a steered
783
- turn behaves oddly.
784
-
785
- ---
786
-
787
- ## 13. Security
788
-
789
- v1 §9 holds unchanged. Replay adds two exposures, and neither is small.
790
-
791
- ### 13.1 The scenario payload is executable code
792
-
793
- `toolInputLogic`, `toolOutputLogic`, `paramsLogic` and `responseParamsLogic` are JavaScript function
794
- *bodies*, compiled with `new Function` and run **in the control server process** — a process holding
795
- the user's BaseIn credentials and, through the proxies, live authenticated access to every wrapped
796
- MCP server.
797
-
798
- BaseIn's own `logic.ts` argues the eval is safe because *"these run only over data the owner already
799
- produced… so the dynamic eval is scoped to first-party content."* That argument holds here too, and
800
- the blast radius does not: BaseIn evaluates them against a database row, we evaluate them next to a
801
- browser and a GitHub token.
802
-
803
- What actually constrains it:
804
-
805
- - The payload arrives **only** in the response to an authenticated `POST /recordings/runs` against
806
- the configured `BIR_AUTH_URL`, over the user's own token, describing the user's own runs. It is not
807
- attacker-supplied in any ordinary threat model.
808
- - A `new Function` body gets its declared arguments **and the global scope**. `require` and `import`
809
- are not reachable (ESM, no CommonJS wrapper), but `fetch`, `process` and `globalThis` are. Say this
810
- plainly rather than implying a sandbox: **there is no sandbox.**
811
- - `BIR_REPLAY` is opt-in (D14), so a compromised or misconfigured `BIR_AUTH_URL` cannot start
812
- executing logic on a machine whose operator never asked for replay.
813
-
814
- If that is not enough for a deployment, the mitigation is a real one and is out of scope here:
815
- evaluate the logic in a `node:vm` context with a frozen, minimal global. Filed as §20 question 3.
816
-
817
- **A hand-edited step is the same code, and runs the same way.** Since 2026-09-25 (the service's
818
- `editSteps.md`, not yet released) the owner — or Claude working for them — can change a step's
819
- logic with `bir scenario edit`. The service checks the change against the recording before it saves
820
- it, but that check is about *what the code computes*. Once saved, the step arrives in the same
821
- payload and runs here unattended, with `new Function`, next to the same credentials, exactly like
822
- calculated code. The provenance argument still holds: only the owner can edit, over their own token,
823
- and anyone else — an admin included — gets `404`. What changes is that *"the model wrote it"* is no
824
- longer always true.
825
-
826
- The service now sandboxes its own evaluation of every logic body: one worker thread with an empty
827
- environment and a memory cap, a fresh `node:vm` context per evaluation with a time limit and nothing
828
- but JavaScript's own built-ins, values in and out as JSON text only. It had to: there, one account's
829
- code runs in the process that holds every account's data and the service's secrets. **The runner
830
- still does not sandbox.** Here the code runs on the owner's machine, for the owner's account. One
831
- consequence to know: code that uses `fetch`, `process` or a timer throws in the service's check, so
832
- it cannot be saved without `--force`; a forced step would run here with all of them.
833
-
834
- ### 13.2 Replay bypasses permission prompts
835
-
836
- This is the sharpest edge in the whole design, and it must not be buried.
837
-
838
- - **Steer mode** answers `PreToolUse` with `permissionDecision: "allow"`, so a pinned call is
839
- auto-approved. The `BIR_CORRELATION_DECISION` note in `bir-hooks.ts` already documents this for
840
- correlated MCP calls; steering extends it to **built-ins** — including `Bash` and `Edit`.
841
- - **Direct mode** never reaches `PreToolUse` at all. A `tools/call` dispatched to a proxy is not
842
- visible to Claude Code's permission system in any form.
843
-
844
- So a scenario is, in effect, a **pre-approved list of tool calls with computed arguments**. That is
845
- precisely what makes it valuable, and precisely what makes it dangerous. Four mitigations, all
846
- required:
847
-
848
- 1. `bir install --replay` a distinct, non-default flag; `BIR_REPLAY=0` a one-line kill switch once installed.
849
- 2. `BIR_REPLAY_ALLOW_SERVERS` — a comma-separated allowlist of server keys eligible for **direct**
850
- execution. Unset means *all wrapped servers*; the guide (§4 of `calculatedReplayGuide.md`)
851
- recommends setting it.
852
- 3. `bir doctor` prints `replay: on` and the effective allowlist, loudly, at the top. A capability this
853
- sharp must never be discoverable only by reading a log.
854
- 4. Every armed plan logs `plan.armed` with the scenario id, mode, step count and the ordered tool
855
- names **before** the first call runs. The audit line precedes the action.
856
-
857
- ### 13.3 The control plane
858
-
859
- `/scenario/run`, `/proxy/poll` and `/proxy/result` are ordinary control-server routes: loopback only,
860
- bearer token from the `0600` discovery file, same as `/proxy/step`. `/proxy/poll` holds a connection
861
- open for up to 25 s — a long-lived authenticated request, not a new trust boundary.
862
-
863
- ---
864
-
865
- ## 14. Degradation matrix
866
-
867
- The governing rule is unchanged and outranks every optimisation here: **the host session must never
868
- fail because of BaseInstRunner.**
869
-
870
- | Failure | Behaviour |
871
- |---|---|
872
- | `getMatch()` slower than the match budget | No plan. Ordinary turn. One `replay.decision` line |
873
- | No `ANTHROPIC_API_KEY` | The service derives instead (segmented.md R-PARAM-5). Signed out as well: settings take their recorded `sampleValue`s, a scenario with a target declines `no_derive_key`, `deriveCostUsd: 0` |
874
- | The service declines, is slow, or cannot be reached | The same, with its reason carried into the `replay.derived` and `replay.decision` lines. Never a throw |
875
- | Derivation call fails or times out | Same as above, plus a `replay.derive_failed` line |
876
- | `toolInputLogic` throws | Divergence (§8) — never a crash |
877
- | `toolOutputLogic` throws | Retire the plan, abort to an ordinary turn, outcome `failed` |
878
- | Model calls an unexpected tool | Divergence (§8) |
879
- | A proxy has no open long-poll (dead, or `BIR_REPLAY` unset in its env) | That step falls back to `recordedOutput`, else is skipped with a warning |
880
- | A direct step exceeds `BIR_STEP_TIMEOUT_MS` | Same as above; the work item is abandoned server-side |
881
- | Whole plan exceeds `BIR_REPLAY_BUDGET_MS` | Return what completed, marked partial; outcome `diverged` |
882
- | `bir-scenario` cannot find the control server | `run_scenario` returns `isError: true`; the model proceeds normally |
883
- | Ticket redemption fails (offline, 409) | Logged; the turn is unaffected. An unredeemed ticket books nothing |
884
- | Two reports for one ticket (retry, `Stop` racing `SessionEnd`) | Server books once, answers `200 { duplicate: true }` |
885
- | `BIR_REPLAY` unset | Exactly v1: stop recording on a match, do nothing else |
886
-
887
- ---
888
-
889
- ## 15. Solution layout
890
-
891
- ```
892
- src/
893
- ├─ replay/ ← new
894
- │ ├─ plan.ts # ScenarioReplayPlan (§9)
895
- │ ├─ logic.ts # the four new Function shims — port BaseIn's (§9)
896
- │ ├─ derive.ts # Haiku parameter derivation over raw fetch (§16.3)
897
- │ ├─ pricing.ts # MODEL_PRICING + PRICING_VERSION — copy BaseIn's (§11.3)
898
- │ ├─ coverage.ts # reachOf / modeFor (§5)
899
- │ ├─ bundle.ts # assembleBundle / truncate — port verbatim
900
- │ ├─ executor.ts # direct step dispatch + the work queue (§16.2)
901
- │ ├─ source-run.ts # lazy GET /recordings/runs/:id → recorded outputs by tool (§8.1)
902
- │ └─ types.ts # SerializedScenario, SerializedScenarioStep
903
- ├─ bin/
904
- │ ├─ bir-scenario.ts ← new: the one-tool MCP server (§6.3)
905
- │ └─ bir.ts # + `--replay`, `bir scenario …`, `bir replay` (§16.5)
906
- ├─ control/
907
- │ ├─ server.ts # + arm / pin / thread / dispatch / report
908
- │ └─ transcript.ts # + markTranscriptUsage / usageSince (§11.4)
909
- ├─ proxy/
910
- │ └─ session.ts # + the long-poll work loop (§16.2)
911
- ├─ record/
912
- │ ├─ recorder.ts # + reportExecution() on the Recorder surface
913
- │ └─ remote-recorder.ts # + POST /scenarios/:id/executions
914
- └─ config/adapters/claude-code.ts # UserPromptSubmit timeout 5 → 15; the `bir` MCP entry
915
- ```
916
-
917
- `package.json` gains one bin: `"bir-scenario": "dist/bin/bir-scenario.js"`. Runtime dependencies stay
918
- at **zero** (§16.3).
919
-
920
- ---
921
-
922
- ## 16. Contracts
923
-
924
- ### 16.1 Control-plane routes (new)
925
-
926
- ```ts
927
- // bir-scenario → control server
928
- POST /scenario/run {}
929
- → 200 { ok: true, text: string, responseModel: object, steps: number, partial?: boolean }
930
- → 200 { ok: false, why: "no_plan" | "not_ready" | "budget_exceeded" }
931
-
932
- // proxy → control server (long-poll; holds up to 25 s, then answers empty)
933
- POST /proxy/poll { serverName: string }
934
- → 200 {} | 200 { work: { workId: string; toolName: string; arguments: unknown; timeoutMs: number } }
935
-
936
- // proxy → control server
937
- POST /proxy/result { workId: string, result?: unknown, error?: string }
938
- → 200 { ok: true }
939
- ```
940
-
941
- ### 16.2 The direct executor
942
-
943
- ```ts
944
- // src/replay/executor.ts
945
- export type ExecuteStep = (step: SerializedScenarioStep, input: Record<string, unknown>) => Promise<string>;
946
-
947
- /** Hands work to the proxy that owns `serverName` and awaits its result. */
948
- export class ProxyWorkQueue {
949
- /** Called by POST /proxy/poll. Resolves with work, or with undefined at the poll deadline. */
950
- waitForWork(serverName: string, pollDeadlineMs: number): Promise<Work | undefined>;
951
- /** Called by POST /proxy/result. */
952
- complete(workId: string, result?: unknown, error?: string): void;
953
- /** Dispatch and await one call. Rejects on timeout or on no registered proxy. */
954
- call(serverName: string, toolName: string, args: unknown, timeoutMs: number): Promise<unknown>;
955
- }
956
- ```
957
-
958
- > **[built] "No poller parked" is not the same as "no proxy".** The first implementation rejected
959
- > `call` immediately whenever no poller was waiting, on the reasoning that queued work could stall a
960
- > turn past its budget. That reasoning holds for a server no proxy serves — and is wrong for the case
961
- > that actually dominates: a proxy is *between* polls precisely while it is POSTing one step's result
962
- > and re-polling, which is exactly the gap between two steps of a multi-step plan. The end-to-end test
963
- > caught it as "step 2 of 2 never ran".
964
- >
965
- > So the queue tracks `lastSeen` per server. Work dispatched while a recently-seen proxy is
966
- > mid-round-trip is **held** for its next poll; the per-call timeout still bounds the wait, and a
967
- > server nothing has served for a minute still fails fast into the recorded-output fallback.
968
-
969
- Proxy side, in `ProxySession` — started after a successful `register()`, only when `BIR_REPLAY=1`:
970
-
971
- ```ts
972
- while (!closed) {
973
- const work = await control.poll(this.serverName); // 25 s long-poll
974
- if (!work) continue;
975
- try {
976
- // The reserved `bir:` id space, built for exactly this and never used by the relay.
977
- const result = await this.upstream.request("tools/call",
978
- { name: work.toolName, arguments: work.arguments },
979
- AbortSignal.timeout(work.timeoutMs));
980
- await control.result(work.workId, result);
981
- } catch (err) {
982
- await control.result(work.workId, undefined, errText(err));
983
- }
984
- }
985
- ```
986
-
987
- Note `work.toolName` is the **upstream-local** name (`navigate_page`), not the qualified one —
988
- `parseQualifiedName` splits it at dispatch. Storing both parts is what v1 §5 bought.
989
-
990
- > **[built] The proxy needs no replay configuration of its own.** `/proxy/register` answers
991
- > `{ replay: boolean, pollHoldMs }`, and the loop opens only if that says so. `BIR_REPLAY` is
992
- > therefore a single switch on the control server, with no second copy in each wrapped entry's `env`
993
- > to drift out of step with it.
994
-
995
- ### 16.3 Derivation without a dependency
996
-
997
- ```ts
998
- // src/replay/derive.ts — no SDK; ~50 lines of fetch.
999
- export async function deriveParameters(opts: {
1000
- prompt: string;
1001
- intent: string;
1002
- paramsObject: Record<string, { sampleValue: unknown; description: string }> | null;
1003
- apiKey?: string; // ANTHROPIC_API_KEY — an override
1004
- service?: { baseUrl: string; token: string; scenarioId: string }; // the default path
1005
- model?: string; // default claude-haiku-4-5-20251001
1006
- signal?: AbortSignal;
1007
- }): Promise<{ params: Record<string, unknown>; costUsd: number; derived: boolean }>;
1008
- ```
1009
-
1010
- Behaviour, matching RRepeat's `deriveParametersFromScenarioPayload` exactly so the two produce the
1011
- same parameters from the same prompt:
1012
-
1013
- - No `paramsObject` keys → `{}`, `costUsd: 0`.
1014
- - **No `apiKey` but a `service` → `POST {baseUrl}/scenarios/{id}/derive`** with the turn (prompt,
1015
- live call, recent results) and a bearer token, `derived` as the service answered. This is the
1016
- ordinary path: nearly every session runs inside Claude Code on a subscription and has no key of
1017
- its own (segmented.md R-PARAM-5). The answer is re-normalised here against this runner's own copy
1018
- of the schema, so the gate reads a `missing` computed from what will actually run.
1019
- - **Neither → return every key's recorded `sampleValue`**, `costUsd: 0`, `derived: false`. Settings
1020
- stand on their samples; a scenario with a target declines rather than run on a stale one.
1021
- - With an `apiKey`, `POST https://api.anthropic.com/v1/messages` with the same extraction prompt,
1022
- tolerant JSON extraction (first balanced `{…}`, `undefined` → `null`), and any `null`/missing
1023
- **setting** filled from its `sampleValue` — a `null` target is named in `missing` instead.
1024
-
1025
- Keeping this dependency-free is not purity. `bir-proxy` is spawned inside the host's process tree for
1026
- *every* wrapped server; an npm install that pulls a model SDK into that path is a startup-latency and
1027
- supply-chain cost paid N times per session, for a feature most sessions never arm.
1028
-
1029
- ### 16.4 `Recorder` gains one method
1030
-
1031
- ```ts
1032
- export interface ScenarioReporter {
1033
- reportExecution(r: {
1034
- scenarioId: string; ticket?: string;
1035
- outcome: "steered_full" | "diverged" | "not_steered" | "failed";
1036
- deriveCostUsd: number; sessionCostUsd: number; fallbackCostUsd: number;
1037
- durationMs: number; stepsPlanned: number; stepsPinned: number;
1038
- measured: boolean; pricingVersion: string; prompt?: string;
1039
- }): void;
1040
- }
1041
- export function isScenarioReporter(r: Recorder): r is Recorder & ScenarioReporter;
1042
- ```
1043
-
1044
- Optional capability, discovered the same way `isMatchAware` is — so `NullRecorder` needs no change
1045
- and a session with no BaseIn credentials still replays (it just books nothing).
1046
-
1047
- ### 16.5 CLI surface
1048
-
1049
- `bir` gains one flag and one command group. Everything here is a thin client over endpoints that
1050
- already exist server-side, except `bir replay`, which drives §16.2's executor directly.
1051
-
1052
- ```
1053
- bir install --replay register the `bir` MCP server; raise the prompt-hook timeout to 15 s
1054
- bir uninstall --replay reverse exactly that, leaving the rest of the install alone
1055
-
1056
- bir scenario list GET /recordings/runs runs, iterations, scenario state
1057
- bir scenario show <runId> GET /recordings/runs/:id/scenario intent, params, steps, logic
1058
- bir scenario show <scnId> GET /scenarios/:id the same, by scenario id (segments too)
1059
- [--step <n>: one step]
1060
- bir scenario calc <runId> POST /recordings/runs/:id/calculate [--force to re-derive in place]
1061
- [--force --discard-edits: re-derive a plan with hand edits]
1062
- bir scenario replay <scnId> --prompt "…" [--dry]
1063
-
1064
- bir scenario check <id> --step <n> POST /scenarios/:id/steps/:n/check try a change; writes nothing
1065
- bir scenario edit <id> --step <n> PATCH /scenarios/:id/steps/:n the same check, then save
1066
- bir scenario edits <id> GET /scenarios/:id/edits the history, newest first
1067
- bir scenario undo <id> [--edit <e>] POST /scenarios/:id/edits/:e/revert put one step back
1068
- bir scenario editing on|off|status (no route) ~/.baseinstrunner/installed.json which `bir` MCP tools are offered
1069
-
1070
- bir replay --scenario <scnId> --prompt "…" [--dry] alias; the Tier 2 / debugging path (§12.1)
1071
- ```
1072
-
1073
- The second group, and `show <scnId>`, `--step` and `--discard-edits`, were added on 2026-09-25 by
1074
- the service's `editSteps.md` and are not yet released; their routes are new on the service. They
1075
- are the one place `bir` changes a scenario, so the rules are strict:
1076
-
1077
- - **Owner only.** Anyone else, an admin included, gets `404`, never `403`.
1078
- - **A change is saved only if it reproduces the recording and is not a copy**, or on purpose with
1079
- `--force --note "<why>"`. `check` and `edit` run the same check; `check` never writes.
1080
- - `<id>` is a `run_` id, resolved to its whole-run scenario with `GET /recordings/runs/:id/scenario`,
1081
- or a `scn_` id, used as it is. Logic goes in with `--input-logic` / `--output-logic` from a file,
1082
- or `-` for stdin — never inline, because shell quoting mangles JavaScript. `edit` also takes
1083
- `--freeze`, `--unfreeze`, `--note` and `--revision <n>` (the `expectedRevision` of the body).
1084
- - `undo` without `--edit` undoes the newest edit that can be undone. Every save and every undo bumps
1085
- the scenario's `chainRevision`.
1086
- - `calc --force` answers `409 scenario_has_edits` on a plan with hand edits, unless
1087
- `--discard-edits` is given.
1088
- - Exit codes: `0` checked OK, saved or undone; `1` refused, not found, or a service error; `2`
1089
- usage. `--json` prints the service's body unchanged.
1090
-
1091
- The same commands are tools on the `bir` MCP server. `scenario_show`, `scenario_edits` and
1092
- `investigate` only read, and are always offered. `scenario_check`, `scenario_edit` and
1093
- `scenario_undo` are offered only in a project where `bir scenario editing on` was run, because that
1094
- server runs in every session of every installed project. Each tool runs the installed `bir … --json`;
1095
- a refusal comes back as an MCP error result carrying the report. The operator's walk-through is
1096
- [calculatedReplayGuide.md](calculatedReplayGuide.md) §9.2.
1097
-
1098
- `--dry` maps to BaseIn's `POST /scenarios/:id/replay`, which evaluates the stored logic against the
1099
- source run's *recorded* outputs — no real tools, no side effects, one Haiku call. Without `--dry`,
1100
- `bir replay` runs the same plan through the same executor against the live proxies. `bir doctor`
1101
- gains a `replay` block: on/off, the effective `BIR_REPLAY_ALLOW_SERVERS` allowlist,
1102
- `BIR_MIN_STEER_SIMILARITY`, and who derives — this machine, the service, or nobody, which is the
1103
- state in which a scenario with a target never runs (§13.2, mitigation 3).
1104
-
1105
- ---
1106
-
1107
- ## 17. Implementation phases
1108
-
1109
- Each phase ends in something runnable. Do not start a phase before its predecessor's acceptance test
1110
- passes.
1111
-
1112
- ### Phase R0 — Plan, logic, coverage (no wiring)
1113
- Port `logic.ts` from BaseIn, `bundle.ts` and `ScenarioReplayPlan` from RRepeat, write `coverage.ts`.
1114
- Pure, dependency-free, no I/O.
1115
- **Acceptance:** unit tests replay a fixture `SerializedScenario` end to end against a stub executor
1116
- and reproduce the recorded outputs byte for byte.
1117
-
1118
- ### Phase R1 — Derivation and pricing
1119
- `derive.ts` + `pricing.ts`. Both fully testable against a fake `fetch`.
1120
- **Acceptance:** with no key, derivation returns sample values at zero cost; with a stubbed key, it
1121
- parses fenced JSON, bare JSON and prose-wrapped JSON, and prices a known `usage` to the same cent as
1122
- BaseIn's `cost.ts`.
1123
-
1124
- ### Phase R2 — Arming and the decision log
1125
- `onPrompt` awaits `getMatch()` under budget, runs the gate ladder, computes the mode, arms the plan,
1126
- returns the directive. **No execution yet.**
1127
- **Acceptance:** a fake recorder returns a match; assert exactly one `plan.armed` or one
1128
- `replay.decision` line per prompt, the right mode, and that the prompt hook returns inside the
1129
- budget when `getMatch` never resolves.
1130
-
1131
- ### Phase R3 — Steer mode
1132
- `/tool/pre` pinning (with the §7.1 `updatedInput` merge), `/tool/post` threading (with the §7.2
1133
- source split), retire on done.
1134
- **Acceptance:** a scripted hook sequence drives a 3-step plan to `steered_full`; a wrapped-MCP step
1135
- threads from `/proxy/step` and not from the hook; reordering the model's calls produces exactly one
1136
- divergence and no crash.
1137
-
1138
- ### Phase R4 — The work queue and direct execution
1139
- `ProxyWorkQueue`, `/proxy/poll`, `/proxy/result`, the proxy-side loop.
1140
- **Acceptance:** against `test/fake-upstream.ts`, a 3-step direct plan executes with `respParams`
1141
- threaded between steps; killing the proxy mid-plan degrades to `recordedOutput`/skip and the session
1142
- survives.
1143
-
1144
- ### Phase R5 — `bir-scenario` and direct delivery
1145
- The MCP server, `/scenario/run`, `bir install --replay`.
1146
- **Acceptance:** a real Claude Code session on a fully-wrapped scenario answers from one
1147
- `run_scenario` call, and the transcript shows a genuine `tool_result`.
1148
-
1149
- ### Phase R6 — Divergence
1150
- Direct-execute the remainder; the lazy source-run fetch for `live` steps (§8.1); Bash-clean,
1151
- `bir`-tool and deny delivery; the fair-share bundle cap.
1152
- **Acceptance:** forcing an off-script call produces a bundle under `MAX_REPLAY_REASON` containing
1153
- every remaining step; a `live` step is served its recorded output, marked `(recorded)`, and threads
1154
- `respParams` for the steps after it; a `live` step with no recorded output is skipped with a warning
1155
- and later steps still resolve.
1156
-
1157
- ### Phase R7 — Reporting
1158
- The usage watermark, `reportExecution`, ticket redemption in `sealRun`, `bir doctor` fields.
1159
- **Acceptance:** three prompts in one session produce three *non-cumulative* `sessionCostUsd` values;
1160
- a declined match books a baseline sample; a doubled report books once.
1161
-
1162
- ---
1163
-
1164
- ## 18. Testing
1165
-
1166
- Extends v1 §11; same `node:test` harness, same `test/fake-upstream.ts`.
1167
-
1168
- 1. **Plan semantics** — `respParams` threading, `paramsLogic` applied once, `isDone` after the last
1169
- step, throw-propagation for each of the four logic hooks.
1170
- 2. **Coverage** — every combination of wrapped/unwrapped MCP and built-in maps to the right mode,
1171
- including the empty-steps and all-live cases.
1172
- 3. **The `updatedInput` merge (§7.1)** — a pinned wrapped-MCP call carries both the pinned arguments
1173
- and `__bir_call_id__`; the correlation's `hookArgs` are the *pinned* ones; `--no-correlation`
1174
- still pins.
1175
- 4. **The threading source (§7.2)** — a wrapped step threads the proxy's `result`, not the hook's
1176
- `tool_response`, and threads the hook's view only after the grace window, with the warning.
1177
- 5. **Budgets** — a `getMatch` that never resolves still returns the prompt hook inside its budget;
1178
- a derivation that never resolves still pins, using sample values.
1179
- 6. **Direct execution** — end to end through `fake-upstream`, including a step timeout, a dead proxy
1180
- and a `tools/call` that returns `isError: true` (which is a *result*, not a failure — it must
1181
- thread).
1182
- 7. **Divergence** — each of the three delivery channels; delimiter neutralisation in the Bash path;
1183
- the fair-share cap with one enormous response and four small ones.
1184
- 8. **Reporting** — the watermark across three prompts; each outcome's server-side effect against a
1185
- fake BaseIn; ticket idempotence.
1186
- 9. **Integration** — a real Claude Code session against `fake-upstream` with a hand-written scenario,
1187
- asserting a `steered_full` report. The only test that catches host-behaviour drift.
1188
-
1189
- ---
1190
-
1191
- ## 19. Rejected alternatives
1192
-
1193
- **19.1 Deliver direct-mode results as `additionalContext`.** No extra MCP server, no extra tool in the
1194
- list. Rejected: it must execute inside `UserPromptSubmit`'s timeout (§10), and it asserts to the model
1195
- that tools ran without a transcript record that they did. §6.2.
1196
-
1197
- **19.2 A callback URL per proxy instead of long-polling.** Faster dispatch, no idle connection.
1198
- Rejected: N new loopback listeners, N new tokens, N new things `bir doctor` must verify — for a
1199
- latency saving of milliseconds on a path already dominated by the upstream's own round trip. Dialling
1200
- out is also the posture v1 §9 already chose everywhere else. D12.
1201
-
1202
- **19.3 Give `bir-scenario` its own upstream connections.** Then no work queue at all. Rejected
1203
- outright: it double-spawns every upstream. Two `chrome-devtools-mcp` processes means two browsers,
1204
- two profiles, and a scenario that navigates one and screenshots the other.
1205
-
1206
- **19.4 Port RRepeat's Agent-SDK divergence fallback.** Rejected: it is the dominant cost of RRepeat's
1207
- calculated run (`savingCalcRunner.md` §4), it adds `@anthropic-ai/claude-agent-sdk` to a
1208
- zero-dependency package, it needs anti-recursion guards, and here it buys nothing the proxies do not
1209
- already buy for free. §8.
1210
-
1211
- **19.5 Block the prompt and answer from the scenario alone.** The original RRepeat design, superseded
1212
- there (`replay.md` §1, §10). Rejected for the same reasons plus one of ours: it forfeits the live
1213
- session's built-in tools entirely, which is exactly the half of a mixed scenario the proxy cannot
1214
- reach.
1215
-
1216
- **19.6 Match on the whole conversation rather than the prompt.** Arguably better "similar meaning".
1217
- Rejected as a *server* concern: `findSimilarRun` embeds `run.input`, and changing what is embedded
1218
- changes the corpus, the threshold and every stored vector. §20 question 1.
1219
-
1220
- ---
1221
-
1222
- ## 20. Open questions
1223
-
1224
- 1. **Should the match embed more than the first prompt?** A turn's meaning often lives in the
1225
- preceding turn. Changing it is a BaseIn-side decision with a corpus migration attached, and it
1226
- would let `bir` match mid-conversation rather than only at a turn boundary.
1227
- 2. **Does BaseIn want a `direct` outcome?** Today direct mode reports `steered_full`, which is
1228
- accurate but hides that the run cost no session tokens at all. A fifth enum member would let the
1229
- ledger separate "the model was steered cheaply" from "the model did not run the task."
1230
- 3. **`node:vm` for the logic bodies?** §13.1 states plainly that there is no sandbox. A frozen,
1231
- minimal context would cost little and close the gap between "first-party by provenance" and
1232
- "first-party by construction."
1233
- 4. **Should a declined match still arm a *dry* plan?** Deriving parameters and computing the first
1234
- step's input without executing anything would make `bir doctor` able to say *"this scenario would
1235
- have run these three calls"* — excellent for tuning `BIR_MIN_STEER_SIMILARITY`, at the price of a
1236
- Haiku call on turns that replay nothing.
1237
- 5. **Should `scenario_steps` carry `recorded_output`?** §8.1 works around its absence with one lazy
1238
- `GET /recordings/runs/:runId`. A column would make it a field read, would let a scenario outlive
1239
- the deletion of its source run, and would finally make RRepeat's `recordedOutput` field mean
1240
- something. Cost: the scenario row grows by the size of every step's output.
1241
- 6. **Cwd fit.** A scenario recorded in project A may pin paths that do not exist in project B, and
1242
- nothing currently checks. A cheap sixth gate — compare `run.metadata.cwd` — would catch the common
1243
- case; the general case (relative paths, monorepos) is harder and probably belongs in
1244
- `toolInputLogic` generation server-side.
1
+ # Calculated Replay — running a scenario on a similar-meaning match
2
+
3
+ > Design doc for BaseInstRunnerMCP **v2** ([BaseInstRunner.md](BaseInstRunner.md) §12), the
4
+ > milestone the v1 design deferred: *"Serve recorded results; port RRepeat's divergence handling."*
5
+ >
6
+ > **What it adds.** When the user's prompt means the same thing as a prompt they have run before,
7
+ > the BaseIn service says so and hands back the *calculated scenario* derived from that earlier run.
8
+ > This document specifies what BaseInstRunnerMCP does with it: derive the new prompt's parameters,
9
+ > run the scenario's tool sequence against live data, and let the model answer from genuine results —
10
+ > instead of rediscovering the same plan token by token.
11
+ >
12
+ > **Status:** implemented. Phases R0–R7 (§17) are in `src/replay/`, wired through the control server
13
+ > and the proxies, covered by `test/replay.test.ts` and `test/replay-server.test.ts`, and proved end
14
+ > to end with real processes by `test/smoke-replay.mjs`. Notes marked **[built]** are where the
15
+ > implementation settled something this design left open, and notes marked **[changed]** are where
16
+ > the production-readiness work later changed what a phase built. [production.md](../production.md)
17
+ > holds the production contract: budgets, environment variables, and what fails how.
18
+ > **Reference implementation:** RRepeat's [`docs/replay.md`](../../../BasInstRunner/RRepeat/docs/replay.md)
19
+ > §3, §12, §13 and [`docs/savingCalcRunner.md`](../../../BasInstRunner/RRepeat/docs/savingCalcRunner.md).
20
+ > **Server contract:** BaseIn `src/recordings/routes.ts`, `src/scenarios/{routes,repo,replay,logic}.ts`.
21
+
22
+ ---
23
+
24
+ ## 0. The one fact that makes this different from RRepeat
25
+
26
+ RRepeat and BaseInstRunnerMCP are solving the same problem from opposite sides of the tool call.
27
+
28
+ RRepeat **observes** a Claude Code session through hooks. It has no execution path of its own, so
29
+ when a replay diverges it must spawn a *second* Claude Agent SDK session to run the remaining tools
30
+ for real (`sdk-tool-session.ts`, `replay.md` §13). Its own savings doc names the consequence:
31
+
32
+ > *"the **dominant** cost of today's calculated run is a full second agent session, and it is the one
33
+ > cost nobody counts."* — `savingCalcRunner.md` §4
34
+
35
+ BaseInstRunnerMCP **is** the execution path for MCP. Every wrapped server is reached through a
36
+ `bir-proxy` process that already holds an initialized, live `UpstreamClient` to it
37
+ ([`src/upstream/client.ts`](../src/upstream/client.ts)). Issuing a `tools/call` on that connection
38
+ costs **zero tokens** and requires no second agent, no subprocess, and no model in the loop.
39
+
40
+ Everything below follows from that. Where RRepeat has one replay mode and an expensive fallback,
41
+ this design has a ladder whose cheapest rung is free — and the more of a stack sits behind
42
+ `bir-proxy`, the further down the ladder a scenario lands.
43
+
44
+ | | RRepeat | BaseInstRunnerMCP |
45
+ |---|---|---|
46
+ | Runs MCP tools itself | No (`replay.md` §12: *"rrepeat never executes tools — not even MCP"*) | **Yes**, on the connection it already owns |
47
+ | Divergence fallback | A second Agent SDK session — the dominant cost | Direct `tools/call` through the proxies — **$0** |
48
+ | Reaches built-ins (`Bash`, `Read`) | Only by steering the live session | Same — only by steering the live session |
49
+ | Needs `@anthropic-ai/*` at runtime | Yes | No (§16.3) |
50
+
51
+ ---
52
+
53
+ ## 1. Decisions locked
54
+
55
+ Continuing v1's D1–D8.
56
+
57
+ | # | Decision | Choice | Consequence |
58
+ |---|---|---|---|
59
+ | **D9** | What a match buys | **Execute the scenario**, do not merely stop recording | v1's `watchForMatch` already detects the hit and stops; this replaces the no-op with work |
60
+ | **D10** | Who decides the mode | **Tool coverage**, computed per scenario at arm time (§5) | A fully-wrapped scenario replays for free; a mixed one steers the live session |
61
+ | **D11** | Execution ownership for MCP steps | **The proxy that owns the upstream** | No second connection, no double-spawned browser, no Agent SDK dependency |
62
+ | **D12** | Control → proxy channel | **Long-poll**, proxy dials out | No new listeners, no new ports, no new tokens (§13.3, §19.2) |
63
+ | **D13** | Result delivery in direct mode | A **first-party MCP server** (`bir`), one tool | The model reads a genuine `tool_result`, never a `deny` reason it distrusts (§19.1) |
64
+ | **D14** | Default state | **On** once `bir install --replay` has run; `BIR_REPLAY=0` turns it off | Replay bypasses permission prompts (§13.2); the install flag is the opt-in, the env var is the kill switch |
65
+ | **D15** | Parameter derivation | One Haiku call over raw `fetch`; **no API key ⇒ recorded sample values** | Preserves v1's zero-runtime-dependency property, and degrades to a free replay |
66
+ | **D16** | Savings reporting | Every armed match reports to `POST /scenarios/:id/executions`, including declines | A decline is a *baseline sample*, not silence — it is what keeps the ledger honest |
67
+
68
+ ### 1.1 Non-goals
69
+
70
+ - **Replay in Tier 2.** A match is a match *on the prompt*, and a standalone proxy never sees one
71
+ (§0.1 of the v1 design). Tier 2 gets the manual `bir replay` command (§12.1) and nothing automatic.
72
+ - **Chaining scenarios.** RRepeat's `next`/`getScenarioChain` has no counterpart in BaseIn's
73
+ `serializeScenario` payload, which serializes exactly one scenario. Out of scope.
74
+ - **Editing tool descriptions to steer selection.** Still a non-goal, unchanged. §6's `bir` server
75
+ adds one *first-party* tool and touches no upstream schema — and it costs context in every session,
76
+ which is why D14 makes it opt-in.
77
+ - **Recording the replayed turn.** The service created no run for it, by design. §12.
78
+
79
+ ---
80
+
81
+ ## 2. The match, and where it already works
82
+
83
+ Nothing in this section is new. It is already implemented and already runs on every prompt.
84
+
85
+ ```
86
+ UserPromptSubmit
87
+ └─ ControlServer.onPrompt src/control/server.ts
88
+ └─ ensureRun → recorder.startRun(input, metadata)
89
+ └─ RemoteRecorder.startRun POST /recordings/runs { id, framework, input, metadata }
90
+ └─ BaseIn recordings/routes.ts:
91
+ findSimilarRun(userId, input) ← embed + cosine over the user's runs
92
+ ├─ miss → 201 { id, matched: null }
93
+ └─ hit → 200 { id: null, matched: {...} } ← no run is created
94
+ ```
95
+
96
+ On a hit the server answers with the whole thing — there is no follow-up fetch:
97
+
98
+ ```jsonc
99
+ {
100
+ "id": null,
101
+ "matched": {
102
+ "runId": "run_…", // the canonical earlier run
103
+ "scenarioId": "scn_…", // its calculated scenario, when one is ready
104
+ "similarity": 0.94, // cosine, ≥ SIMILARITY_THRESHOLD (server default 0.9)
105
+ "executionTicket": "tkt_…", // redeem exactly once at POST /scenarios/:id/executions
106
+ "scenario": { /* serializeScenario(): intent, paramsObject, paramsLogic,
107
+ responseParamsLogic, steps[{ stepIndex, toolName,
108
+ toolInputLogic, toolOutputLogic, reasoning }] */ }
109
+ }
110
+ }
111
+ ```
112
+
113
+ [`RemoteRecorder.getMatch(runId)`](../src/record/remote-recorder.ts) already exposes it, and
114
+ [`ControlServer.watchForMatch`](../src/control/server.ts) already consumes it — to do exactly one
115
+ thing:
116
+
117
+ ```ts
118
+ run.recording = false;
119
+ logLine("run.matched", { …, why: "similar prompt — the service kept its own run; not recording this one" });
120
+ ```
121
+
122
+ **That is the whole of v1's replay support: a correct stop.** This design replaces the stop with a
123
+ plan, and keeps the stop.
124
+
125
+ ### 2.1 What "similar meaning" means, precisely
126
+
127
+ Server-side, and worth stating because the threshold in §4 is a judgement about it:
128
+
129
+ - The embedding is of the run's **`input`** — the user's prompt — computed by
130
+ `Xenova/all-MiniLM-L6-v2` (`EMBEDDING_MODEL`), mean-pooled and normalized
131
+ (`recordings/embeddings.ts`).
132
+ - Candidates are **the caller's own canonical runs only** (`listUserRunEmbeddings(userId)`);
133
+ duplicates and sub-threshold runs are excluded from the pool at record time.
134
+ - The match is the single best cosine ≥ `SIMILARITY_THRESHOLD` (default **0.9**).
135
+ - A matched run only carries a `scenarioId` when its scenario exists **and** is `state: 'ready'`.
136
+
137
+ So "similar meaning" is: *the same user asked for something that embeds within 0.1 cosine of this,
138
+ and we finished calculating a reusable plan for it.* It is not "the same words", and it is not
139
+ cross-tenant.
140
+
141
+ ---
142
+
143
+ ## 3. Architecture
144
+
145
+ ```mermaid
146
+ graph TB
147
+ subgraph host["Claude Code session"]
148
+ M[model loop]
149
+ H[hooks]
150
+ end
151
+
152
+ subgraph bir["BaseInstRunnerMCP"]
153
+ C["bir-hooks — control server<br/>ScenarioReplayPlan lives here"]
154
+ S["bir-scenario<br/>MCP server: run_scenario"]
155
+ P1["bir-proxy — chrome-devtools"]
156
+ P2["bir-proxy — github"]
157
+ end
158
+
159
+ U1[chrome-devtools-mcp]
160
+ U2[github mcp]
161
+ R[("BaseIn /recordings/* /scenarios/*")]
162
+
163
+ H -->|"UserPromptSubmit → directive"| C
164
+ H -->|"PreToolUse → pinned updatedInput"| C
165
+ H -->|"PostToolUse → real output"| C
166
+ M -->|"mcp__bir__run_scenario"| S
167
+ S -->|"POST /scenario/run"| C
168
+ C -.->|"work: tools/call"| P1
169
+ C -.->|"work: tools/call"| P2
170
+ P1 -->|"bir: id request"| U1
171
+ P2 -->|"bir: id request"| U2
172
+ C -->|"match + execution report"| R
173
+ ```
174
+
175
+ Three additions to the v1 topology, and nothing else moves:
176
+
177
+ | Component | New? | Role in replay |
178
+ |---|---|---|
179
+ | `bir-hooks` control server | extended | Holds the `ScenarioReplayPlan` on `RunState`. Arms it, pins inputs, threads outputs, dispatches direct steps, redeems the ticket |
180
+ | `bir-proxy` | extended | Long-polls for work; executes a `tools/call` on the upstream it already owns |
181
+ | `bir-scenario` | **new**, ~120 lines | A stdio MCP server exposing one tool, `run_scenario`. The delivery channel for direct mode (§6). It later gained the scenario tools of §16.5 |
182
+
183
+ ---
184
+
185
+ ## 4. The gate ladder
186
+
187
+ Ported from RRepeat's `applyMatch`, with one gate this repo needs and RRepeat cannot have.
188
+ Every gate that declines logs **one** `replay.decision` line carrying the reason — because
189
+ *"a scenario existed and was not used"* is a fact an auditor needs as much as *"a scenario ran"*.
190
+
191
+ | # | Gate | Declines when | Outcome floor |
192
+ |---|---|---|---|
193
+ | 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched) |
194
+ | 2 | **Replay enabled** | `BIR_REPLAY === "0"`, or, with it unset, the project's stored switch is off (`bir replay off`, or a `bir install` without `--replay`) | `not_steered` |
195
+ | 3 | **Scenario ready** | `scenario == null`, `state !== "ready"`, or `steps.length === 0` | `not_steered` |
196
+ | 4 | **Similarity** | `similarity < BIR_MIN_STEER_SIMILARITY` (default **0.92**) | `not_steered` |
197
+ | 5 | **Tool coverage** | No step is executable anywhere (§5 → `mode: "none"`) | `not_steered` |
198
+
199
+ Past the table, a plan can still be refused before the model hears of it, each refusal with its own
200
+ `replay.decision` code. The match is not waited on at all while the service is cooling down after a
201
+ failure (`service_unhealthy`), or while prompts are holding off a service that answered matches
202
+ slowly (`match_slow`, the match gate, §10). A match or a plan that is not ready
203
+ inside the prompt's budget (§10) is dropped (`match_timeout`, `arm_timeout`), and one that lands
204
+ after the prompt was answered is retired unannounced (`match_late`, `arm_unannounced`). A `direct`
205
+ plan is declined when no running `bir-scenario` has said it serves `run_scenario`
206
+ (`no_delivery_tool`, §6.3). In plan mode (`permission_mode: "plan"`) nothing is announced at all.
207
+
208
+ Gate 4 deserves its own note. The server's own detection threshold is 0.9, and it uses it to decide
209
+ *"do not record this again"* — a cheap claim: the worst case is a lost recording. Steering makes a
210
+ much stronger claim: *"do not think about this again."* A prompt that is 0.90-similar usually covers
211
+ only part of the live task, and steering it produces a confident, thin, wrong answer. So the steer
212
+ threshold sits **above** the detection threshold by default, and `BIR_MIN_STEER_SIMILARITY` is the
213
+ single knob an operator turns after watching `replay.decision` lines for a week.
214
+
215
+ `not_steered` as the floor is not a failure code. It is the **control group**: the prompt matched,
216
+ the agent ran the task the ordinary way, and the cost of doing so is a fresh measurement of what this
217
+ task costs — which is exactly what a baseline is made of. BaseIn's
218
+ `POST /scenarios/:id/executions` routes `not_steered` to `addBaselineSample` rather than the
219
+ savings ledger, on purpose (`scenarios/routes.ts`). §11.
220
+
221
+ ---
222
+
223
+ ## 5. Tool coverage decides the mode
224
+
225
+ The classification is a pure function of the scenario's steps and the set of server keys registered
226
+ with this control server (`ControlServer.wrapped`, grown by `/proxy/register`).
227
+
228
+ ```ts
229
+ // src/replay/coverage.ts
230
+ export type StepReach = "direct" | "live";
231
+ export type ReplayMode = "direct" | "steer" | "none";
232
+
233
+ export function reachOf(toolName: string, wrapped: ReadonlySet<string>): StepReach {
234
+ const mcp = parseQualifiedName(toolName); // src/control/correlation.ts
235
+ return mcp && wrapped.has(mcp.serverName) ? "direct" : "live";
236
+ }
237
+
238
+ export function modeFor(steps: SerializedScenarioStep[], wrapped: ReadonlySet<string>): ReplayMode {
239
+ if (steps.length === 0) return "none";
240
+ return steps.every((s) => reachOf(s.toolName, wrapped) === "direct") ? "direct" : "steer";
241
+ }
242
+ ```
243
+
244
+ | Every step | Mode | Where the tools run | Model spend |
245
+ |---|---|---|---|
246
+ | a wrapped MCP tool | **`direct`** (§6) | The proxies, on connections already open | one Haiku derivation + one turn that reads the results |
247
+ | anything else, or mixed | **`steer`** (§7) | The live session, inputs pinned | one Haiku derivation + the live turn |
248
+ | — | `none` | nowhere | declines at gate 5 |
249
+
250
+ `live` covers three genuinely different cases and treats them identically, correctly: a built-in
251
+ (`Bash`, `Read`, `Edit`, `Grep`, `Task`, `WebFetch`), an MCP server the user chose not to wrap, and
252
+ `claude-in-chrome` — which is `scope: "dynamic"`, appears in no config file, and therefore can never
253
+ be wrapped (v1 design §1). None of them can be reached except through the session.
254
+
255
+ A **mixed** scenario in `steer` mode still gets direct execution for its wrapped steps once it
256
+ diverges (§8), so coverage is not all-or-nothing at the divergence boundary — only at the arm-time
257
+ mode choice.
258
+
259
+ ---
260
+
261
+ ## 6. Mode A — direct execution
262
+
263
+ The cheap rung. Every step is a wrapped MCP tool, so the model never has to emit a single tool call.
264
+
265
+ ### 6.1 Sequence
266
+
267
+ ```mermaid
268
+ sequenceDiagram
269
+ participant CC as Claude Code (model)
270
+ participant CS as bir-hooks (control server)
271
+ participant SC as bir-scenario (MCP)
272
+ participant PX as bir-proxy
273
+ participant UP as upstream MCP server
274
+
275
+ CC->>CS: UserPromptSubmit
276
+ CS->>CS: gates pass, mode=direct → arm plan; start derivation (not awaited)
277
+ CS-->>CC: additionalContext: "call mcp__bir__run_scenario once"
278
+ CC->>CS: PreToolUse run_scenario → allow (binds the call to this session)
279
+ CC->>SC: tools/call run_scenario {}
280
+ SC->>CS: POST /scenario/run { toolUseId }
281
+ CS->>CS: await plan.ready() (derivation lands)
282
+ loop each step
283
+ CS->>CS: toolInputForCurrentStep()
284
+ CS-->>PX: work { workId, toolName, arguments } (answers an open long-poll)
285
+ PX->>UP: tools/call (bir: id, on the open connection)
286
+ UP-->>PX: CallToolResult
287
+ PX->>CS: POST /proxy/result { workId, result }
288
+ CS->>CS: applyOutput(JSON.stringify(result)) → respParams
289
+ end
290
+ CS->>CS: responseParamsLogic → response model; assembleBundle()
291
+ CS-->>SC: { text, responseModel, steps }
292
+ SC-->>CC: tool_result (genuine)
293
+ CC->>CC: answers from the results
294
+ CC->>CS: Stop → seal, report execution (ticket)
295
+ ```
296
+
297
+ ### 6.2 Why a first-party MCP server, and not `additionalContext`
298
+
299
+ Executing inside `UserPromptSubmit` and returning the results as `additionalContext` is the obvious
300
+ shortcut, and it is wrong for three separate reasons:
301
+
302
+ 1. **Budget.** `UserPromptSubmit` has a hook timeout (§10). N live MCP round trips do not fit inside
303
+ one, and if it expires Claude Code proceeds *without* the context — so the work is burned and the
304
+ user waits for nothing.
305
+ 2. **Trust.** RRepeat learned this expensively. Its `deny`-with-a-reason delivery is *"inherently
306
+ read as a denial by the model"* (`hook-server.ts`), which is why it grew a Bash-rewrite path and
307
+ then a whole MCP bridge (`mcp-bridge.ts`) to hand the same bytes back as a genuine `tool_result`.
308
+ A first-party MCP tool is that bridge, without the shadowing.
309
+ 3. **Truth.** A `tool_result` in the transcript is a real record of a real call. `additionalContext`
310
+ asserting that tools ran is a claim the transcript cannot corroborate.
311
+
312
+ The cost is honest and worth naming: one extra tool in the model's tool list, in **every** session
313
+ where `bir install --replay` ran. That is why it is a separate install flag and why `bir status`
314
+ prints it.
315
+
316
+ ### 6.3 `run_scenario`
317
+
318
+ ```jsonc
319
+ // tools/list, from bir-scenario
320
+ {
321
+ "name": "run_scenario",
322
+ "description":
323
+ "Returns the results BaseInstRunner prepared for the current request. Call it only when a BaseInstRunner note in this turn (text beginning \"[BaseInstRunner calculated replay]\", in a system note or a tool result) tells you to; never otherwise.",
324
+ "inputSchema": { "type": "object", "properties": {}, "additionalProperties": false }
325
+ }
326
+ ```
327
+
328
+ No arguments **by design**: the armed plan is server-side state keyed by the control server's open
329
+ run, so there is nothing for the model to get wrong and nothing it can point at a different scenario.
330
+ Which session's plan runs is the control server's call. The `PreToolUse` for `run_scenario` records
331
+ the session and its `tool_use_id` (and answers `allow` for an announced direct plan, §13.2), and
332
+ `bir-scenario` forwards the host's id for the call, Claude Code's
333
+ `params._meta["claudecode/toolUseId"]`, as `toolUseId`. With no `PreToolUse` waiting, nothing runs.
334
+
335
+ `run_scenario` never answers with an error (`isError: false`), because a model sometimes calls it
336
+ unasked, and a red error under a tool the user never asked about helps nobody:
337
+
338
+ | Situation | What the model reads |
339
+ |---|---|
340
+ | No plan for this call, or no control server running for this directory | `Nothing to run for this request; continue normally.` |
341
+ | The plan stopped part-way | the controller's hand-over note (§8), which begins `[BaseInstRunner calculated replay]` and says which steps ran, then the results of those steps |
342
+ | The plan ran nothing live | the hand-over note alone |
343
+ | A second call while the plan is still running | `The prepared results for this request are already being produced for an earlier run_scenario call, and they come back there. Do not call run_scenario again, and do not repeat its steps.` |
344
+ | Anything else: a control server that does not answer, a timeout, an HTTP error | `The prepared results are not available for this request. Do the task normally with your tools.` |
345
+
346
+ A bundle the plan's time budget cut short ends with `(partial: the scenario stopped before all of its
347
+ steps ran)`. The real reason is in the `scenario.answered` line on the server's stderr;
348
+ `BIR_SCENARIO_DEBUG=1` in its environment brings back the raw reasons as `isError` results. The
349
+ request waits at most 290 s (`BIR_SCENARIO_RUN_TIMEOUT_MS`), below Claude Code's 5-minute idle
350
+ timeout for an MCP call. Esc (`notifications/cancelled`) aborts it, and the control server starts
351
+ no further step.
352
+
353
+ By default the server lists only `run_scenario`; §16.5 has the others. While the runner is off
354
+ (`bir off`, `BIR_OFF=1`) it lists no tools, and it tells Claude Code when that changes
355
+ (`notifications/tools/list_changed`).
356
+
357
+ `bir-scenario` is a client of the control server, discovered the same way a proxy discovers it —
358
+ `~/.baseinstrunner/control/<key>.json`, mode `0600`, bearer token
359
+ ([`src/control/discovery.ts`](../src/control/discovery.ts)). It holds no state, no credentials and
360
+ no upstream connections.
361
+
362
+ **It says it is there.** A configured `bir` server is not a connected one: a stale node path, a
363
+ crash, the server disabled in `/mcp` or a session started with `BIR_OFF=1` all leave the model
364
+ without the tool. So `bir-scenario` sends `POST /scenario/hello` (pid, version, how many tools, off
365
+ or not) on `initialize`, on `tools/list`, when the off switch flips, whenever its recorder changes
366
+ (it looks every 2 s) and every 5 minutes, bounded at 1 s and never awaited. The control server
367
+ announces a direct plan only while a live `bir-scenario` pid has said hello within the last 30
368
+ minutes, with at least one tool and not off; otherwise the plan is declined (`no_delivery_tool`)
369
+ and the agent does the task. When the recorder's pid is dead and its port refuses connections,
370
+ `bir-scenario` also starts `bir-hooks revive` for an installed project (detached and rate-limited,
371
+ never for a missing discovery file, so never after `bir down`), and logs `scenario.recorder_revive`.
372
+
373
+ ### 6.4 The steering directive (direct)
374
+
375
+ ```
376
+ [BaseInstRunner calculated replay] A known-good tool sequence was recorded
377
+ for this request (intent: <scenario.intent>).
378
+
379
+ It runs these tools, in order, and returns their results:
380
+
381
+ 1. mcp__chrome-devtools__navigate_page — open the fleet dashboard
382
+ 2. mcp__chrome-devtools__take_snapshot — read the device table
383
+
384
+ Call mcp__bir__run_scenario once, with no arguments, before any other tool.
385
+ Then answer the user's request from the results it returns.
386
+ ```
387
+
388
+ **A direct plan never diverges.** If the model's first call (host housekeeping such as `ToolSearch`
389
+ aside) is anything but `run_scenario`, the likeliest reason is that it has no such tool, and
390
+ executing every step unattended inside that call's hook is the wrong answer to a broken delivery
391
+ tool. The plan is retired without running anything (`replay.vetoed`, booked `not_steered`), the
392
+ call runs as the model wrote it, and its `PostToolUse` tells the model that the scenario is not
393
+ being run. A plan whose `run_scenario` was called but whose `/scenario/run` never arrived — the
394
+ user refused the call, a rule denied it, or it failed on the way — is retired the same way, at that
395
+ call's `PostToolUse` or the next call, so a refusal is never overridden by running the same steps
396
+ another way. Divergence (§8) is for steer plans only.
397
+
398
+ ---
399
+
400
+ ## 7. Mode B — model-steered live replay
401
+
402
+ The mixed rung. Ported from RRepeat's `replay.md` §3/§12, which is the right design and is already
403
+ proven: the model chooses *which* tool and *when*; the control server overrides the *arguments* and
404
+ threads state between steps. The session never leaves the model's hands, so every
405
+ `tool_use`/`tool_result` in the transcript is genuine and continuation is automatic.
406
+
407
+ **Arm** at `UserPromptSubmit` — plan on `run.plan`, `run.planToolUse = new Set()`, directive via
408
+ `additionalContext`, **no `decision`** so the prompt still reaches the model:
409
+
410
+ ```
411
+ [BaseInstRunner calculated replay] A known-good tool sequence was previously
412
+ recorded for this request. Call exactly these tools, in order, one at a time:
413
+
414
+ 1. mcp__chrome-devtools__navigate_page — open the fleet dashboard
415
+ arguments: {"url":"https://fleet.internal/eu-west-1"}
416
+ 2. mcp__chrome-devtools__take_snapshot — read the device table
417
+ arguments: {}
418
+ 3. Read — load the alert thresholds
419
+
420
+ Context: <scenario.intent>
421
+
422
+ The system supplies the exact arguments for each call. Where a step shows
423
+ its arguments above, they are the ones it runs with: pass them as shown.
424
+ Do not call any other tools until this sequence is complete,
425
+ then answer the user's request from the tool results.
426
+ ```
427
+
428
+ Where a step's arguments are known when the plan arms, they are shown under it: `command: …` for
429
+ `Bash` and `PowerShell`, `arguments: {…}` for anything else, redacted and at most 240 characters.
430
+ A step whose input reads an earlier step's output (step 3 above) or runs a multi-line command shows
431
+ none, and a plan holding a value the safety check (§13.2) would refuse shows no arguments at all.
432
+ With no arguments shown at all, the closing lines
433
+ are the older *"The system supplies the exact arguments for each call — you do not need to compute
434
+ them."* The arguments matter because of the pin guard below: told only that the system supplies
435
+ them, a model writes a placeholder call that the guard rightly declines. A steer plan with a shell
436
+ or file-write step gets up to 1.2 s of the prompt's budget for its parameters to settle, so that its
437
+ directive can show them.
438
+
439
+ **Pin** at `PreToolUse`, while a plan is active and the call is not `mcp__bir__run_scenario`:
440
+
441
+ ```ts
442
+ if (toolName === plan.expectedTool()) {
443
+ const input = plan.toolInputForCurrentStep(); // may throw → hand over (§8)
444
+ if (!pinCompatible(toolName, modelArgs, input).ok) // control/pin-guard.ts
445
+ return asWritten(); // no pin; the plan stays armed
446
+ run.planToolUse.add(toolUseId);
447
+ return { hookSpecificOutput: { hookEventName: "PreToolUse",
448
+ permissionDecision: "allow",
449
+ updatedInput: withCallId(input) } }; // §7.1
450
+ }
451
+ return divergence(…); // §8
452
+ ```
453
+
454
+ **The same tool is not the same call.** A pin swaps the model's arguments for the step's and
455
+ answers `allow`, so it is kept only for a call that plausibly *is* the step
456
+ ([`pin-guard.ts`](../src/control/pin-guard.ts)):
457
+
458
+ - `Bash` / `PowerShell`: the same commands, chained the same way (`&&`, `||`, `;`, `|`, `&`, a new
459
+ line), each with the same head — the program plus its subcommand (`git push`, `npm test`), or
460
+ plus what it runs (`npm run lint`, `npx eslint`, `python deploy.py`, `python -m pytest`,
461
+ `uv run pytest`; `bash -c 'git push'` counts as `git push`). A step with a command substitution
462
+ (`$(…)`, backticks) pins only to an exact copy.
463
+ - Tools that write a file: the same file. Read-only file tools (`Read`, `Grep`, `Glob`, …) may be
464
+ steered to the step's path.
465
+ - MCP tools whose input is run — a query, a statement, a shell command: every statement the step
466
+ runs, supplied with the same verb (`SELECT` for `SELECT`, never for `DELETE`), or no arguments at
467
+ all. An empty statement argument of the step need not be supplied, but a call that runs something
468
+ there is not the step.
469
+ - Everything else: at least one argument in common, and no argument naming *where* the call acts
470
+ (a path, a URL) pointing somewhere else. A call with no arguments at all is pinned.
471
+
472
+ A call the guard declines runs exactly as the model wrote it, and the plan stays armed for the call
473
+ that is the step (`replay.pin_declined`). If the model then calls a *different* tool, it did the
474
+ step its own way and moved on: the plan is retired without running anything (`replay.pin_abandoned`),
475
+ that call runs as written, and its `PostToolUse` tells the model the arguments are its own from now
476
+ on.
477
+
478
+ **One call at a time.** A pin moves the plan's cursor only when its output threads. A call that
479
+ arrives while the previous pin has not threaded first waits for it inside `PreToolUse`: up to 1.75 s
480
+ after the host reported the pinned call finished, 200 ms while it has not, and not at all when the
481
+ plan's last step is the one in flight. A call still running beside a pin after that hands the plan
482
+ over (`replay.concurrent_call`), booked `fell_back` or `failed`, and carries the note in its place.
483
+ A pinned step the host reports as failed (`PostToolUseFailure`), or whose proxy report is an error,
484
+ hands over as well; nothing after it is pinned or approved.
485
+
486
+ **Thread** at `PostToolUse`, for a `tool_use_id ∈ run.planToolUse` — but *from the right source*
487
+ (§7.2). When `plan.isDone()`, retire the plan; the model then writes its answer from the real tool
488
+ history.
489
+
490
+ ### 7.1 Steering and correlation both write `updatedInput` — they must merge
491
+
492
+ This collision does not exist in RRepeat and is easy to miss here.
493
+
494
+ `onToolPre` can return an `updatedInput` for a **wrapped MCP** call of its own: it is how the
495
+ correlation id rides along to the proxy ([`src/control/correlation.ts`](../src/control/correlation.ts)).
496
+ By default it does so only where answering `allow` changes nothing the user is asked — a
497
+ replay-pinned step, or a session in `bypassPermissions` mode — or where the operator opted in with
498
+ `BIR_CORRELATION_DECISION=allow|ask`; never in plan mode, and never into a server with no live
499
+ proxy to strip the id. Every other wrapped call gets no decision at all and joins its proxy report
500
+ by fingerprint. A pinned step whose tool is a wrapped MCP tool therefore needs *both* payloads in
501
+ one object, and the order is not arbitrary:
502
+
503
+ ```ts
504
+ /** Pinned arguments plus, for a wrapped MCP tool, the correlation id. */
505
+ function withCallId(pinned: Record<string, unknown>): Record<string, unknown> {
506
+ const mcp = parseQualifiedName(toolName);
507
+ if (!mcp || !this.wrapped.has(mcp.serverName) || this.opts.noCorrelation) return pinned;
508
+ if (!liveProxy(mcp.serverName)) return pinned; // allow + pinned input, no id
509
+ const correlation = mintCorrelation(run, mcp, pinned, context); // hookArgs = the PINNED args
510
+ return { ...pinned, [BIR_CALL_ID]: correlation.callId };
511
+ }
512
+ ```
513
+
514
+ Three consequences, all load-bearing:
515
+
516
+ - **The correlation's `hookArgs` must be the pinned arguments, not the model's.** `settle()` records
517
+ `correlation.hookArgs ?? report.args`; recording the arguments the model *proposed* rather than the
518
+ ones that *ran* would put a lie in the corpus. (In a replayed turn nothing is recorded — §12 — but
519
+ the same code path serves recorded turns, and the invariant must hold there.)
520
+ - **The key must already be declared**, or the host may reject it. It is: while a control server
521
+ may inject call ids, the proxy's `tools/list` declares `__bir_call_id__` as a string property
522
+ (*"Reserved. Leave unset."*) and leaves `additionalProperties` as the upstream had it
523
+ (`ProxySession.declaresCallId`). Under `--no-correlation`, `withCallId` is a no-op and pinning
524
+ still works — only the *result* join degrades to fingerprints.
525
+ - **`permissionDecision: "allow"` auto-approves.** A pinned step is answered `allow` — built-ins
526
+ and wrapped MCP tools alike — because replay is its own consent and the host may drop an
527
+ `updatedInput` it did not approve. An ordinary wrapped call is not auto-approved unless the
528
+ operator set `BIR_CORRELATION_DECISION=allow`. §13.2.
529
+
530
+ ### 7.2 Thread from the same source that recorded
531
+
532
+ `toolOutputLogic` was authored by BaseIn's `analyzeRun` against the exact bytes stored in that step's
533
+ `tool_output`. Feeding it a differently-shaped rendering of the same result does not fail loudly — it
534
+ silently derives nothing, and every later step reading `respParams` produces a wrong input.
535
+
536
+ In this repo those bytes have **two different origins**:
537
+
538
+ | Step's tool | What was recorded | Therefore thread from |
539
+ |---|---|---|
540
+ | Wrapped MCP | `serializeCapped(redact(report.result))` — the proxy's whole `CallToolResult`, via `settle()` → `record()` | the **proxy's `/proxy/step` report** |
541
+ | Built-in / unwrapped MCP | `serializeCapped(redact(payload.tool_response))` — Claude Code's rendering, via `onToolPost` | the **hook's `tool_response`** |
542
+
543
+ So `/tool/post` must not thread unconditionally. It threads for a `live` step, and for a `direct`
544
+ step it does what it already does for a correlated call: **arm the existing
545
+ `PROXY_REPORT_GRACE_MS` fallback**, and let `onProxyStep` thread when the report lands.
546
+
547
+ ```ts
548
+ // onToolPost, for a pinned step
549
+ if (reachOf(step.toolName, this.wrapped) === "live") {
550
+ plan.applyOutput(serializeCapped(payload.tool_response ?? {}));
551
+ } else {
552
+ correlation.threadOnReport = true; // onProxyStep calls plan.applyOutput(report.result)
553
+ correlation.fallback = setTimeout(() => { // …unless the proxy never reports
554
+ logLine("replay.thread_fallback", { why: "no proxy report — threading the hook's view; shapes may differ" });
555
+ plan.applyOutput(serializeCapped(payload.tool_response ?? {}));
556
+ }, PROXY_REPORT_GRACE_MS);
557
+ }
558
+ ```
559
+
560
+ This reuses machinery that exists and is already correct. It also means the §7.3 ordering constraint
561
+ binds a little harder: the proxy report must land before the next `PreToolUse`, and
562
+ `PROXY_REPORT_GRACE_MS` (1500 ms) is the bound on how long that can take before we degrade.
563
+
564
+ ### 7.3 Synchronous `PostToolUse` is a requirement, not a preference
565
+
566
+ Step *N*'s `/tool/post` must finish updating `respParams` before step *N+1*'s `/tool/pre` reads it.
567
+ In [`HOOK_ROUTES`](../src/config/adapters/claude-code.ts) neither `PreToolUse` nor `PostToolUse`
568
+ carries `async: true`, so this already holds — **and must never be relaxed**. Add it to the comment
569
+ block there, next to the `SessionEnd` note that records the same class of lesson.
570
+
571
+ ---
572
+
573
+ ## 8. Divergence
574
+
575
+ In a steer plan, the model called a tool the plan did not expect, or the call had no `tool_use_id`.
576
+ (A direct plan never diverges, §6.4. A `toolInputLogic` that throws is not a divergence either:
577
+ re-executing would start at that same step and throw again, so the step is recorded as failed and
578
+ the task is handed to the model, with a note in place of the call.) Three tiers, tried in order.
579
+ Here is where the §0 asymmetry pays.
580
+
581
+ **Tier 1 — direct-execute the remainder (new; the common case).**
582
+ Take every not-yet-run step, execute it, and hand the composed result back. This happens inside the
583
+ diverging call's `PreToolUse`, so it is bounded by `BIR_PRETOOL_REPLAY_BUDGET_MS` (§10): no step
584
+ starts after that budget, or after the hook has had to answer without it. Steps with
585
+ `reach === "direct"` run through their proxy, and only on servers `BIR_REPLAY_ALLOW_SERVERS`
586
+ allows. Steps with `reach === "live"` cannot run here at all — a hook process is not the session.
587
+
588
+ **A step that produced no result ends the bundle there**: a `live` step, a server off the
589
+ allowlist, no proxy to take the call, a timeout, an upstream refusal, a tool error or a logic
590
+ failure. Nothing after it runs. This is RRepeat's rule about skipped steps taken to its end: every
591
+ later step reading `respParams` would compute on a result that does not exist. The bundle then opens
592
+ with a hand-over note that names the step and says what may be assumed about it — a call that was
593
+ sent and never answered *"may or may not have completed"*, a step whose tool ran but whose output
594
+ logic failed is *"Do not run it again"* — and the model carries on from there. If nothing ran here
595
+ at all, no bundle is sent: the model's own call goes ahead, unless a step was sent and never
596
+ answered, in which case the note alone takes the call's place.
597
+
598
+ Every hand-over note ([`handover.ts`](../src/replay/handover.ts)) — here, in a direct run's answer,
599
+ or in place of a call — opens with *"[BaseInstRunner calculated replay] The calculated scenario for
600
+ this request stopped before finishing. Continue the task yourself from here, calling tools as
601
+ needed."* It says which steps ran (*"Steps 1–k of N ran under the scenario … Do not repeat them."*,
602
+ split between the tool calls above and the results below when both exist; or *"No step of the
603
+ scenario ran; the whole task is yours."*), then names the stopped step and what may be assumed
604
+ about it:
605
+
606
+ - not sent: *"Step k of N (tool) did not run: … Nothing of it happened; do it yourself."*
607
+ - sent, and never answered: *"… was sent but did not answer (…). It may or may not have completed:
608
+ check whether its work happened before you run it again."* When nothing else ran, the note says
609
+ *"No step of the scenario is known to have finished; the task is yours."* instead of *"No step …
610
+ ran"*.
611
+ - its tool ran, its output logic failed: *"… ran — its result is real and its side effects are in
612
+ place — but the scenario could not read its output (…), so it stops there. Do not run it again."*
613
+ - out of time, or cancelled: *"… did not run: the scenario's time budget ran out before it."* /
614
+ *"… the scenario run was cancelled before it."*
615
+ - beside a pin still in flight: *"… was not run by the scenario: another of its calls was still in
616
+ flight, and a scenario runs one call at a time."*, and the pin itself *"was started and has not
617
+ reported back"*.
618
+
619
+ It ends with the remaining planned steps and the parameters derived for this request, within 4 000
620
+ characters.
621
+
622
+ With `BIR_REPLAY_RECORDED_OUTPUTS=1`, a step that could not be *sent* (a built-in, no proxy, off the
623
+ allowlist) is served its **recorded output** instead (§8.1), marked `(recorded)` in the bundle so the
624
+ model is never told a stale result is fresh. A timeout or an upstream error never is.
625
+
626
+ Delivery, best channel first. (A direct plan's own results go through `run_scenario`, §6.)
627
+
628
+ 1. **Bash-clean** — if the divergent call is `Bash`, rewrite the command to `cat` the bundle. The
629
+ model sees ordinary command output. (RRepeat's `bashCleanPath`.) Up to 6 000 characters ride
630
+ inline as a heredoc, delimiter neutralised; a longer text is written to a `0600` file under
631
+ `<BIR_HOME or ~/.baseinstrunner>/tmp` that is removed after an hour, and the command reads it.
632
+ The text is capped at `BASH_CHANNEL_MAX` (25 000 chars), because the host cuts long command
633
+ output too. If the file cannot be written, deny-inject carries the text.
634
+ 2. **Deny-inject** — `permissionDecision: "deny"`, bundle as `permissionDecisionReason`, capped at
635
+ `MAX_REPLAY_REASON` (60 000 chars) with the same fair-share truncation
636
+ (`assembleBundle`/`truncate`, portable verbatim from `payload-executor.ts`). Works for any tool,
637
+ and is read as adversarial — hence last.
638
+
639
+ **Tier 2 — abort.** Composition itself threw (`replay.compose_failed`). Retire the plan, return
640
+ `{}`, continue as an ordinary turn. Outcome `failed`.
641
+
642
+ **The cost line.** RRepeat's Tier 1 is an Agent SDK session, and its own doc calls that the dominant
643
+ cost of a calculated run. Here Tier 1 is N `tools/call`s on connections that are already open:
644
+ **`fallbackCostUsd` is 0.00 by construction**, for every step the proxies can reach.
645
+
646
+ > **Do not port the unconditional-divergence line.** RRepeat's `steerPreTool` ends with a deliberate
647
+ > `return await injectBundle(...)` that bypasses the pin branch on *every* armed plan
648
+ > (`savingCalcRunner.md` §4 documents the consequence: `stepsPinned` is 0 on every row). That was a
649
+ > choice made when the bundle was the only path that ran real tools. It is not this design's choice:
650
+ > pinning is tried first, always, and divergence is what its name says.
651
+
652
+ ### 8.1 Where a recorded output actually comes from
653
+
654
+ RRepeat reads it from `SerializedScenarioStep.recordedOutput`, whose own comment says the field is
655
+ optional because *"it is the server's payload that decides whether to send it, and older payloads do
656
+ not."* Verify this before relying on it: **no payload does.** BaseIn's `scenario_steps` table has
657
+ columns `tool_input_logic`, `tool_output_logic`, `reasoning`, `reasoning_vector` — and no
658
+ `recorded_output`; `serializeScenario` emits no such field. RRepeat's fallback is unreachable in
659
+ production today.
660
+
661
+ There is a path that needs no server change, and the match payload already carries its key:
662
+
663
+ ```
664
+ GET /recordings/runs/:runId → { run, steps } // steps carry tool_name + tool_output
665
+ ↑ matched.runId — the canonical source run this scenario was calculated from
666
+ ```
667
+
668
+ So: fetch the source run **lazily**, on the first `live` step of the first divergence, and index its
669
+ `tool_response` steps by tool name into the same FIFO queue BaseIn's own `dryReplay` builds
670
+ (`recordedOutputsByTool` in `scenarios/replay.ts` — a `Map<toolName, string[]>` drawn in order, with
671
+ a per-tool cursor). One authenticated GET, cached for the turn, and the queue semantics match the
672
+ server's dry replay exactly — which matters, because a scenario that behaves one way under
673
+ `bir scenario replay --dry` and another way here would be untestable.
674
+
675
+ Adding `recorded_output` to `scenario_steps` would make this a field read instead of a fetch, and is
676
+ a reasonable server-side follow-up. It is not a prerequisite.
677
+
678
+ > **[built]** [`src/replay/source-run.ts`](../src/replay/source-run.ts). The type still carries the
679
+ > optional `recordedOutput` field and prefers it when present, so a server that starts sending it
680
+ > needs no change here — but nothing depends on it.
681
+ >
682
+ > **[changed]** Recorded outputs no longer stand in by default: a step with no result stops the
683
+ > plan (§8). The source run is fetched only when `BIR_REPLAY_RECORDED_OUTPUTS=1` (or
684
+ > `ReplayOptions.recordedFallback`) allows a stand-in, and in `bir replay`, which keeps it. The
685
+ > lookup is bounded by the hook's or the plan's deadline and skipped when less than 250 ms remain.
686
+ > A recorded stand-in never counts as work done under the plan (§11.1).
687
+
688
+ ---
689
+
690
+ ## 9. The plan, and state threading across hook calls
691
+
692
+ `ScenarioReplayPlan` ports from RRepeat's [`payload-executor.ts`](../../../BasInstRunner/RRepeat/src/server/payload-executor.ts)
693
+ with three changes.
694
+
695
+ ```ts
696
+ // src/replay/plan.ts
697
+ export class ScenarioReplayPlan {
698
+ readonly scenarioId: string;
699
+ readonly mode: ReplayMode;
700
+ readonly stepCount: number;
701
+
702
+ /** Resolves when derivation has landed and paramsLogic has been applied. */
703
+ ready(): Promise<void>;
704
+
705
+ currentStepIndex: number;
706
+ expectedTool(): string | undefined;
707
+ isDone(): boolean;
708
+
709
+ /** Pinned input for the current step. Throws → the plan hands over (§8). */
710
+ toolInputForCurrentStep(): Record<string, unknown>;
711
+ /** Thread a real output into respParams and advance. Throws → the plan hands over (§8). */
712
+ applyOutput(realOutput: string): void;
713
+
714
+ steeringDirective(): string;
715
+ /** Execute every remaining step through `execute`, and lay the results out under `maxChars`. */
716
+ composeBundle(maxChars: number, execute: ExecuteStep): Promise<BundleResult>;
717
+ /** responseParamsLogic over the accumulated respParams. */
718
+ responseModel(): Record<string, unknown>;
719
+ }
720
+ ```
721
+
722
+ **Change 1 — lifetime is a run, not a session.** The plan hangs off `RunState`, not `SessionState`.
723
+ In this codebase a run *is* a turn (`UserPromptSubmit → Stop`, `server.ts` §onPrompt), which is
724
+ exactly the plan's lifetime. `sealRun` retires it; a second prompt in the same session cannot inherit
725
+ a stale plan, which in RRepeat requires care.
726
+
727
+ **Change 2 — `params` are a promise.** The constructor takes `Promise<Record<string, unknown>>`, not
728
+ a value, so arming does not wait for the Haiku call (§10). `paramsLogic` is applied once, inside
729
+ `ready()`, the first time it resolves.
730
+
731
+ **Change 3 — `execute` is a step, not a tool name.** `composeBundle` takes
732
+ `(step, input) => Promise<string>` rather than `(toolName, input)`, because the executor must know
733
+ the step's `reach` to decide between the proxy, the recorded output (only when opted in, §8.1), and
734
+ stopping the plan.
735
+
736
+ The four `new Function` shims (`evalParamsLogic`, `evalToolInputLogic`, `evalToolOutputLogic`,
737
+ `evalResponseParamsLogic`) go in `src/replay/logic.ts`, ported from BaseIn's
738
+ [`src/scenarios/logic.ts`](../../../BaseIntSever/BaseIn/src/scenarios/logic.ts) — which is the same
739
+ code with better comments and an `assertPlainObject` guard the other copies lack. **Port BaseIn's
740
+ version**, not RRepeat's: it is the one the server evaluates against, so a divergence between them is
741
+ a divergence between what a scenario does in a dry replay and what it does here. (Each body is
742
+ still parsed as a `new Function` body, but now runs through `vm.Script#runInThisContext` under a
743
+ time limit, §13.1; argument order and the plain-object guard still match the service.)
744
+
745
+ ### 9.1 What flows between steps
746
+
747
+ - **`params`** — computed once (derivation → `paramsLogic`), read by every `toolInputLogic`.
748
+ - **`respParams`** — starts `{}`; each step's `toolOutputLogic` merges keys in
749
+ (`respParams = { ...respParams, ...derived }`); later steps' `toolInputLogic` read the accumulation.
750
+ This is how step 1's `highestRiskDeviceId` reaches step 2's input.
751
+
752
+ In direct mode both live inside one `for` loop. In steer mode they are split across *separate HTTP
753
+ requests* — which is what §7.3's ordering rule protects.
754
+
755
+ ---
756
+
757
+ ## 10. Timing: what may block, and for how long
758
+
759
+ The hook is on the user's critical path. Every await here is a stall the user watches.
760
+
761
+ | Moment | Route | Hook timeout | Answered by | What may be awaited | Budget |
762
+ |---|---|---|---|---|---|
763
+ | `UserPromptSubmit` | `/session/prompt` | 10 s | 7 s | the match; for a scenario with a target, its derivation; a steer plan's argument preview | match `BIR_MATCH_BUDGET_MS` = **1500**; match and arming together ≤ **6 s**; derivation ≤ `BIR_HOOK_DERIVE_BUDGET_MS` = **4000** |
764
+ | `PreToolUse` | `/tool/pre` | 15 s | 12 s | `plan.ready()` (derivation); on a divergence, the remaining steps | derivation ≤ min(`BIR_DERIVE_BUDGET_MS`, `BIR_HOOK_DERIVE_BUDGET_MS`) = **4000**; all replay work ≤ `BIR_PRETOOL_REPLAY_BUDGET_MS` = **9000** (held at ≤ 10 000) |
765
+ | `run_scenario` | `/scenario/run` | `bir-scenario`'s own, **290 s** (`BIR_SCENARIO_RUN_TIMEOUT_MS`) | — | `ready()` + N steps | derivation `BIR_DERIVE_BUDGET_MS` = **8000**; plan `BIR_REPLAY_PLAN_BUDGET_MS` = **100000** (`BIR_REPLAY_BUDGET_MS` overrides it where set); per step `BIR_STEP_TIMEOUT_MS` = **60000**, clipped to what the plan has left |
766
+ | `PostToolUse` | `/tool/post` | 5 s | 2 s | nothing | — |
767
+ | `Stop` | `/session/stop` | 5 s | 2 s | nothing (report is queued) | — |
768
+
769
+ "Hook timeout" is what `bir install` writes into
770
+ [`HOOK_ROUTES`](../src/config/adapters/claude-code.ts): the user's worst case if the control server
771
+ hung. "Answered by" is the control server's own deadline for the route, well inside it: past it the
772
+ route answers `{}` (at `/tool/pre`, a divergence's hand-over note if one was under way) and logs
773
+ `control.hook_deadline`. Every scenario logic body is stopped after `BIR_LOGIC_TIMEOUT_MS`
774
+ (**1000**, at most 10 000). Values set past a ceiling are brought back under it and logged as
775
+ `replay.budget_clamped`. [production.md](../production.md) has the full budget and environment
776
+ reference.
777
+
778
+ **Derivation is awaited at the prompt only when it must be.** The steering directive needs only the
779
+ scenario's ordered tool names and reasonings — it does not need parameter values. So for a scenario
780
+ whose parameters are all settings the prompt hook returns the directive the moment the match lands,
781
+ and the Haiku call runs un-awaited; the *first* `PreToolUse` awaits it, within its own hook budget.
782
+ A scenario with a *target* derives before its directive goes out, inside the prompt's budget, so a
783
+ plan that cannot find what it is to act on is never announced (gate 7, `segmented.md` R-PARAM-4). A
784
+ steer plan with a shell or file-write step may spend up to 1.2 s of that budget letting its
785
+ parameters settle, so that its directive can show them (§7). On a mode-`direct` turn the wait lands
786
+ inside `run_scenario` instead, which waits the full `BIR_DERIVE_BUDGET_MS`.
787
+
788
+ **A direct run is held under 270 s**, so it answers before `bir-scenario` stops waiting: derivation
789
+ plus plan plus 12 s of slack at the default logic limit (one last step's 1 s minimum, six logic
790
+ bodies at their limit, a 5 s margin); past the ceiling the derivation wait is cut to 60 s first,
791
+ then the plan. The defaults come to 120 s, or 140 s with `BIR_REPLAY_BUDGET_MS=120000`. A plan budget
792
+ much past 100 s also risks Claude Code moving a long `run_scenario` call to the background after 2
793
+ minutes, while its steps keep starting. `bir replay` is not held to the ceiling.
794
+
795
+ **Budget expiry is never an error.** Match budget expired → no plan, ordinary turn, one
796
+ `replay.decision` line (`match_timeout`). **The match gate:** a match that answered slower than
797
+ `BIR_MATCH_SLOW_MS` (**800**), or not at all, also makes prompts stop waiting on the match for 5
798
+ minutes, doubling up to 30 while the service stays slow; a fast match ends the hold
799
+ (`replay.match_gate`). A healthy service answers in 300–450 ms, so past 800 ms waiting would add
800
+ that much to every prompt. The match is still asked meanwhile, so a hit still stops the recording. Derivation budget expired →
801
+ settings take their recorded `sampleValue`s and targets stay `null`, never `{}`; a plan that needs a
802
+ missing target, or whose `paramsLogic` throws, hands over instead of running (in steer mode the
803
+ note replaces the call, in direct mode `run_scenario` answers with it). The derivation request
804
+ itself times out after max(`BIR_SERVICE_TIMEOUT_MS`, `BIR_DERIVE_BUDGET_MS`). None of these waits
805
+ happens while the service is cooling down after a failure: the turn simply runs normally.
806
+
807
+ **Intent arming mid-turn is opt-in.** An intent probe is sent from `PreToolUse` for hit counting and
808
+ is never waited on, unless `BIR_INTENT_ARM=1` or `BIR_SEGMENT_ARM=1` lets it arm a plan in that
809
+ call's place; then it waits at most `BIR_INTENT_ARM_BUDGET_MS` (**500**) for the probe and 1.5 s for
810
+ arming. `BIR_INTENT_ARM=0` never waits.
811
+
812
+ `UserPromptSubmit` must stay non-`async`: an async hook is fire-and-forget and cannot return
813
+ `additionalContext` at all.
814
+
815
+ ---
816
+
817
+ ## 11. Reporting: outcome, ticket, cost
818
+
819
+ This is the half that makes replay *provable* rather than merely fast, and v1 does none of it.
820
+
821
+ ### 11.1 The outcome ladder
822
+
823
+ ```ts
824
+ // Upgrade-only. `not_steered` is the floor and the safe answer if the turn dies mid-flight.
825
+ const OUTCOME_RANK = { not_steered: 0, failed: 1, fell_back: 2, diverged: 3, steered_full: 4 } as const;
826
+ ```
827
+
828
+ | Outcome | Means | Server-side effect |
829
+ |---|---|---|
830
+ | `not_steered` | Matched, declined at a gate, or retired before it ran anything (a direct plan whose `run_scenario` never came, §6.4); the agent ran the task | **baseline sample** (`addBaselineSample`) |
831
+ | `failed` | Armed, then the machinery gave up before any step did its work; the agent ran the task | **baseline sample** |
832
+ | `fell_back` | Some steps did their work under the plan, then it handed the rest of the task to the model | ledger row, measured against the whole turn (may be negative) |
833
+ | `diverged` | The model went off-script and the bundle answered; or a direct run's budget ran out after some steps | ledger row, `saved_usd` computed |
834
+ | `steered_full` | Every planned step ran with scenario-derived inputs | ledger row, `saved_usd` computed |
835
+
836
+ `diverged` is booked only when something ran under the plan — a step executed now, or a pin that
837
+ threaded before the divergence. A divergence that ran nothing books `failed`, and so does one that
838
+ stopped at a step that broke (a logic failure, a tool error, a call sent and never answered). A
839
+ recorded stand-in (`BIR_REPLAY_RECORDED_OUTPUTS=1`) is never work done: it counts toward neither
840
+ `stepsPinned` nor the upgrade, and a direct run that used one is `diverged` at best. A call that ran
841
+ beside a pinned one hands over as `fell_back` or `failed` (§7). A direct run cancelled by its caller
842
+ (Esc, or `bir-scenario`'s own timeout) books `fell_back` or `failed` with `measured: false`, never
843
+ `diverged`: that turn's cost says nothing about what the task costs.
844
+
845
+ Direct mode reports `steered_full`. BaseIn's zod enum
846
+ (mirrored in [`replay/types.ts`](../src/replay/types.ts)) has no `direct` member, and `steered_full`
847
+ means exactly what happened: every planned step executed under the scenario's control and the
848
+ response model was produced. A `direct` discriminator would be a *server* change and is §20's
849
+ question 2, not a blocker.
850
+
851
+ ### 11.2 The ticket
852
+
853
+ `matched.executionTicket` is the server's claim token, and **it is the execution row's id** — which
854
+ is what makes a doubled report book once (`scenarios/routes.ts`: a `claimed` ticket returns
855
+ `200 { duplicate: true }`, not an error). Redeem it exactly once, in `sealRun`, which queues the
856
+ report and returns; the run's drain — its own `sealing` promise (the answer resolution), then a
857
+ bounded flush — runs in the background, so `Stop` never blocks on the network:
858
+
859
+ ```
860
+ POST /scenarios/:scenarioId/executions
861
+ { ticket, outcome, deriveCostUsd, sessionCostUsd, fallbackCostUsd,
862
+ durationMs, stepsPlanned, stepsPinned, measured, pricingVersion, prompt,
863
+ steps, error, errorStage, errorStepIndex, errorToolName }
864
+ ```
865
+
866
+ An **unredeemed** ticket leaves no execution row at all. That asymmetry is intentional and worth
867
+ preserving: silence loses a saving; it never invents one.
868
+
869
+ ### 11.2a What each step did (the service's errorshandling.md)
870
+
871
+ The last five fields are the diagnosis, and none of the money depends on them. The service's
872
+ recording page draws the scenario's chain step by step; without these, a scenario whose step 2
873
+ throws on every replay renders as four healthy-looking steps beside a saving that never grows. The
874
+ error exists — in a log line, on the operator's machine. The person who owns the scenario is looking
875
+ at a web page.
876
+
877
+ ```ts
878
+ steps: Array<{
879
+ stepIndex: number; // the scenario's own index, not a position in this array
880
+ toolName: string;
881
+ status: "ok" | "recorded" | "skipped" | "failed";
882
+ stage?: ExecutionStage; // where in the step, for `failed` / `skipped`
883
+ error?: string; // message only — this is rendered in a web page
884
+ durationMs?: number;
885
+ }>
886
+ ```
887
+
888
+ `ok` and `recorded` are passes (`recorded` with an asterisk: the values came from the source run,
889
+ which happens only with `BIR_REPLAY_RECORDED_OUTPUTS=1` and in `bir replay`). `skipped` produced no
890
+ result here: its call could not be sent (a built-in outside the session, no proxy for its server,
891
+ only another open session's proxy), or it was sent and timed out or was refused. The plan stops at
892
+ a `skipped` step. **`failed` is the only one that means the scenario is broken** rather than
893
+ the environment — it is the scenario's own logic throwing, and it is what `Recalculate` fixes.
894
+
895
+ Three rules this must keep:
896
+
897
+ 1. **A step the turn never reached is absent**, not `skipped`. The page shows those as "not run",
898
+ and a turn that dies at step 2 of 5 must not paint steps 3-5 red.
899
+ 2. **Every field is optional on the wire.** An older service ignores them; the costs still book.
900
+ 3. **The headline and the steps are one statement.** `error*` is lifted from the first `failed`
901
+ step (`blameStep`), never assembled independently, so the page's card cannot contradict the step
902
+ list underneath it.
903
+
904
+ The verdicts come from `plan.ts`'s `onStep`, which has always known them — with one addition: a
905
+ logic failure is announced *before* the throw propagates (`reportingStage`), because after it
906
+ nothing knows which step the stack trace belonged to. In steer mode there is no loop to observe, so
907
+ the controller records the verdict at both hooks: `PreToolUse` for an input-logic throw, `PostToolUse`
908
+ for the pass or the output-logic throw. `ReplayState.stepResults` is keyed by step index so the
909
+ second word on a step replaces the first.
910
+
911
+ One consequence for the rule that a costless decline is not worth sending (`buildReport`, and the
912
+ service's matching `202 { recorded: false }`): it now *is* worth sending when a step broke. No cost
913
+ **and** no error remains nothing to report.
914
+
915
+ ### 11.3 Cost, and a live pricing hazard
916
+
917
+ | Component | Source | Notes |
918
+ |---|---|---|
919
+ | `deriveCostUsd` | our own Haiku call's `usage`, priced locally | the only tokens replay itself spends |
920
+ | `sessionCostUsd` | transcript usage **delta** for this run (§11.4) | the live turn: prompt, tool calls, answer |
921
+ | `fallbackCostUsd` | **0.00** | direct execution spends no tokens (§8) |
922
+
923
+ > ### ⚠ Verify the pricing table before copying it
924
+ >
925
+ > A saving is `baseline − actual`, where the baseline is priced by BaseIn's
926
+ > [`src/scenarios/cost.ts`](../../../BaseIntSever/BaseIn/src/scenarios/cost.ts) and the actual by the
927
+ > runner's copy. Both currently declare `PRICING_VERSION = '2026-08-30'`, and **the two tables
928
+ > already disagree**: RRepeat prices `claude-opus-4-6` at `3.0 / 15.0`, BaseIn at `5.0 / 25.0`. Equal
929
+ > versions are supposed to mean *"these are comparable"*. Right now, for that row, they are not.
930
+ >
931
+ > So: **copy BaseIn's table, not RRepeat's** — the server owns the baseline, and the baseline is the
932
+ > number a saving is measured *against*. Copy `PRICING_VERSION` with it. Treat any price change as a
933
+ > three-repository change, and file the drift above against RRepeat rather than reproducing it.
934
+ >
935
+ > Practical exposure here is small — derivation runs on `claude-haiku-4-5-20251001`, which both
936
+ > tables price identically at `1.0 / 5.0` — but the invariant is what matters, not today's blast
937
+ > radius.
938
+
939
+ ### 11.4 The per-run usage watermark
940
+
941
+ `savingCalcRunner.md` §3 documents this trap in RRepeat, and **BaseInstRunnerMCP has it too**, for
942
+ the same structural reason: Claude Code writes one transcript per *session*, and `onPrompt` rolls a
943
+ new run on every prompt. Summing the whole file gives run 3 the cost of runs 1+2+3.
944
+
945
+ Every one of those numbers is a **baseline** on the server. Do not ship the whole-file version.
946
+
947
+ ```ts
948
+ // src/control/transcript.ts — two new exports beside contextForToolUse / settledLastAssistantText
949
+ /** Per-model cumulative usage, captured when a run starts. */
950
+ export function markTranscriptUsage(path: string | undefined): UsageMark;
951
+ /** Per-model delta since `mark`, clamped at 0 per field. */
952
+ export function usageSince(path: string | undefined, mark: UsageMark): ModelUsage[];
953
+ ```
954
+
955
+ Take the mark in `ensureRun`, beside `startedAtMs`, so the cost window and the duration window are
956
+ the same window. Clamp per field, so a compacted or rotated transcript degrades to *"no usage"*
957
+ rather than to a negative cost. A run with **no mark** (one opened lazily by a proxy step before any
958
+ prompt) reports `measured: false` — better to say "unmeasured" than to hand the ledger an inflated
959
+ baseline.
960
+
961
+ `measured` is `true` only when a mark existed **and** `pricingVersion` matches the server's. The
962
+ server enforces the second half itself and stores a mismatched report as unmeasured rather than
963
+ trusting it.
964
+
965
+ ---
966
+
967
+ ## 12. What the replayed turn records
968
+
969
+ **Nothing, and that is correct.** The service answered `{ id: null }`; no run exists; every step post
970
+ would 404. `run.recording = false` already handles it — `record()` early-returns and `sealRun` skips
971
+ both the final answer and `finishRun`.
972
+
973
+ Three consequences to keep straight:
974
+
975
+ 1. **Step indices are still allocated.** `ordering` runs regardless, correlations are still minted
976
+ and settled, and `bir doctor` still shows a live step count. Only the *sending* stops.
977
+ 2. **The matched run is the canonical one.** The server bumped its `iterations` at match time. The
978
+ corpus does not grow on a replay, by design — that is what stops one popular task from flooding
979
+ the similarity pool with near-duplicates.
980
+ 3. **Direct-mode calls bypass the recorder entirely.** They are issued via `UpstreamClient.request`,
981
+ which allocates a `bir:` id and consumes its own response in `accept()` before the relay's
982
+ handlers ever see it ([`src/upstream/client.ts`](../src/upstream/client.ts)). So
983
+ `RecordingInterceptor` never observes them and cannot double-record. That interface comment says
984
+ `request()` is *"Never used by the relay"* — it was built for `bir doctor` and tests, and this is
985
+ its second, load-bearing user.
986
+
987
+ ### 12.1 `bir replay` — the Tier 2 and debugging path
988
+
989
+ ```bash
990
+ bir replay --scenario scn_… --prompt "check the fleet for high-risk devices" # run it
991
+ bir replay --scenario scn_… --prompt "…" --dry # derive + plan only
992
+ ```
993
+
994
+ Runs the same plan through the same executor, against the same live proxies, printing the per-step
995
+ input/output and the response model. It is how you test a scenario without a session, how a Tier 2
996
+ or non-Claude-Code client gets any replay at all, and the first thing to reach for when a steered
997
+ turn behaves oddly. The operator named the scenario, so it keeps its own contract: a step it cannot
998
+ send stands on its recorded output, the allowlist does not gate it, and it is not held to the 270 s
999
+ direct-run ceiling (§10).
1000
+
1001
+ ---
1002
+
1003
+ ## 13. Security
1004
+
1005
+ v1 §9 holds unchanged. Replay adds two exposures, and neither is small.
1006
+
1007
+ ### 13.1 The scenario payload is executable code
1008
+
1009
+ `toolInputLogic`, `toolOutputLogic`, `paramsLogic` and `responseParamsLogic` are JavaScript function
1010
+ *bodies*, compiled as function bodies and run **in the control server process**, in that process's
1011
+ own context (`vm.Script#runInThisContext`) — a process holding the user's BaseIn credentials and,
1012
+ through the proxies, live authenticated access to every wrapped MCP server.
1013
+
1014
+ BaseIn's own `logic.ts` argues the eval is safe because *"these run only over data the owner already
1015
+ produced… so the dynamic eval is scoped to first-party content."* That argument holds here too, and
1016
+ the blast radius does not: BaseIn evaluates them against a database row, we evaluate them next to a
1017
+ browser and a GitHub token.
1018
+
1019
+ What actually constrains it:
1020
+
1021
+ - The payload arrives **only** in the response to an authenticated `POST /recordings/runs` against
1022
+ the configured `BIR_AUTH_URL`, over the user's own token, describing the user's own runs. It is not
1023
+ attacker-supplied in any ordinary threat model.
1024
+ - A body gets its declared arguments **and the global scope**. `require` and `import` are not
1025
+ reachable (ESM, no CommonJS wrapper), but `fetch`, `process` and `globalThis` are. Say this
1026
+ plainly rather than implying a sandbox: **there is no sandbox.**
1027
+ - **A time limit, and nothing more.** Each body is stopped after `BIR_LOGIC_TIMEOUT_MS` (default
1028
+ 1000 ms, at most 10 000). A regex that backtracks on a larger output than the recording's, or a
1029
+ loop saved with `--force`, then fails as a logic error and the plan hands over at that step,
1030
+ instead of freezing every hook of the project. The limit covers synchronous work only:
1031
+ continuations a body queues (promises, `queueMicrotask`, timers) run afterwards and are not
1032
+ bounded, and a body that returns a promise or any thenable is refused. There is no memory limit.
1033
+ - Replay is opt-in per project (D14): `bir install --replay`, which `bir setup` passes, turns it on,
1034
+ and a `bir install` without it stores replay off in a project that never asked for it. A
1035
+ compromised or misconfigured `BIR_AUTH_URL` therefore cannot start executing logic on a machine
1036
+ whose operator never asked for replay.
1037
+
1038
+ If that is not enough for a deployment, the mitigation is a real one and is out of scope here:
1039
+ evaluate the logic in a separate `node:vm` context with a frozen, minimal global. Filed as §20 question 3.
1040
+
1041
+ **A hand-edited step is the same code, and runs the same way.** Since 2026-09-25 (the service's
1042
+ `editSteps.md`; built, not yet released) the owner — or Claude working for them — can change a step's
1043
+ logic with `bir scenario edit`. The service checks the change against the recording before it saves
1044
+ it, but that check is about *what the code computes*. Once saved, the step arrives in the same
1045
+ payload and runs here unattended, in the same process, next to the same credentials, exactly like
1046
+ calculated code. The provenance argument still holds: only the owner can edit, over their own token,
1047
+ and anyone else — an admin included — gets `404`. What changes is that *"the model wrote it"* is no
1048
+ longer always true.
1049
+
1050
+ The service sandboxes its own evaluation of every logic body (built with the same service change;
1051
+ not yet released). It had to: there, one account's code runs next to every account's data and the
1052
+ service's secrets. How it is built:
1053
+
1054
+ - **A fresh `node:vm` context for each evaluation**, with a time limit (`LOGIC_TIMEOUT_MS`). Only
1055
+ JavaScript's own built-ins are in it, less `ArrayBuffer`, `SharedArrayBuffer`, `DataView`, the
1056
+ typed arrays, `Atomics`, `WebAssembly` and `FinalizationRegistry`; `Symbol.for` and
1057
+ `Symbol.keyFor` are removed too. No `require`, `process`, `fetch`, `Buffer` or timers. Values go
1058
+ in and come out as JSON text only.
1059
+ - **That context lives in a separate evaluator process**, not in the service: an empty
1060
+ environment, `--permission` (no files, no child processes), `--disallow-code-generation-from-strings`,
1061
+ `--frozen-intrinsics`, and a heap of `LOGIC_MEMORY_MB`.
1062
+ - **A relay worker thread of the service drives that process.** No logic runs in the relay. It
1063
+ kills the evaluator when a body runs too long, and the next job starts a new one; a body that
1064
+ runs out of memory takes down only the evaluator, and is run once more on a fresh one before it
1065
+ is blamed. The relay also replaces the evaluator when its heap stays above half of
1066
+ `LOGIC_MEMORY_MB` after a job, and after 2,000 jobs. A body that overran its time or memory, or
1067
+ took the evaluator down twice, is refused at once, with the same message, for the next 10
1068
+ minutes.
1069
+ - **The hand-edit check runs in a check worker thread of its own**, with its own relay and
1070
+ evaluator process, under a sandbox-time budget
1071
+ (`EDIT_CHECK_BUDGET_MS`, 5000 ms by default; past it, *too costly to check*), so a slow check
1072
+ never holds up the service's other requests.
1073
+
1074
+ **The runner still does not sandbox.** Here the code runs on the owner's machine, for the owner's
1075
+ account, in the control server's own context with a time limit and nothing else. One consequence
1076
+ to know: **the service's check only judges what a
1077
+ body returns for the recording's values.** Code that calls `fetch`, `process` or a timer on that
1078
+ path, and does not catch the error, throws in the check, so it cannot be saved without `--force`.
1079
+ But a call behind a `typeof` guard (`if (typeof fetch === "function") …`), or on a branch the
1080
+ recording's values never take, is never reached in the check; one inside a `try` fails there
1081
+ quietly. Such code passes, it is saved like any other change, and here it runs with `fetch`,
1082
+ `process` and timers all present. The service's sandbox protects the service; it proves
1083
+ nothing about what a body does on the runner. Read every hand edit before it is saved.
1084
+
1085
+ ### 13.2 Replay bypasses permission prompts
1086
+
1087
+ This is the sharpest edge in the whole design, and it must not be buried.
1088
+
1089
+ - **Steer mode** answers `PreToolUse` with `permissionDecision: "allow"` for a pinned call — one
1090
+ the pin guard (§7) accepts as the step — so it is auto-approved, **built-ins** included (`Bash`,
1091
+ `Edit`). Nothing else is: an ordinary wrapped MCP call gets no decision from the runner, and
1092
+ Claude Code asks about it as it would without the runner, unless the operator set
1093
+ `BIR_CORRELATION_DECISION=allow` where the recorder starts (see `bir-hooks.ts`).
1094
+ - **Direct mode** never reaches `PreToolUse` at all. A `tools/call` dispatched to a proxy is not
1095
+ visible to Claude Code's permission system in any form. The `run_scenario` call that starts it is
1096
+ answered `allow` while the runner has announced a direct plan, so it brings no surprise prompt
1097
+ (and no denial under `claude -p`); the user's own deny or ask rules for it still apply.
1098
+ - **Plan mode** (`permission_mode: "plan"`) gets none of it: no directive, no pin, no `allow`.
1099
+
1100
+ So a scenario is, in effect, a **pre-approved list of tool calls with computed arguments**. That is
1101
+ precisely what makes it valuable, and precisely what makes it dangerous. Six mitigations, all
1102
+ required:
1103
+
1104
+ 1. `--replay` is a distinct flag: `bir setup` passes it, and a `bir install` without it leaves
1105
+ replay off in a project that never asked for it. `bir replay off` or `BIR_REPLAY=0` turns it off
1106
+ once installed, and `bir off` takes the whole runner out of the way.
1107
+ 2. `BIR_REPLAY_ALLOW_SERVERS` — a comma-separated allowlist of the server keys bir may call by
1108
+ itself: **direct** runs and divergence recovery (§8), though not `bir replay`, which an operator
1109
+ runs by hand. Unset means *all wrapped servers*; the guide (§5 of `calculatedReplayGuide.md`)
1110
+ recommends setting it.
1111
+ 3. `bir doctor` prints `replay: on` and the effective allowlist, loudly, at the top. A capability this
1112
+ sharp must never be discoverable only by reading a log.
1113
+ 4. Every armed plan logs `plan.armed` with the scenario id, mode, step count and the ordered tool
1114
+ names **before** the first call runs. The audit line precedes the action.
1115
+ 5. The pin guard (§7): a pin replaces the arguments of a call only when that call plausibly is the
1116
+ step, so a model's `Bash ls` is never turned into a step's `git push`.
1117
+ 6. The injection guard ([`safety.ts`](../src/replay/safety.ts)). In a plan with a step whose input
1118
+ is run — `Bash`, `PowerShell`, or an MCP tool or server named for SQL, queries, `exec` or a shell,
1119
+ in any naming convention — a value derived for this turn that differs from its recorded sample
1120
+ must be free of shell and SQL metacharacters and keep the sample's shape (a number stays a
1121
+ number, nothing starts with `-`, no whitespace, glob, brace or parenthesis the sample did not
1122
+ have), or the plan hands over; found at arm time, it declines as `missing_target`. A plan whose
1123
+ only such steps write files (`Edit`, `MultiEdit`, `Write`, `NotebookEdit`) rejects only line
1124
+ breaks and NUL. Plans with a step whose input is run do not send recent tool results to
1125
+ derivation, and the derivation prompt is redacted before it leaves the machine.
1126
+
1127
+ ### 13.3 The control plane
1128
+
1129
+ `/scenario/run`, `/scenario/hello`, `/proxy/poll` and `/proxy/result` are ordinary control-server
1130
+ routes: loopback only, bearer token from the `0600` discovery file, same as `/proxy/step`.
1131
+ `/proxy/poll` holds a connection open for up to 25 s — a long-lived authenticated request, not a
1132
+ new trust boundary.
1133
+
1134
+ ---
1135
+
1136
+ ## 14. Degradation matrix
1137
+
1138
+ The governing rule is unchanged and outranks every optimisation here: **the host session must never
1139
+ fail because of BaseInstRunner.**
1140
+
1141
+ | Failure | Behaviour |
1142
+ |---|---|
1143
+ | `getMatch()` slower than the match budget | No plan. Ordinary turn. One `replay.decision` line (`match_timeout`); a match that lands later is `match_late` and never steers. A slow match also makes prompts stop waiting on it for a while (`replay.match_gate`, §10) |
1144
+ | The service is cooling down after a failure | Nothing waits on it: no match wait (`service_unhealthy`), no derivation wait, no intent probe. Ordinary turn |
1145
+ | No derivation key here (`BIR_DERIVE_API_KEY`; the background recorder ignores `ANTHROPIC_API_KEY` unless `BIR_USE_ANTHROPIC_API_KEY=1`) | The service derives instead (segmented.md R-PARAM-5). Signed out as well: settings take their recorded `sampleValue`s, a scenario with a target declines `no_derive_key`, `deriveCostUsd: 0` |
1146
+ | The service declines, is slow, or cannot be reached | The same, with its reason carried into the `replay.derived` and `replay.decision` lines. Never a throw |
1147
+ | Derivation call fails or times out | Settings take their samples and targets stay `null`, never `{}`; a plan that needs a missing target hands over instead of running. Plus a `replay.derive_failed` or `replay.derive_timeout` line |
1148
+ | `paramsLogic` or `toolInputLogic` throws | The step is recorded as failed and the plan hands over: the note replaces the call (steer), or is `run_scenario`'s answer (direct). Never a crash |
1149
+ | `toolOutputLogic` throws | The step's tool ran, so the plan hands over with a note that says so (*"Do not run it again"*); outcome `fell_back` |
1150
+ | A logic body runs past `BIR_LOGIC_TIMEOUT_MS` | Stopped (*"… ran longer than N ms and was stopped"*), and handled as that logic throwing |
1151
+ | Model calls an unexpected tool (steer plan) | Divergence (§8) |
1152
+ | A direct plan's first call is not `run_scenario`, or its `run_scenario` never reaches the runner | Retired without running anything (`replay.vetoed`), `not_steered`; the call runs as written (§6.4) |
1153
+ | The expected tool, but not plausibly the step | No pin: it runs as written and the plan stays armed (`replay.pin_declined`). A call to another tool next retires the plan (`replay.pin_abandoned`) |
1154
+ | A call arrives while a pinned step is still running | It waits briefly for the pin to thread; still beside it, the plan hands over (`replay.concurrent_call`), `fell_back` or `failed` |
1155
+ | A pinned step fails (`PostToolUseFailure`, an error report from its proxy) | The plan hands over; nothing after it is pinned or approved |
1156
+ | A step cannot be sent: a built-in reached by a divergence, no proxy polling for its server, a server off `BIR_REPLAY_ALLOW_SERVERS` | Nothing is sent. The plan stops at that step and hands over (*"did not run … Nothing of it happened"*). With `BIR_REPLAY_RECORDED_OUTPUTS=1` its recorded output stands in instead |
1157
+ | Two sessions are open, and none of the server's proxies is known to be this session's | Nothing is sent: the step is not run, the plan hands over, and the model calls the tool on its own upstream. A lone session uses its unbound proxy as before |
1158
+ | A direct step exceeds `BIR_STEP_TIMEOUT_MS` | The proxy cancels the call upstream (`notifications/cancelled`) just before the control server gives up; the plan stops and hands over (*"was sent but did not answer … may or may not have completed"*). No recorded output stands in |
1159
+ | Whole plan exceeds its budget (`BIR_REPLAY_PLAN_BUDGET_MS`) | Return what completed behind a hand-over note, marked partial; outcome `diverged` (`failed` when no step had run) |
1160
+ | `run_scenario` cancelled (Esc) or `bir-scenario`'s 290 s timeout | No further step starts. The note saying which steps ran reaches the model on its next prompt or tool call; `fell_back` or `failed`, `measured: false` |
1161
+ | A `PreToolUse`'s replay work reaches its budget or deadline | No divergence step starts past `BIR_PRETOOL_REPLAY_BUDGET_MS`. At the 12 s route deadline the hook answers anyway: `{}` with a plan still getting ready retired (`replay.pin_overran`), or a divergence's note (`replay.diverge_overran`) |
1162
+ | Any hook route passes its deadline | It answers `{}` and logs `control.hook_deadline`; the work finishes in the background |
1163
+ | No running `bir` server has said hello | A direct plan is declined (`no_delivery_tool`); the agent does the task |
1164
+ | `bir-scenario` cannot find the control server | `run_scenario` answers *"Nothing to run for this request; continue normally."* (`isError: false`); the model proceeds normally |
1165
+ | A second `run_scenario` while the plan is running | Refused; the model is told the results come back on the first call |
1166
+ | `run_scenario` pending in two sessions and the request names neither | Nothing runs (`replay.run_ambiguous`); each model does its own task |
1167
+ | The runner is off (`bir off`, or `BIR_OFF=1` for a session) | Hooks answer `{}` and nothing arms; the `bir` server lists no tools |
1168
+ | Plan mode | Nothing is announced, pinned or approved |
1169
+ | Ticket redemption fails (offline, 409) | Logged; the turn is unaffected. An unredeemed ticket books nothing |
1170
+ | Two reports for one ticket (retry, `Stop` racing `SessionEnd`) | Server books once, answers `200 { duplicate: true }` |
1171
+ | Replay off (`BIR_REPLAY=0`, `bir replay off`, or never installed with `--replay`) | Exactly v1: stop recording on a match, do nothing else, and no prompt waits for the match |
1172
+
1173
+ ---
1174
+
1175
+ ## 15. Solution layout
1176
+
1177
+ ```
1178
+ src/
1179
+ ├─ replay/ ← new
1180
+ │ ├─ plan.ts # ScenarioReplayPlan (§9)
1181
+ │ ├─ logic.ts # the four logic evaluators, under a time limit — port BaseIn's (§9, §13.1)
1182
+ │ ├─ safety.ts # the injection guard for derived values (§13.2)
1183
+ │ ├─ derive.ts # Haiku parameter derivation over raw fetch (§16.3)
1184
+ │ ├─ pricing.ts # MODEL_PRICING + PRICING_VERSION — copy BaseIn's (§11.3)
1185
+ │ ├─ coverage.ts # reachOf / modeFor (§5)
1186
+ │ ├─ bundle.ts # assembleBundle / truncate — port verbatim
1187
+ │ ├─ executor.ts # direct step dispatch + the work queue (§16.2)
1188
+ │ ├─ source-run.ts # lazy GET /recordings/runs/:id → recorded outputs by tool (§8.1)
1189
+ │ └─ types.ts # SerializedScenario, SerializedScenarioStep
1190
+ ├─ bin/
1191
+ │ ├─ bir-scenario.ts ← new: the `bir` MCP server — `run_scenario` (§6.3), later the scenario tools (§16.5)
1192
+ │ └─ bir.ts # + `--replay`, `bir scenario …`, `bir replay` (§16.5)
1193
+ ├─ control/
1194
+ │ ├─ server.ts # + arm / pin / thread / dispatch / report
1195
+ │ ├─ pin-guard.ts # whether a call is plausibly the step before it is pinned (§7)
1196
+ │ └─ transcript.ts # + markTranscriptUsage / usageSince (§11.4)
1197
+ ├─ proxy/
1198
+ │ └─ session.ts # + the long-poll work loop (§16.2)
1199
+ ├─ record/
1200
+ │ ├─ recorder.ts # + reportExecution() on the Recorder surface
1201
+ │ └─ remote-recorder.ts # + POST /scenarios/:id/executions
1202
+ └─ config/adapters/claude-code.ts # the hook timeouts (§10); the `bir` MCP entry
1203
+ ```
1204
+
1205
+ `package.json` gains one bin: `"bir-scenario": "dist/bin/bir-scenario.js"`. Runtime dependencies stay
1206
+ at **zero** (§16.3).
1207
+
1208
+ ---
1209
+
1210
+ ## 16. Contracts
1211
+
1212
+ ### 16.1 Control-plane routes (new)
1213
+
1214
+ ```ts
1215
+ // bir-scenario → control server. toolUseId is Claude Code's params._meta["claudecode/toolUseId"];
1216
+ // only the plan of the session whose run_scenario PreToolUse is waiting runs (§6.3).
1217
+ POST /scenario/run { toolUseId?: string, sessionId?: string }
1218
+ → 200 { ok: true, text: string, responseModel: object, steps: number, partial?: boolean }
1219
+ → 200 { ok: false, why: "no_plan" } // no run_scenario pre waiting for it
1220
+ → 200 { ok: false, why: "[BaseInstRunner calculated replay] …", note: true } // hand-over: nothing ran live
1221
+ → 200 { ok: false, why: "this scenario is already running for this turn — …" } // a second call
1222
+ → 200 { ok: false, why: "not_ready" | "the scenario could not be run" } // the run itself threw
1223
+
1224
+ // bir-scenario → control server (never awaited; bounded at 1 s). Proof the delivery tool is live.
1225
+ POST /scenario/hello { pid: number, version: string, tools: number, off: boolean }
1226
+ → 200 { ok: true }
1227
+
1228
+ // proxy → control server (long-poll; holds up to 25 s, then answers empty)
1229
+ POST /proxy/poll { serverName: string, proxyId?: string, startedAt?: number, pid?: number, holdMs?: number }
1230
+ → 200 {} | 200 { work: { workId: string; toolName: string; arguments: unknown; timeoutMs: number } }
1231
+ → 200 { replay: false } // replay is off: poll slowly
1232
+
1233
+ // proxy → control server
1234
+ POST /proxy/result { workId: string, result?: unknown, error?: string }
1235
+ → 200 { ok: true }
1236
+ ```
1237
+
1238
+ ### 16.2 The direct executor
1239
+
1240
+ ```ts
1241
+ // src/replay/executor.ts
1242
+ export type ExecuteStep = (step: SerializedScenarioStep, input: Record<string, unknown>) => Promise<string>;
1243
+
1244
+ /** Hands work to the proxy that owns `serverName` and awaits its result. */
1245
+ export class ProxyWorkQueue {
1246
+ /** Called by POST /proxy/poll. Resolves with work, or with undefined at the poll deadline. */
1247
+ waitForWork(serverName: string, pollDeadlineMs: number): Promise<Work | undefined>;
1248
+ /** Called by POST /proxy/result. */
1249
+ complete(workId: string, result?: unknown, error?: string): void;
1250
+ /**
1251
+ * Dispatch and await one call. Rejects on timeout, on no polling proxy, and with
1252
+ * NoSessionProxyError when no polling proxy can be shown to be `route.sessionId`'s.
1253
+ */
1254
+ call(serverName: string, toolName: string, args: unknown, timeoutMs: number,
1255
+ route?: { sessionId?: string }): Promise<unknown>;
1256
+ }
1257
+ ```
1258
+
1259
+ > **[built] "No poller parked" is not the same as "no proxy".** The first implementation rejected
1260
+ > `call` immediately whenever no poller was waiting, on the reasoning that queued work could stall a
1261
+ > turn past its budget. That reasoning holds for a server no proxy serves — and is wrong for the case
1262
+ > that actually dominates: a proxy is *between* polls precisely while it is POSTing one step's result
1263
+ > and re-polling, which is exactly the gap between two steps of a multi-step plan. The end-to-end test
1264
+ > caught it as "step 2 of 2 never ran".
1265
+ >
1266
+ > So the queue tracks `lastSeen` per server. Work dispatched while a recently-seen proxy is
1267
+ > mid-round-trip is **held** for its next poll; the per-call timeout still bounds the wait, and a
1268
+ > server nothing has served for a minute still fails fast: the step is not sent, and the plan stops
1269
+ > there (§8) unless `BIR_REPLAY_RECORDED_OUTPUTS=1` lets its recorded output stand in.
1270
+ >
1271
+ > **[built] Which proxy.** Every Claude Code session starts its own proxies, and they are not
1272
+ > interchangeable: each holds its own upstream (another browser page, another database session). A
1273
+ > step goes to a proxy bound to the step's session — learned from a correlated call it reported, or
1274
+ > an earlier step of the session it ran — and otherwise to an unbound proxy **only while no other
1275
+ > session is live**. With two sessions open, a session with no bound proxy for the server gets none:
1276
+ > the call rejects with `NoSessionProxyError`, nothing is sent, the step is recorded as `skipped`,
1277
+ > and the plan hands over so the model calls the tool on its own upstream. `bir replay` has no
1278
+ > session and takes the newest proxy.
1279
+
1280
+ Proxy side, in `ProxySession` — started as soon as the proxy is bound, whether replay is on or not:
1281
+
1282
+ ```ts
1283
+ while (!closed && !upstream.dead) {
1284
+ const { work, ...answer } = await control.pollWork(this.serverName, holdMs); // 25 s long-poll
1285
+ if (!work) { pace(answer); continue; } // see below
1286
+ try {
1287
+ // The reserved `bir:` id space, built for exactly this and never used by the relay.
1288
+ // A little under the server's own budget, so a timeout's notifications/cancelled
1289
+ // reaches the upstream before the control server gives up on the step.
1290
+ const budget = work.timeoutMs - Math.min(1_000, work.timeoutMs / 10);
1291
+ const result = await this.upstream.request("tools/call",
1292
+ { name: work.toolName, arguments: work.arguments },
1293
+ AbortSignal.timeout(budget));
1294
+ await control.result(work.workId, result);
1295
+ } catch (err) {
1296
+ await control.result(work.workId, undefined, errText(err));
1297
+ }
1298
+ }
1299
+ ```
1300
+
1301
+ The pace: a held poll re-polls at once; a control server that answers `{ replay: false }`, or
1302
+ answers at once, is polled every 30 s; one that cannot be reached (a recorder restarting) is retried
1303
+ with a backoff of at most 5 s, and the proxy registers again when it answers. The loop never stops
1304
+ on its own, so replay turned on with `bir up --restart` reaches open sessions within about 30 s.
1305
+ An aborted `bir:` request sends `notifications/cancelled` upstream. A proxy report or
1306
+ `/proxy/result` with no result at all is a call that was sent and may or may not have run — never
1307
+ a success.
1308
+
1309
+ Note `work.toolName` is the **upstream-local** name (`navigate_page`), not the qualified one —
1310
+ `parseQualifiedName` splits it at dispatch. Storing both parts is what v1 §5 bought.
1311
+
1312
+ > **[built] The proxy needs no replay configuration of its own.** `/proxy/register` answers
1313
+ > `{ replay: boolean, pollHoldMs }`, and `/proxy/poll` answers `{ replay: false }` while replay is
1314
+ > off, which only slows the loop down. `BIR_REPLAY` is therefore a single switch on the control
1315
+ > server, with no second copy in each wrapped entry's `env` to drift out of step with it.
1316
+
1317
+ ### 16.3 Derivation without a dependency
1318
+
1319
+ ```ts
1320
+ // src/replay/derive.ts — no SDK; ~50 lines of fetch.
1321
+ export async function deriveParameters(opts: {
1322
+ prompt: string;
1323
+ intent: string;
1324
+ paramsObject: Record<string, { sampleValue: unknown; description: string }> | null;
1325
+ apiKey?: string; // BIR_DERIVE_API_KEY — an override (see below)
1326
+ service?: { baseUrl: string; token: string; scenarioId: string }; // the default path
1327
+ model?: string; // default claude-haiku-4-5-20251001
1328
+ signal?: AbortSignal; // default: times out after BIR_SERVICE_TIMEOUT_MS
1329
+ }): Promise<{ params: Record<string, unknown>; costUsd: number; derived: boolean }>;
1330
+ ```
1331
+
1332
+ Behaviour, matching RRepeat's `deriveParametersFromScenarioPayload` exactly so the two produce the
1333
+ same parameters from the same prompt:
1334
+
1335
+ - No `paramsObject` keys → `{}`, `costUsd: 0`.
1336
+ - **No `apiKey` but a `service` → `POST {baseUrl}/scenarios/{id}/derive`** with the turn (prompt,
1337
+ live call, recent results) and a bearer token, `derived` as the service answered. This is the
1338
+ ordinary path: nearly every session runs inside Claude Code on a subscription and has no key of
1339
+ its own (segmented.md R-PARAM-5). The answer is re-normalised here against this runner's own copy
1340
+ of the schema, so the gate reads a `missing` computed from what will actually run.
1341
+ - **Neither → return every key's recorded `sampleValue`**, `costUsd: 0`, `derived: false`. Settings
1342
+ stand on their samples; a scenario with a target declines rather than run on a stale one.
1343
+ - With an `apiKey`, `POST https://api.anthropic.com/v1/messages` with the same extraction prompt,
1344
+ tolerant JSON extraction (first balanced `{…}`, `undefined` → `null`), and any `null`/missing
1345
+ **setting** filled from its `sampleValue` — a `null` target is named in `missing` instead.
1346
+
1347
+ The key comes from a variable set for this purpose alone, `BIR_DERIVE_API_KEY`. The background
1348
+ recorder has Claude Code's environment, and an API-billed Claude Code carries its
1349
+ `ANTHROPIC_API_KEY` there, so the recorder ignores that variable (logging `derive.api_key_ignored`)
1350
+ unless `BIR_USE_ANTHROPIC_API_KEY=1`; a `bir-hooks` run by hand in a terminal still uses it. The
1351
+ prompt (or an intent arm's reasoning) and any live call or recent results are redacted before they
1352
+ leave the machine, either way. The controller gives the request max(`BIR_SERVICE_TIMEOUT_MS`,
1353
+ `BIR_DERIVE_BUDGET_MS`) before it aborts it, and does not wait on the service at all while it is
1354
+ cooling down after a failure.
1355
+
1356
+ Keeping this dependency-free is not purity. `bir-proxy` is spawned inside the host's process tree for
1357
+ *every* wrapped server; an npm install that pulls a model SDK into that path is a startup-latency and
1358
+ supply-chain cost paid N times per session, for a feature most sessions never arm.
1359
+
1360
+ ### 16.4 `Recorder` gains one method
1361
+
1362
+ ```ts
1363
+ export interface ScenarioReporter {
1364
+ reportExecution(r: {
1365
+ scenarioId: string; ticket?: string;
1366
+ outcome: "steered_full" | "diverged" | "not_steered" | "failed" | "fell_back";
1367
+ deriveCostUsd: number; sessionCostUsd: number; fallbackCostUsd: number;
1368
+ durationMs: number; stepsPlanned: number; stepsPinned: number;
1369
+ measured: boolean; pricingVersion: string; prompt?: string;
1370
+ }): void;
1371
+ }
1372
+ export function isScenarioReporter(r: Recorder): r is Recorder & ScenarioReporter;
1373
+ ```
1374
+
1375
+ Optional capability, discovered the same way `isMatchAware` is — so `NullRecorder` needs no change
1376
+ and a session with no BaseIn credentials still replays (it just books nothing).
1377
+
1378
+ ### 16.5 CLI surface
1379
+
1380
+ `bir` gains one flag and one command group. Everything here is a thin client over service
1381
+ endpoints, except `bir replay`, which drives §16.2's executor directly.
1382
+
1383
+ ```
1384
+ bir install --replay register the `bir` MCP server (local scope) and switch replay on here
1385
+ bir uninstall --replay take the `bir` server out and switch replay off; the rest of the install stays
1386
+ bir replay status|on|off|allow <a,b>|allow all
1387
+ this project's replay switches, kept in ~/.baseinstrunner/installed.json
1388
+
1389
+ bir scenario list GET /recordings/runs runs, iterations, scenario state
1390
+ bir scenario show <runId> GET /recordings/runs/:id/scenario intent, params, steps, logic
1391
+ bir scenario show <scnId> GET /scenarios/:id the same, by scenario id (segments too)
1392
+ [--step <n>: one step, with scenarioId and chainRevision first]
1393
+ bir scenario calc <runId> POST /recordings/runs/:id/calculate [--force to re-derive in place]
1394
+ [--force --discard-edits: re-derive a plan with hand edits]
1395
+ bir scenario calc <scnId> --force
1396
+ POST /scenarios/:id/recalculate recalculate in place (segments too)
1397
+ [--discard-edits: sends {discardEdits: true}]
1398
+ bir scenario replay <scnId> --prompt "…" [--dry]
1399
+
1400
+ bir scenario check <id> --step <n> POST /scenarios/:id/steps/:n/check try a change; writes nothing
1401
+ bir scenario edit <id> --step <n> PATCH /scenarios/:id/steps/:n the same check, then save
1402
+ bir scenario edits <id> GET /scenarios/:id/edits the history, newest first
1403
+ bir scenario undo <id> [--edit <e> | --step <n>]
1404
+ POST /scenarios/:id/edits/:e/revert take back the newest edit (of step n)
1405
+ bir scenario editing on|off|status (no route) ~/.baseinstrunner/installed.json which `bir` MCP tools are offered
1406
+
1407
+ bir replay --scenario <scnId> --prompt "…" [--dry] alias; the Tier 2 / debugging path (§12.1)
1408
+ ```
1409
+
1410
+ The second group, and `show <scnId>`, `--step`, `calc <scnId>` and `--discard-edits`, were added
1411
+ on 2026-09-25 by the service's `editSteps.md`. They are built and tested, and not yet released in
1412
+ the form described here (version 0.2.11 on npm has an early form): they need the next
1413
+ `@basein/runner` version and a service deploy, because their routes are new on the service. They
1414
+ are where `bir` changes a scenario by hand, so the rules are strict:
1415
+
1416
+ - **Owner only.** Anyone else, an admin included, gets `404`, never `403`.
1417
+ - **A change is saved only if it reproduces the recording and is not a copy**, or on purpose with
1418
+ `--force --note "<why>"`. `check` and `edit` run the same check; `check` never writes. When the
1419
+ check had any problem, a forced save is recorded as forced, and `bir investigate` keeps saying so.
1420
+ - `<id>` is a `run_` id, resolved to its whole-run scenario with `GET /recordings/runs/:id/scenario`,
1421
+ or a `scn_` id, used as it is. A sub-task scenario has only its `scn_` id; for a run with only
1422
+ sub-task scenarios, `bir` lists their `scn_` ids. Logic goes in with `--input-logic` /
1423
+ `--output-logic` from a file, or `-` for stdin — never inline, because shell quoting mangles
1424
+ JavaScript. `edit` also takes `--freeze`, `--unfreeze`, `--note` and `--revision <n>` (the
1425
+ `expectedRevision` of the body).
1426
+ - `show --step <n>` prints the step object with `scenarioId` and `chainRevision` first. The revision
1427
+ is the scenario's, and is what `--revision` takes.
1428
+ - **`undo` walks back.** Without `--edit` it takes back the newest edit (never an undo entry) that
1429
+ can be undone; with `--step <n>`, the newest edit of step `n`. Run again, it goes one further
1430
+ back. `--edit` and `--step` cannot be given together. To redo, undo the undo entry by its
1431
+ `sedit_` id; the undo prints that command. The service checks an undo like a save, for the later
1432
+ steps and the final answer (so only when it changes output logic), and answers
1433
+ `422 undo_refused` when it would break them, unless `--force --note "<why>"` is given. Every
1434
+ save and every undo bumps the scenario's `chainRevision`.
1435
+ - A save lifts a mark made by hand (`--freeze`, `by_hand`) only with `--unfreeze`; undoing the
1436
+ freeze, or `calc --force --discard-edits`, also removes it. A forced save lifts only a mark the
1437
+ calculation set.
1438
+ - `calc --force` answers `409 scenario_has_edits` on a plan with hand edits, unless
1439
+ `--discard-edits` is given. `calc <scnId>` without `--force` is a usage error: nothing is sent.
1440
+ - Answers `bir` puts in words: `400 nothing_to_change` (the code and mark asked for are the step's
1441
+ already), `413 payload_too_large`, `503 logic_sandbox_unavailable` (nothing was checked, saved or
1442
+ undone), and a check report with `noChange: true` (`check` then says nothing would change, and
1443
+ exits `0`).
1444
+ - Exit codes: `0` checked OK, saved or undone; `1` refused, not found, or a service error; `2`
1445
+ usage. `--json` prints the service's body unchanged.
1446
+
1447
+ The same commands are tools on the `bir` MCP server, which by default lists only `run_scenario`:
1448
+ every tool definition rides along with every request of every session. `scenario_show` (`target`,
1449
+ `step`), `scenario_edits` (`target`) and `investigate` (`id`) only read; they are listed in a project
1450
+ where `bir scenario editing on` was run, or with `BIR_SCENARIO_TOOLS=read` in the `bir` server's
1451
+ environment, and answer a host that calls them unlisted all the same. `scenario_check` (`target`,
1452
+ `step`, `inputLogic`, `outputLogic`, `unfreeze`), `scenario_edit` (the same, plus `freeze`, `force`,
1453
+ `note`, `revision`) and `scenario_undo` (`target`, `edit` or `step`, `force`, `note`) are offered
1454
+ only in a project where `bir scenario editing on` was run, because that server runs in every session
1455
+ of every installed project. Each tool runs this package's own `bir … --json`, with logic bodies
1456
+ passed through temporary files; a refusal comes back as an MCP error result carrying the report. A
1457
+ reading tool's answer is capped at 24 000 characters (`BIR_SCENARIO_MAX_OUTPUT_CHARS`) and gives up
1458
+ after `BIR_READ_TIMEOUT_MS` (15 s) plus 10 s; a changing tool gets 120 s. A call the host cancels, or
1459
+ leaves behind when it exits, has its `bir` stopped and its files removed. While the runner is off,
1460
+ the server lists nothing; a reading tool answers that there is nothing to read, and a changing tool
1461
+ is refused.
1462
+
1463
+ **`bir`'s own calls are never recorded** (the service's `editSteps.md` D13; not yet released). The
1464
+ `bir` server's tools other than `run_scenario`, and a `Bash` or `PowerShell` command made only of
1465
+ `bir` invocations (and a plain `cd`), are host housekeeping, like `ToolSearch`
1466
+ ([housekeeping.ts](../src/record/housekeeping.ts)). They are never recorded as steps, and never
1467
+ count as divergence from a plan. A session that fixes a plan must not become a plan that edits
1468
+ plans when it is replayed. The operator's walk-through is
1469
+ [calculatedReplayGuide.md](calculatedReplayGuide.md) §9.2.
1470
+
1471
+ `--dry` maps to BaseIn's `POST /scenarios/:id/replay`, which evaluates the stored logic against the
1472
+ source run's *recorded* outputs — no real tools, no side effects, one Haiku call. Without `--dry`,
1473
+ `bir replay` runs the same plan through the same executor against the live proxies. `bir doctor`
1474
+ gains a `replay` block: on/off, the effective `BIR_REPLAY_ALLOW_SERVERS` allowlist,
1475
+ `BIR_MIN_STEER_SIMILARITY`, and who derives — this machine, the service, or nobody, which is the
1476
+ state in which a scenario with a target never runs (§13.2, mitigation 3).
1477
+
1478
+ ---
1479
+
1480
+ ## 17. Implementation phases
1481
+
1482
+ Each phase ends in something runnable. Do not start a phase before its predecessor's acceptance test
1483
+ passes.
1484
+
1485
+ ### Phase R0 — Plan, logic, coverage (no wiring)
1486
+ Port `logic.ts` from BaseIn, `bundle.ts` and `ScenarioReplayPlan` from RRepeat, write `coverage.ts`.
1487
+ Pure, dependency-free, no I/O.
1488
+ **Acceptance:** unit tests replay a fixture `SerializedScenario` end to end against a stub executor
1489
+ and reproduce the recorded outputs byte for byte.
1490
+
1491
+ ### Phase R1 — Derivation and pricing
1492
+ `derive.ts` + `pricing.ts`. Both fully testable against a fake `fetch`.
1493
+ **Acceptance:** with no key, derivation returns sample values at zero cost; with a stubbed key, it
1494
+ parses fenced JSON, bare JSON and prose-wrapped JSON, and prices a known `usage` to the same cent as
1495
+ BaseIn's `cost.ts`.
1496
+
1497
+ ### Phase R2 — Arming and the decision log
1498
+ `onPrompt` awaits `getMatch()` under budget, runs the gate ladder, computes the mode, arms the plan,
1499
+ returns the directive. **No execution yet.**
1500
+ **Acceptance:** a fake recorder returns a match; assert exactly one `plan.armed` or one
1501
+ `replay.decision` line per prompt, the right mode, and that the prompt hook returns inside the
1502
+ budget when `getMatch` never resolves.
1503
+
1504
+ ### Phase R3 — Steer mode
1505
+ `/tool/pre` pinning (with the §7.1 `updatedInput` merge), `/tool/post` threading (with the §7.2
1506
+ source split), retire on done.
1507
+ **Acceptance:** a scripted hook sequence drives a 3-step plan to `steered_full`; a wrapped-MCP step
1508
+ threads from `/proxy/step` and not from the hook; reordering the model's calls produces exactly one
1509
+ divergence and no crash.
1510
+
1511
+ ### Phase R4 — The work queue and direct execution
1512
+ `ProxyWorkQueue`, `/proxy/poll`, `/proxy/result`, the proxy-side loop.
1513
+ **Acceptance:** against `test/fake-upstream.ts`, a 3-step direct plan executes with `respParams`
1514
+ threaded between steps; killing the proxy mid-plan degrades to `recordedOutput`/skip and the session
1515
+ survives. **[changed]** The plan now stops at that step and hands over (§8); a recorded output stands
1516
+ in only with `BIR_REPLAY_RECORDED_OUTPUTS=1`.
1517
+
1518
+ ### Phase R5 — `bir-scenario` and direct delivery
1519
+ The MCP server, `/scenario/run`, `bir install --replay`.
1520
+ **Acceptance:** a real Claude Code session on a fully-wrapped scenario answers from one
1521
+ `run_scenario` call, and the transcript shows a genuine `tool_result`.
1522
+
1523
+ ### Phase R6 — Divergence
1524
+ Direct-execute the remainder; the lazy source-run fetch for `live` steps (§8.1); Bash-clean,
1525
+ `bir`-tool and deny delivery; the fair-share bundle cap.
1526
+ **Acceptance:** forcing an off-script call produces a bundle under `MAX_REPLAY_REASON` containing
1527
+ every remaining step; a `live` step is served its recorded output, marked `(recorded)`, and threads
1528
+ `respParams` for the steps after it; a `live` step with no recorded output is skipped with a warning
1529
+ and later steps still resolve. **[changed]** Recorded outputs stand in only with
1530
+ `BIR_REPLAY_RECORDED_OUTPUTS=1`, and a step with no result ends the bundle there: later steps do
1531
+ not run on nothing, and the bundle opens with a hand-over note.
1532
+
1533
+ ### Phase R7 — Reporting
1534
+ The usage watermark, `reportExecution`, ticket redemption in `sealRun`, `bir doctor` fields.
1535
+ **Acceptance:** three prompts in one session produce three *non-cumulative* `sessionCostUsd` values;
1536
+ a declined match books a baseline sample; a doubled report books once.
1537
+
1538
+ ---
1539
+
1540
+ ## 18. Testing
1541
+
1542
+ Extends v1 §11; same `node:test` harness, same `test/fake-upstream.ts`.
1543
+
1544
+ 1. **Plan semantics** — `respParams` threading, `paramsLogic` applied once, `isDone` after the last
1545
+ step, throw-propagation for each of the four logic hooks.
1546
+ 2. **Coverage** — every combination of wrapped/unwrapped MCP and built-in maps to the right mode,
1547
+ including the empty-steps and all-live cases.
1548
+ 3. **The `updatedInput` merge (§7.1)** — a pinned wrapped-MCP call carries both the pinned arguments
1549
+ and `__bir_call_id__`; the correlation's `hookArgs` are the *pinned* ones; `--no-correlation`
1550
+ still pins.
1551
+ 4. **The threading source (§7.2)** — a wrapped step threads the proxy's `result`, not the hook's
1552
+ `tool_response`, and threads the hook's view only after the grace window, with the warning.
1553
+ 5. **Budgets** — a `getMatch` that never resolves still returns the prompt hook inside its budget;
1554
+ a derivation that never resolves still pins a scenario of settings, using sample values, and
1555
+ hands over one that needs a target it was not given; the direct-run worst case stays under the
1556
+ 270 s ceiling.
1557
+ 6. **Direct execution** — end to end through `fake-upstream`, including a step timeout, a dead proxy
1558
+ and a `tools/call` that returns `isError: true` (a *result*, not a transport failure: it is read,
1559
+ and the plan stops and hands over rather than threading on it).
1560
+ 7. **Divergence** — both delivery channels; delimiter neutralisation in the Bash path, and the
1561
+ file route past 6 000 characters; the fair-share cap with one enormous response and four small
1562
+ ones; the bundle ending at a step with no result.
1563
+ 8. **Reporting** — the watermark across three prompts; each outcome's server-side effect against a
1564
+ fake BaseIn; ticket idempotence.
1565
+ 9. **Integration** — a real Claude Code session against `fake-upstream` with a hand-written scenario,
1566
+ asserting a `steered_full` report. The only test that catches host-behaviour drift.
1567
+
1568
+ ---
1569
+
1570
+ ## 19. Rejected alternatives
1571
+
1572
+ **19.1 Deliver direct-mode results as `additionalContext`.** No extra MCP server, no extra tool in the
1573
+ list. Rejected: it must execute inside `UserPromptSubmit`'s timeout (§10), and it asserts to the model
1574
+ that tools ran without a transcript record that they did. §6.2.
1575
+
1576
+ **19.2 A callback URL per proxy instead of long-polling.** Faster dispatch, no idle connection.
1577
+ Rejected: N new loopback listeners, N new tokens, N new things `bir doctor` must verify — for a
1578
+ latency saving of milliseconds on a path already dominated by the upstream's own round trip. Dialling
1579
+ out is also the posture v1 §9 already chose everywhere else. D12.
1580
+
1581
+ **19.3 Give `bir-scenario` its own upstream connections.** Then no work queue at all. Rejected
1582
+ outright: it double-spawns every upstream. Two `chrome-devtools-mcp` processes means two browsers,
1583
+ two profiles, and a scenario that navigates one and screenshots the other.
1584
+
1585
+ **19.4 Port RRepeat's Agent-SDK divergence fallback.** Rejected: it is the dominant cost of RRepeat's
1586
+ calculated run (`savingCalcRunner.md` §4), it adds `@anthropic-ai/claude-agent-sdk` to a
1587
+ zero-dependency package, it needs anti-recursion guards, and here it buys nothing the proxies do not
1588
+ already buy for free. §8.
1589
+
1590
+ **19.5 Block the prompt and answer from the scenario alone.** The original RRepeat design, superseded
1591
+ there (`replay.md` §1, §10). Rejected for the same reasons plus one of ours: it forfeits the live
1592
+ session's built-in tools entirely, which is exactly the half of a mixed scenario the proxy cannot
1593
+ reach.
1594
+
1595
+ **19.6 Match on the whole conversation rather than the prompt.** Arguably better "similar meaning".
1596
+ Rejected as a *server* concern: `findSimilarRun` embeds `run.input`, and changing what is embedded
1597
+ changes the corpus, the threshold and every stored vector. §20 question 1.
1598
+
1599
+ ---
1600
+
1601
+ ## 20. Open questions
1602
+
1603
+ 1. **Should the match embed more than the first prompt?** A turn's meaning often lives in the
1604
+ preceding turn. Changing it is a BaseIn-side decision with a corpus migration attached, and it
1605
+ would let `bir` match mid-conversation rather than only at a turn boundary.
1606
+ 2. **Does BaseIn want a `direct` outcome?** Today direct mode reports `steered_full`, which is
1607
+ accurate but hides that the run cost no session tokens at all. A sixth enum member would let the
1608
+ ledger separate "the model was steered cheaply" from "the model did not run the task."
1609
+ 3. **A separate `node:vm` context for the logic bodies?** §13.1 states plainly that there is no
1610
+ sandbox. The bodies already run through `node:vm`, but in this process's own context, which buys
1611
+ a time limit and nothing more. A frozen, minimal context would close the gap between
1612
+ "first-party by provenance" and "first-party by construction", at the price of the shared realm
1613
+ and argument objects the bodies rely on today.
1614
+ 4. **Should a declined match still arm a *dry* plan?** Deriving parameters and computing the first
1615
+ step's input without executing anything would make `bir doctor` able to say *"this scenario would
1616
+ have run these three calls"* — excellent for tuning `BIR_MIN_STEER_SIMILARITY`, at the price of a
1617
+ Haiku call on turns that replay nothing.
1618
+ 5. **Should `scenario_steps` carry `recorded_output`?** §8.1 works around its absence with one lazy
1619
+ `GET /recordings/runs/:runId`. A column would make it a field read, would let a scenario outlive
1620
+ the deletion of its source run, and would finally make RRepeat's `recordedOutput` field mean
1621
+ something. Cost: the scenario row grows by the size of every step's output.
1622
+ 6. **Cwd fit.** A scenario recorded in project A may pin paths that do not exist in project B, and
1623
+ nothing currently checks. A cheap sixth gate — compare `run.metadata.cwd` — would catch the common
1624
+ case; the general case (relative paths, monorepos) is harder and probably belongs in
1625
+ `toolInputLogic` generation server-side.