@basein/runner 0.3.8 → 0.3.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +46 -5
- package/dist/auth/client.d.ts +15 -0
- package/dist/auth/client.js +34 -12
- package/dist/bin/bir-hooks.d.ts +1 -1
- package/dist/bin/bir-hooks.js +1 -1
- package/dist/bin/bir.d.ts +2 -0
- package/dist/bin/bir.js +60 -6
- package/dist/bin/investigate.d.ts +5 -0
- package/dist/bin/investigate.js +117 -15
- package/dist/bin/setup.d.ts +12 -0
- package/dist/bin/setup.js +185 -56
- package/dist/bin/update.d.ts +81 -0
- package/dist/bin/update.js +194 -0
- package/dist/config/adapters/claude-code.d.ts +17 -0
- package/dist/config/adapters/claude-code.js +28 -1
- package/dist/config/generate.d.ts +14 -0
- package/dist/control/daemon.d.ts +26 -1
- package/dist/control/daemon.js +53 -3
- package/dist/control/pin-guard.d.ts +6 -1
- package/dist/control/pin-guard.js +15 -1
- package/dist/control/recorder-keeper.js +2 -1
- package/dist/control/server.d.ts +200 -2
- package/dist/control/server.js +741 -48
- package/dist/control/transcript.d.ts +4 -0
- package/dist/control/transcript.js +4 -0
- package/dist/proxy/session.js +1 -0
- package/dist/record/housekeeping.d.ts +7 -1
- package/dist/record/housekeeping.js +11 -3
- package/dist/record/recorder.d.ts +34 -0
- package/dist/record/remote-recorder.d.ts +14 -0
- package/dist/record/remote-recorder.js +107 -13
- package/dist/record/switchable-recorder.js +1 -0
- package/dist/record/truncate.d.ts +11 -0
- package/dist/record/truncate.js +11 -0
- package/dist/replay/controller.d.ts +146 -4
- package/dist/replay/controller.js +340 -31
- package/dist/replay/derive.js +2 -0
- package/dist/replay/handover.d.ts +13 -1
- package/dist/replay/handover.js +19 -0
- package/dist/replay/plan.d.ts +6 -0
- package/dist/replay/plan.js +13 -3
- package/dist/replay/safety.d.ts +15 -4
- package/dist/replay/safety.js +19 -6
- package/dist/replay/source-run.js +2 -1
- package/dist/replay/types.d.ts +16 -0
- package/dist/util/service-headers.d.ts +24 -0
- package/dist/util/service-headers.js +47 -0
- package/docs/BaseInstRunner.md +115 -1
- package/docs/calculatedReplay.md +239 -27
- package/docs/calculatedReplayGuide.md +134 -12
- package/docs/installRun.md +91 -10
- package/docs/quickstart.md +64 -9
- package/package.json +1 -1
- package/production.md +81 -16
- package/scripts/install.ps1 +25 -1
package/docs/calculatedReplay.md
CHANGED
|
@@ -45,7 +45,7 @@ this design has a ladder whose cheapest rung is free — and the more of a stack
|
|
|
45
45
|
|---|---|---|
|
|
46
46
|
| Runs MCP tools itself | No (`replay.md` §12: *"rrepeat never executes tools — not even MCP"*) | **Yes**, on the connection it already owns |
|
|
47
47
|
| Divergence fallback | A second Agent SDK session — the dominant cost | Direct `tools/call` through the proxies — **$0** |
|
|
48
|
-
| Reaches built-ins (`Bash`, `Read`) | Only by steering the live session | `Read`, `Glob`, `Grep`, `Edit`, `Write` and `Bash` run on the runner itself, jailed to the project (`BIR_BUILTIN_TOOLS`, `BIR_BASH_ALLOW`); any other built-in is steered |
|
|
48
|
+
| Reaches built-ins (`Bash`, `Read`) | Only by steering the live session | `Read`, `Glob`, `Grep`, `Edit`, `Write` and `Bash` run on the runner itself, jailed to the project (`BIR_BUILTIN_TOOLS`, `BIR_BASH_ALLOW`); any other built-in is steered. In a host that is not Claude Code a tool with one of these names is the application's own, and the runner runs none of them (§7.4) |
|
|
49
49
|
| Needs `@anthropic-ai/*` at runtime | Yes | No (§16.3) |
|
|
50
50
|
|
|
51
51
|
---
|
|
@@ -131,6 +131,10 @@ Server-side, and worth stating because the threshold in §4 is a judgement about
|
|
|
131
131
|
(`recordings/embeddings.ts`).
|
|
132
132
|
- Candidates are **the caller's own canonical runs only** (`listUserRunEmbeddings(userId)`);
|
|
133
133
|
duplicates and sub-threshold runs are excluded from the pool at record time.
|
|
134
|
+
- **Inside one host** — on the service as built, which is not deployed yet. A run created with
|
|
135
|
+
`framework` `langchain` or `langchain:<scope>` (§7.4) is matched only against recordings and
|
|
136
|
+
plans of that same text. Everything else (`baseinstrunner`, `custom`, the built-ins) is Claude
|
|
137
|
+
Code's pool, as before.
|
|
134
138
|
- The match is the single best cosine ≥ `SIMILARITY_THRESHOLD` (default **0.9**).
|
|
135
139
|
- A matched run only carries a `scenarioId` when its scenario exists **and** is `state: 'ready'`.
|
|
136
140
|
|
|
@@ -191,7 +195,7 @@ Every gate that declines logs **one** `replay.decision` line carrying the reason
|
|
|
191
195
|
| # | Gate | Declines when | Outcome floor |
|
|
192
196
|
|---|---|---|---|
|
|
193
197
|
| 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched) |
|
|
194
|
-
| 2 | **Replay enabled** | `BIR_REPLAY === "0"`, or, with it unset, the project's stored switch is off (`bir replay off`, or a `bir install` without `--replay`) | `not_steered` |
|
|
198
|
+
| 2 | **Replay enabled** | `BIR_REPLAY === "0"`, or, with it unset, the project's stored switch is off (`bir replay off`, or a `bir install` without `--replay`). In a session of another host, also when the run's adapter asked for no plan with its prompt (`host_records_only`, §7.4) | `not_steered` |
|
|
195
199
|
| 3 | **Scenario ready** | `scenario == null`, `state !== "ready"`, or `steps.length === 0` | `not_steered` |
|
|
196
200
|
| 4 | ~~**Similarity**~~ | Retired: a match is the decision to use the scenario (below) | — |
|
|
197
201
|
| 5 | **Tool coverage** | No step is executable anywhere (§5 → `mode: "none"`) | `not_steered` |
|
|
@@ -204,6 +208,9 @@ inside the prompt's budget (§10) is dropped (`match_timeout`, `arm_timeout`), a
|
|
|
204
208
|
after the prompt was answered is retired unannounced (`match_late`, `arm_unannounced`). A `direct`
|
|
205
209
|
plan is declined when no running `bir-scenario` has said it serves `run_scenario`
|
|
206
210
|
(`no_delivery_tool`, §6.3). In plan mode (`permission_mode: "plan"`) nothing is announced at all.
|
|
211
|
+
In a session of another host (§7.4) a plan that armed is taken back, before any step of it runs,
|
|
212
|
+
when it names a tool the agent that asked does not hold (`host_tool_missing`): it is booked like
|
|
213
|
+
any gate's decline, `not_steered`, with no step marked failed.
|
|
207
214
|
|
|
208
215
|
Gate 4 is retired. Up to 0.3.6 it declined a match under `BIR_MIN_STEER_SIMILARITY` (0.92),
|
|
209
216
|
above the server's 0.9, on the argument that the server only decided *"do not record this again"*
|
|
@@ -277,6 +284,12 @@ A **mixed** scenario in `steer` mode still gets direct execution for its wrapped
|
|
|
277
284
|
diverges (§8), so coverage is not all-or-nothing at the divergence boundary — only at the arm-time
|
|
278
285
|
mode choice.
|
|
279
286
|
|
|
287
|
+
**None of this applies in a session of another host.** When a session's requests name a host that
|
|
288
|
+
is not Claude Code (§7.4), a tool's name carries no Claude Code meaning: `Bash`, `Read` or an
|
|
289
|
+
`mcp__` prefix name the application's own tools, no proxy of this folder sees their calls, and the
|
|
290
|
+
runner runs none of them, whatever `BIR_BUILTIN_TOOLS` lists. Every step is `live`, so the mode is
|
|
291
|
+
always `steer`.
|
|
292
|
+
|
|
280
293
|
---
|
|
281
294
|
|
|
282
295
|
## 6. Mode A — direct execution
|
|
@@ -423,7 +436,9 @@ another way. Divergence (§8) is for steer plans only.
|
|
|
423
436
|
The mixed rung. Ported from RRepeat's `replay.md` §3/§12, which is the right design and is already
|
|
424
437
|
proven: the model chooses *which* tool and *when*; the control server overrides the *arguments* and
|
|
425
438
|
threads state between steps. The session never leaves the model's hands, so every
|
|
426
|
-
`tool_use`/`tool_result` in the transcript is genuine and continuation is automatic.
|
|
439
|
+
`tool_use`/`tool_result` in the transcript is genuine and continuation is automatic. That is Claude
|
|
440
|
+
Code. In a host that is not Claude Code the same mode is stepped through by the host's adapter, in
|
|
441
|
+
the model's place, and a call the model makes itself is never pinned (§7.4).
|
|
427
442
|
|
|
428
443
|
**Arm** at `UserPromptSubmit` — plan on `run.plan`, `run.planToolUse = new Set()`, directive via
|
|
429
444
|
`additionalContext`, **no `decision`** so the prompt still reaches the model:
|
|
@@ -589,15 +604,91 @@ In [`HOOK_ROUTES`](../src/config/adapters/claude-code.ts) neither `PreToolUse` n
|
|
|
589
604
|
carries `async: true`, so this already holds — **and must never be relaxed**. Add it to the comment
|
|
590
605
|
block there, next to the `SessionEnd` note that records the same class of lesson.
|
|
591
606
|
|
|
607
|
+
### 7.4 A host that is not Claude Code
|
|
608
|
+
|
|
609
|
+
Everything above assumes Claude Code: its hooks report the turn, and its model makes the plan's
|
|
610
|
+
calls. The recorder's local protocol also serves a host that has neither. Today that is a
|
|
611
|
+
LangChain application: a Python program whose agent is built with LangChain's `create_agent` and
|
|
612
|
+
the package `langchain-basic2202`, a LangChain middleware that talks to the recorder over that
|
|
613
|
+
protocol. The package is *the adapter* below, and it is not on PyPI yet. Claude Code's hooks are
|
|
614
|
+
unchanged, and every new field is optional. A session whose requests carry
|
|
615
|
+
`host: { app, version?, adapter?, scope? }` with an `app` other than `claude-code` is a session of
|
|
616
|
+
another host ([`control/server.ts`](../src/control/server.ts)). The fields named below are what
|
|
617
|
+
such an adapter adds to the hook routes of the control plane
|
|
618
|
+
([BaseInstRunner.md](BaseInstRunner.md), Phase 5). `hostFields` in `/health` tells an adapter how
|
|
619
|
+
much of this a recorder understands: with 1 it records, and from 2 on it may also ask for a plan
|
|
620
|
+
and step through it.
|
|
621
|
+
|
|
622
|
+
For such a session:
|
|
623
|
+
|
|
624
|
+
- **The run names its host.** It is created with `framework` = the app, or `app:scope`
|
|
625
|
+
(`langchain`, `langchain:billing`), and `metadata.host`. A Claude Code run is created as before
|
|
626
|
+
(`framework: "baseinstrunner"`).
|
|
627
|
+
- **Only `/session/prompt` opens a run.** Any other hook that finds no open run opens none and
|
|
628
|
+
records nothing: it is answered `{}`, or `{ noPlan: true }` when it asked for a step.
|
|
629
|
+
- **A plan starts only for a prompt that said `replay: true`.** Otherwise the match is declined at
|
|
630
|
+
gate 2 with `host_records_only` (§4). The prompt's answer carries the armed plan as data too
|
|
631
|
+
(`plan: { scenarioId, mode, steps: [{ stepIndex, toolName }], stopAt? }`): the directive beside
|
|
632
|
+
it is a text written for a model.
|
|
633
|
+
- **The mode is always `steer`, and the runner runs no step itself.** The host runs its own tools.
|
|
634
|
+
Tool names carry no Claude Code meaning: `Bash`, `Read`, `TodoWrite`, `ToolSearch`… are the
|
|
635
|
+
application's own tools, and nothing is skipped as housekeeping except the `bir` server's own
|
|
636
|
+
tools. So there is no direct mode there, and no divergence recovery (§8): nothing can be executed
|
|
637
|
+
here in the host's place.
|
|
638
|
+
- **The adapter asks for each step**, with a `/tool/pre` that says `scripted: true`, and the host
|
|
639
|
+
runs its own tool. The answer is the step's arguments (`hookSpecificOutput.updatedInput`), or
|
|
640
|
+
`{ noPlan: true }` when the recorder holds no live plan, or `{ declined: "host_tool_missing" }`
|
|
641
|
+
when the plan names a tool that is not in `tools` (the agent's tools, sent with the first such
|
|
642
|
+
request of a run), or a refusal whose reason is the hand-over note when the plan stopped in that
|
|
643
|
+
very request. The step's output is threaded from the `/tool/post` that follows: every step is
|
|
644
|
+
`live` (§7.2).
|
|
645
|
+
- **The model's own call is never pinned.** A `/tool/pre` without `scripted` is the model's own
|
|
646
|
+
call, and it is never given the plan's arguments. When a plan is live it is handed over (§8): the
|
|
647
|
+
call runs as the model wrote it, and the hand-over note comes with that call's `/tool/post`.
|
|
648
|
+
- **The host may stop a step itself**, and says so with the step's `/tool/post`.
|
|
649
|
+
`host_stop: "rejected"` means a person said no, or answered in the tool's place, or a call limit
|
|
650
|
+
blocked the call; `"edited"` means a person changed the call. Either way the step's verdict is
|
|
651
|
+
`skipped`, not `failed`, nothing is counted against the plan, and the plan hands over (§8). §11.1
|
|
652
|
+
has what the report says.
|
|
653
|
+
- **There is no transcript.** What Claude Code's transcript gives comes with the requests instead:
|
|
654
|
+
with a `/tool/pre`, what the model wrote and reasoned before the call (`context`, `thinking`);
|
|
655
|
+
with `/session/stop`, the answer and the tokens (`final_answer`, `usage`). The cost comes from
|
|
656
|
+
that `usage` (§11.4). The conversation's earlier tool results do not come with the prompt: a new
|
|
657
|
+
prompt's values are read from the prompt alone, here as in Claude Code (§16.3), and a
|
|
658
|
+
`recent_results` an adapter still sends is not read.
|
|
659
|
+
- **A run can end without an ordinary stop.** With `/session/stop`, `failed` closes the run with
|
|
660
|
+
that text as its `error` (cleaned of secrets, cut) and no final answer, and `incomplete` marks
|
|
661
|
+
the finish `incomplete: { lostSteps, lossy }`. A run sealed after two hours of silence is closed
|
|
662
|
+
with `error: "turn ended early: idle"`. A run that is closed while its plan is still live has the
|
|
663
|
+
plan handed over first, so its report says what happened (§11.1). On the service as built (not
|
|
664
|
+
deployed yet), a run with an `error` or with that mark is never picked for automatic calculation
|
|
665
|
+
and no longer catches its prompt, unless its owner preferred it or it already has a plan. Only a
|
|
666
|
+
run of another host sends either today: a Claude Code run that was interrupted or lost steps is
|
|
667
|
+
finished as before.
|
|
668
|
+
- **A value that a plan reads from the prompt is checked against the host's own sinks** before it
|
|
669
|
+
goes into a tool: the `sinks` the host declared with its prompt (§13.2, mitigation 6).
|
|
670
|
+
- **The switch is the same.** `bir setup --host langchain` stores replay on for the folder
|
|
671
|
+
(`--no-replay` stores it off), and `bir replay on` and `bir replay off` work there as anywhere.
|
|
672
|
+
- **The audit lines name the host**: `run.start … host=langchain`,
|
|
673
|
+
`plan.armed … mode=steer host=langchain`, and `run.finish … ended=<reason>` when the run did not
|
|
674
|
+
end with an ordinary stop. `run.incomplete` and `run.host_failed` are what the host said with its
|
|
675
|
+
stop.
|
|
676
|
+
|
|
677
|
+
The how-to for a LangChain developer — setting up the folder, the two lines of Python, what to do
|
|
678
|
+
when something is wrong — is `langChainGuide.md` in the root of the runner's repository (it is not
|
|
679
|
+
in the npm package), and is not repeated here. The design guide with every decision is
|
|
680
|
+
`integratioLangChane.md` in the service's repository.
|
|
681
|
+
|
|
592
682
|
---
|
|
593
683
|
|
|
594
684
|
## 8. Divergence
|
|
595
685
|
|
|
596
686
|
In a steer plan, the model called a tool the plan did not expect, or the call had no `tool_use_id`.
|
|
597
|
-
(A direct plan never diverges, §6.4.
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
687
|
+
(A direct plan never diverges, §6.4. Nor does a plan in a host that is not Claude Code, §7.4: only
|
|
688
|
+
the host can run its tools, so the plan is handed over and nothing is executed in its place. A
|
|
689
|
+
`toolInputLogic` that throws is not a divergence either: re-executing would start at that same step
|
|
690
|
+
and throw again, so the step is recorded as failed and the task is handed to the model, with a note
|
|
691
|
+
in place of the call.) Three tiers, tried in order. Here is where the §0 asymmetry pays.
|
|
601
692
|
|
|
602
693
|
**Tier 1 — direct-execute the remainder (new; the common case).**
|
|
603
694
|
Take every not-yet-run step, execute it, and hand the composed result back. This happens inside the
|
|
@@ -636,6 +727,12 @@ about it:
|
|
|
636
727
|
- beside a pin still in flight: *"… was not run by the scenario: another of its calls was still in
|
|
637
728
|
flight, and a scenario runs one call at a time."*, and the pin itself *"was started and has not
|
|
638
729
|
reported back"*.
|
|
730
|
+
- stopped by its host, in a session of another host (§7.4): *"… was not run by the host. The result
|
|
731
|
+
above was written by the host, not by the tool. Do not run the step again unless the user asks."*
|
|
732
|
+
(`host_rejected`), or *"… ran with arguments a person chose, not the scenario's. Its result above
|
|
733
|
+
is real and its side effects are in place. Do not run it again."* (`host_edited`). When no step
|
|
734
|
+
ran before it, the note says *"No step of the scenario ran as it was planned."* instead of *"No
|
|
735
|
+
step … ran"*. The report's `fallbackKind` stays `step_failed`.
|
|
639
736
|
|
|
640
737
|
It ends with the remaining planned steps and the parameters derived for this request, within 4 000
|
|
641
738
|
characters.
|
|
@@ -781,8 +878,8 @@ The hook is on the user's critical path. Every await here is a stall the user wa
|
|
|
781
878
|
|
|
782
879
|
| Moment | Route | Hook timeout | Answered by | What may be awaited | Budget |
|
|
783
880
|
|---|---|---|---|---|---|
|
|
784
|
-
| `UserPromptSubmit` | `/session/prompt` | 10 s | 7 s | the match; for a scenario with a target, its derivation; a steer plan's argument preview | match `BIR_MATCH_BUDGET_MS` = **3000**; match and arming together ≤ **6 s**; derivation ≤ `BIR_HOOK_DERIVE_BUDGET_MS` = **
|
|
785
|
-
| `PreToolUse` | `/tool/pre` | 15 s | 12 s | `plan.ready()` (derivation); on a divergence, the remaining steps | derivation ≤ min(`BIR_DERIVE_BUDGET_MS`, `BIR_HOOK_DERIVE_BUDGET_MS`) = **
|
|
881
|
+
| `UserPromptSubmit` | `/session/prompt` | 10 s | 7 s | the match; for a scenario with a target, its derivation; a steer plan's argument preview | match `BIR_MATCH_BUDGET_MS` = **3000**; match and arming together ≤ **6 s**; derivation ≤ `BIR_HOOK_DERIVE_BUDGET_MS` = **8000**, cut short by the 6 s |
|
|
882
|
+
| `PreToolUse` | `/tool/pre` | 15 s | 12 s | `plan.ready()` (derivation); on a divergence, the remaining steps | derivation ≤ min(`BIR_DERIVE_BUDGET_MS`, `BIR_HOOK_DERIVE_BUDGET_MS`) = **8000**; all replay work ≤ `BIR_PRETOOL_REPLAY_BUDGET_MS` = **9000** (held at ≤ 10 000) |
|
|
786
883
|
| `run_scenario` | `/scenario/run` | `bir-scenario`'s own, **290 s** (`BIR_SCENARIO_RUN_TIMEOUT_MS`) | — | `ready()` + N steps | derivation `BIR_DERIVE_BUDGET_MS` = **8000**; plan `BIR_REPLAY_PLAN_BUDGET_MS` = **100000** (`BIR_REPLAY_BUDGET_MS` overrides it where set); per step `BIR_STEP_TIMEOUT_MS` = **60000**, clipped to what the plan has left |
|
|
787
884
|
| `PostToolUse` | `/tool/post` | 5 s | 2 s | nothing | — |
|
|
788
885
|
| `Stop` | `/session/stop` | 5 s | 2 s | nothing (report is queued) | — |
|
|
@@ -863,6 +960,25 @@ beside a pinned one hands over as `fell_back` or `failed` (§7). A direct run ca
|
|
|
863
960
|
(Esc, or `bir-scenario`'s own timeout) books `fell_back` or `failed` with `measured: false`, never
|
|
864
961
|
`diverged`: that turn's cost says nothing about what the task costs.
|
|
865
962
|
|
|
963
|
+
A host that is not Claude Code (§7.4) adds three endings, and none of them is counted against the
|
|
964
|
+
plan. The report says so with two fields: `measured`, and `baselineEligible: false`, which keeps
|
|
965
|
+
the turn's cost out of the baseline samples (absent, the turn is eligible).
|
|
966
|
+
|
|
967
|
+
- **A person's "no".** A step the host did not run (`host_stop: "rejected"`: a person said no, or
|
|
968
|
+
answered in the tool's place, or a call limit blocked the call) is recorded `skipped`, never
|
|
969
|
+
`failed`, and the plan hands over: `fell_back` when a step ran before it, `failed` when none did.
|
|
970
|
+
The report says `measured: false` and `baselineEligible: false`: the task ended early, so the
|
|
971
|
+
turn's cost is no measure of it.
|
|
972
|
+
- **A changed call.** A step that ran with arguments a person chose (`"edited"`) is `skipped` too,
|
|
973
|
+
and the plan hands over the same way. The cost stays measured, with `baselineEligible: false`.
|
|
974
|
+
- **A run closed with its plan live.** The plan is handed over first, so the report says
|
|
975
|
+
`fell_back` (a step ran) or `failed` (none ran), never `not_steered`, with `measured: false` and
|
|
976
|
+
`baselineEligible: false`. A run that ended early for any other reason (it failed, or went
|
|
977
|
+
silent) reports the same two fields.
|
|
978
|
+
|
|
979
|
+
A plan taken back before any step ran (`host_tool_missing`, §4) is a decline like any other:
|
|
980
|
+
`not_steered`.
|
|
981
|
+
|
|
866
982
|
Direct mode reports `steered_full`. BaseIn's zod enum
|
|
867
983
|
(mirrored in [`replay/types.ts`](../src/replay/types.ts)) has no `direct` member, and `steered_full`
|
|
868
984
|
means exactly what happened: every planned step executed under the scenario's control and the
|
|
@@ -887,6 +1003,11 @@ POST /scenarios/:scenarioId/executions
|
|
|
887
1003
|
An **unredeemed** ticket leaves no execution row at all. That asymmetry is intentional and worth
|
|
888
1004
|
preserving: silence loses a saving; it never invents one.
|
|
889
1005
|
|
|
1006
|
+
**How late a report may arrive.** The deployed service redeems a ticket for one hour after its plan
|
|
1007
|
+
was handed out, and answers a later report `409 invalid_ticket`. The service as built (not deployed
|
|
1008
|
+
yet) redeems it for 24 hours. It also answers a report whose write failed `503 report_not_booked`,
|
|
1009
|
+
and the recorder sends that report once more.
|
|
1010
|
+
|
|
890
1011
|
### 11.2a What each step did (the service's errorshandling.md)
|
|
891
1012
|
|
|
892
1013
|
The last five fields are the diagnosis, and none of the money depends on them. The service's
|
|
@@ -903,15 +1024,51 @@ steps: Array<{
|
|
|
903
1024
|
stage?: ExecutionStage; // where in the step, for `failed` / `skipped`
|
|
904
1025
|
error?: string; // message only — this is rendered in a web page
|
|
905
1026
|
durationMs?: number;
|
|
1027
|
+
toolInput?: string; // what the step sent its tool, as JSON text
|
|
1028
|
+
toolOutput?: string; // what its output logic was given
|
|
906
1029
|
}>
|
|
907
1030
|
```
|
|
908
1031
|
|
|
1032
|
+
**What the step sent and received.** `toolInput` is the arguments the step's input logic computed
|
|
1033
|
+
on this turn, and `toolOutput` what its output logic was given: the tool's answer, or for `recorded`
|
|
1034
|
+
the source run's output that stood in. They are this run's, and the service can learn them nowhere
|
|
1035
|
+
else — a matched turn is not recorded (§12). Without them the recording page can only put the
|
|
1036
|
+
recording's own sample under this run's verdict and time: a transfer of 3500 beside "passed", after
|
|
1037
|
+
a replay that sent 3600.
|
|
1038
|
+
|
|
1039
|
+
- **Redacted and capped.** Each is serialized and redacted as a recorded step's payload is
|
|
1040
|
+
(`serializeCapped(redact(…))`; an output is redacted again here, as JSON, because the hook's view
|
|
1041
|
+
of a steered built-in is threaded as the host gave it). Then it is cut to 3,800
|
|
1042
|
+
bytes (`REPORT_PAYLOAD_MAX_BYTES`), as text and with a marker that says how much was dropped: these
|
|
1043
|
+
are read on a page, never parsed, and with its marker a cut payload still fits the 4,000
|
|
1044
|
+
characters the service keeps of each — so the page shows this cut and its sizes, not a second one.
|
|
1045
|
+
The recorder redacts once more as it sends.
|
|
1046
|
+
- **Inside one small request.** The deployed service reads `POST /scenarios/:id/executions` with
|
|
1047
|
+
its 100 kB JSON parser (only `/recordings` has the 5 MB one) and answers a larger body 413, which
|
|
1048
|
+
would lose the turn's cost with it. So the recorder holds the whole report to 96 KiB
|
|
1049
|
+
(`REPORT_MAX_BYTES`): it writes the costs and the verdicts first, then adds payloads while they
|
|
1050
|
+
fit — the failed steps' first, then every step's input, then the outputs, in the order the steps
|
|
1051
|
+
ran. A payload that does not fit is left out, and its step keeps its verdict. The service as
|
|
1052
|
+
built (not deployed yet) accepts 1 MB on this route and on `/scenarios/:id/derive`, and cuts a
|
|
1053
|
+
report's `prompt` at 10,000 characters where the deployed one refuses the whole report with 400.
|
|
1054
|
+
The recorder keeps its report under 100 kB all the same: older services have that limit.
|
|
1055
|
+
- **Absent where there was none.** A step whose input logic threw has no input. A `skipped` step
|
|
1056
|
+
has no output, and neither has a call the host says failed without a result. The exception is a
|
|
1057
|
+
step its host stopped itself (§7.4): its verdict keeps what the host handed back, and after a
|
|
1058
|
+
changed call it has no input, because only the host knows the arguments that call ran with.
|
|
1059
|
+
- **Only for the reported scenario's own steps.** A step inside a called segment is reported with
|
|
1060
|
+
its verdict and its `scenarioId`, without payloads: the console still files verdicts by step
|
|
1061
|
+
number alone (the service's plan-services.md, W1.9), and a called segment numbers its steps from
|
|
1062
|
+
zero too.
|
|
1063
|
+
|
|
909
1064
|
`ok` and `recorded` are passes (`recorded` with an asterisk: the values came from the source run,
|
|
910
1065
|
which happens only with `BIR_REPLAY_RECORDED_OUTPUTS=1` and in `bir replay`). `skipped` produced no
|
|
911
1066
|
result here: its call could not be sent (a built-in outside the session, no proxy for its server,
|
|
912
|
-
only another open session's proxy), or it was sent and timed out or was refused.
|
|
913
|
-
a
|
|
914
|
-
|
|
1067
|
+
only another open session's proxy), or it was sent and timed out or was refused. A step that its
|
|
1068
|
+
host stopped itself, in a host that is not Claude Code, is `skipped` as well: a person refused it,
|
|
1069
|
+
or changed its call (§7.4). The plan stops at a `skipped` step.
|
|
1070
|
+
**`failed` is the only one that means the scenario is broken** rather than the environment — it is
|
|
1071
|
+
the scenario's own logic throwing, and it is what `Recalculate` fixes.
|
|
915
1072
|
|
|
916
1073
|
Three rules this must keep:
|
|
917
1074
|
|
|
@@ -926,8 +1083,10 @@ The verdicts come from `plan.ts`'s `onStep`, which has always known them — wit
|
|
|
926
1083
|
logic failure is announced *before* the throw propagates (`reportingStage`), because after it
|
|
927
1084
|
nothing knows which step the stack trace belonged to. In steer mode there is no loop to observe, so
|
|
928
1085
|
the controller records the verdict at both hooks: `PreToolUse` for an input-logic throw, `PostToolUse`
|
|
929
|
-
for the pass or the output-logic throw
|
|
930
|
-
|
|
1086
|
+
for the pass or the output-logic throw — and keeps the pinned arguments on the pin, so the verdict
|
|
1087
|
+
`PostToolUse` writes can say what the call ran with. `ReplayState.stepResults` is keyed by the
|
|
1088
|
+
scenario that owns the step and its index, so the second word on a step replaces the first and a
|
|
1089
|
+
called segment's step 0 never replaces the caller's.
|
|
931
1090
|
|
|
932
1091
|
One consequence for the rule that a costless decline is not worth sending (`buildReport`, and the
|
|
933
1092
|
service's matching `202 { recorded: false }`): it now *is* worth sending when a step broke. No cost
|
|
@@ -938,7 +1097,7 @@ service's matching `202 { recorded: false }`): it now *is* worth sending when a
|
|
|
938
1097
|
| Component | Source | Notes |
|
|
939
1098
|
|---|---|---|
|
|
940
1099
|
| `deriveCostUsd` | our own Haiku call's `usage`, priced locally | the only tokens replay itself spends |
|
|
941
|
-
| `sessionCostUsd` | transcript usage **delta** for this run (§11.4) | the live turn: prompt, tool calls, answer |
|
|
1100
|
+
| `sessionCostUsd` | transcript usage **delta** for this run (§11.4); in a host that is not Claude Code, the `usage` its adapter sends with `/session/stop` | the live turn: prompt, tool calls, answer |
|
|
942
1101
|
| `fallbackCostUsd` | **0.00** | direct execution spends no tokens (§8) |
|
|
943
1102
|
|
|
944
1103
|
> ### ⚠ Verify the pricing table before copying it
|
|
@@ -979,9 +1138,23 @@ rather than to a negative cost. A run with **no mark** (one opened lazily by a p
|
|
|
979
1138
|
prompt) reports `measured: false` — better to say "unmeasured" than to hand the ledger an inflated
|
|
980
1139
|
baseline.
|
|
981
1140
|
|
|
982
|
-
`measured` is `true` only when a mark existed **and**
|
|
983
|
-
|
|
984
|
-
|
|
1141
|
+
`measured` is `true` only when a mark existed, tokens were found since it, **and** every model of
|
|
1142
|
+
the turn has a price in the runner's table (`priceUsage` and `hasModelPricing`,
|
|
1143
|
+
[`control/server.ts`](../src/control/server.ts)). A model outside the table used to cost $0, and
|
|
1144
|
+
that zero was called measured: a saving nobody made. Now such a turn is not measured, in every
|
|
1145
|
+
host. A `run.unmeasured` line says why, with `why=`, when the run had no mark or when a model has
|
|
1146
|
+
no price (then with `models=`). A turn whose transcript held no tokens since the mark is not
|
|
1147
|
+
measured either, and writes no such line. No price table and no `PRICING_VERSION` changed with this.
|
|
1148
|
+
|
|
1149
|
+
The server judges a report again. It stores one whose `pricingVersion` differs from its own as
|
|
1150
|
+
unmeasured rather than trusting it. As built (not deployed yet) it does the same with a report that
|
|
1151
|
+
says `measured: true` with a session cost of $0, and a finish with a cost of 0 stores no cost.
|
|
1152
|
+
|
|
1153
|
+
**A host that is not Claude Code has no transcript, and no mark** (§7.4). Its cost comes from the
|
|
1154
|
+
`usage` its adapter sends with `/session/stop`, and is measured only when at least one entry has
|
|
1155
|
+
tokens and every model in the list has a price; with no usage at all, the line says
|
|
1156
|
+
`why="the host sent no usage"`. Such a host hands over one number for the whole run, so with more
|
|
1157
|
+
than one plan in one run none of them is measured.
|
|
985
1158
|
|
|
986
1159
|
---
|
|
987
1160
|
|
|
@@ -991,6 +1164,10 @@ trusting it.
|
|
|
991
1164
|
would 404. `run.recording = false` already handles it — `record()` early-returns and `sealRun` skips
|
|
992
1165
|
both the final answer and `finishRun`.
|
|
993
1166
|
|
|
1167
|
+
What the turn's steps sent and received is not lost with it: the execution report carries them, on
|
|
1168
|
+
each step's verdict (§11.2a). That is the only copy the service gets, and it is what the recording
|
|
1169
|
+
page shows as the scenario's last run.
|
|
1170
|
+
|
|
994
1171
|
Three consequences to keep straight:
|
|
995
1172
|
|
|
996
1173
|
1. **Step indices are still allocated.** `ordering` runs regardless, correlations are still minted
|
|
@@ -1014,10 +1191,11 @@ bir replay --scenario scn_… --prompt "…" --dry
|
|
|
1014
1191
|
|
|
1015
1192
|
Runs the same plan through the same executor, against the same live proxies, printing the per-step
|
|
1016
1193
|
input/output and the response model. It is how you test a scenario without a session, how a Tier 2
|
|
1017
|
-
or
|
|
1018
|
-
|
|
1019
|
-
|
|
1020
|
-
direct-run ceiling
|
|
1194
|
+
client — or one that is neither Claude Code nor a host whose adapter steps through plans (§7.4) —
|
|
1195
|
+
gets any replay at all, and the first thing to reach for when a steered turn behaves oddly. The
|
|
1196
|
+
operator named the scenario, so it keeps its own contract: a step it cannot send stands on its
|
|
1197
|
+
recorded output, the allowlist does not gate it, and it is not held to the 270 s direct-run ceiling
|
|
1198
|
+
(§10).
|
|
1021
1199
|
|
|
1022
1200
|
---
|
|
1023
1201
|
|
|
@@ -1055,6 +1233,15 @@ What actually constrains it:
|
|
|
1055
1233
|
and a `bir install` without it stores replay off in a project that never asked for it. A
|
|
1056
1234
|
compromised or misconfigured `BIR_AUTH_URL` therefore cannot start executing logic on a machine
|
|
1057
1235
|
whose operator never asked for replay.
|
|
1236
|
+
- **A small environment in an application's folder.** A recorder started for a folder that is set
|
|
1237
|
+
up for another host and not for Claude Code (`bir setup --host langchain`, `bir up`) gets a small
|
|
1238
|
+
environment, not the whole environment of the window that started it: `PATH`, the home and temp
|
|
1239
|
+
variables, what the system needs to start a process, the proxy and certificate variables, and
|
|
1240
|
+
every `BIR_*` variable (`smallEnv`, [`control/daemon.ts`](../src/control/daemon.ts)). Step code
|
|
1241
|
+
runs inside the recorder with no sandbox, so a model key or a database password of the
|
|
1242
|
+
application must not reach it. A folder that is set up for Claude Code too keeps the whole
|
|
1243
|
+
environment, because Claude Code's own steps may need it. Nothing changes for a Claude Code
|
|
1244
|
+
project.
|
|
1058
1245
|
|
|
1059
1246
|
If that is not enough for a deployment, the mitigation is a real one and is out of scope here:
|
|
1060
1247
|
evaluate the logic in a separate `node:vm` context with a frozen, minimal global. Filed as §20 question 3.
|
|
@@ -1117,6 +1304,9 @@ This is the sharpest edge in the whole design, and it must not be buried.
|
|
|
1117
1304
|
answered `allow` while the runner has announced a direct plan, so it brings no surprise prompt
|
|
1118
1305
|
(and no denial under `claude -p`); the user's own deny or ask rules for it still apply.
|
|
1119
1306
|
- **Plan mode** (`permission_mode: "plan"`) gets none of it: no directive, no pin, no `allow`.
|
|
1307
|
+
- **A host that is not Claude Code** (§7.4) is neither mode as written here: the runner runs no
|
|
1308
|
+
step. It hands the host's adapter each step's arguments, the host runs its own tool, and the
|
|
1309
|
+
adapter reports a step that a person refused or changed (`host_stop`).
|
|
1120
1310
|
|
|
1121
1311
|
So a scenario is, in effect, a **pre-approved list of tool calls with computed arguments**. That is
|
|
1122
1312
|
precisely what makes it valuable, and precisely what makes it dangerous. Six mitigations, all
|
|
@@ -1124,7 +1314,8 @@ required:
|
|
|
1124
1314
|
|
|
1125
1315
|
1. `--replay` is a distinct flag: `bir setup` passes it, and a `bir install` without it leaves
|
|
1126
1316
|
replay off in a project that never asked for it. `bir replay off` or `BIR_REPLAY=0` turns it off
|
|
1127
|
-
once installed,
|
|
1317
|
+
once installed — the recorder reads either when it starts, so `bir up --restart` applies it — and
|
|
1318
|
+
`bir off` takes the whole runner out of the way.
|
|
1128
1319
|
2. `BIR_REPLAY_ALLOW_SERVERS` — a comma-separated allowlist of the server keys bir may call by
|
|
1129
1320
|
itself: **direct** runs and divergence recovery (§8), though not `bir replay`, which an operator
|
|
1130
1321
|
runs by hand. Unset means *all wrapped servers*; the guide (§5 of `calculatedReplayGuide.md`)
|
|
@@ -1143,7 +1334,11 @@ required:
|
|
|
1143
1334
|
have), or the plan hands over; found at arm time, it declines as `missing_target`. A plan whose
|
|
1144
1335
|
only such steps write files (`Edit`, `MultiEdit`, `Write`, `NotebookEdit`) rejects only line
|
|
1145
1336
|
breaks and NUL. Plans with a step whose input is run do not send recent tool results to
|
|
1146
|
-
derivation, and the derivation prompt is redacted before it leaves the machine.
|
|
1337
|
+
derivation, and the derivation prompt is redacted before it leaves the machine. In a session of
|
|
1338
|
+
another host (§7.4) the tool names are the host's own, and Claude Code's lists of built-ins are
|
|
1339
|
+
not read: a value that a plan reads from the prompt is checked before it goes into a tool the
|
|
1340
|
+
host declared in `sinks` with its prompt (`command`, `write`), or whose plain name has a command
|
|
1341
|
+
word (sql, query, exec, shell, command, script, eval).
|
|
1147
1342
|
|
|
1148
1343
|
### 13.3 The control plane
|
|
1149
1344
|
|
|
@@ -1187,7 +1382,13 @@ fail because of BaseInstRunner.**
|
|
|
1187
1382
|
| `run_scenario` pending in two sessions and the request names neither | Nothing runs (`replay.run_ambiguous`); each model does its own task |
|
|
1188
1383
|
| The runner is off (`bir off`, or `BIR_OFF=1` for a session) | Hooks answer `{}` and nothing arms; the `bir` server lists no tools |
|
|
1189
1384
|
| Plan mode | Nothing is announced, pinned or approved |
|
|
1190
|
-
|
|
|
1385
|
+
| Another host (§7.4): the run's adapter asked for no plan with its prompt | Declined at gate 2 (`host_records_only`), `not_steered`; the model does the task |
|
|
1386
|
+
| Another host: the plan names a tool the agent that asked does not hold | Taken back before any step runs (`host_tool_missing`), `not_steered`; the model does the task |
|
|
1387
|
+
| Another host: the model makes a call of its own while the plan is live | The plan is handed over. The call runs as the model wrote it, never with the plan's arguments, and the note comes with its `/tool/post` |
|
|
1388
|
+
| Another host: a person refuses a step, or changes its call | The step is `skipped`, never `failed`, and the plan hands over; `baselineEligible: false`, and after a refusal `measured: false` (§11.1) |
|
|
1389
|
+
| Another host: the run fails, goes silent for two hours, or is closed with its plan still live | A live plan is handed over first: `fell_back` or `failed`. The report says `measured: false` and `baselineEligible: false`. A run that ended early is finished with an `error` and no final answer |
|
|
1390
|
+
| The service answers `429`: an account over one of its limits (built on the service, not deployed yet) | A create, a step or a report is sent once more, at most 2 s later, then dropped (`recorder.send_failed` with `HTTP 429`). A dropped step is a hole in its recording, and a run whose create got the 429 runs without its plan. A 429 that the second try did not cure counts as the service being unwell: for 5 to 60 s no prompt of that recorder waits for a match. An intent probe over its own limit is answered `200` with no match, never `429` |
|
|
1391
|
+
| Ticket redemption fails (offline, 409) | Logged; the turn is unaffected. An unredeemed ticket books nothing. A `503 report_not_booked` (the service as built; not deployed yet) is sent once more |
|
|
1191
1392
|
| Two reports for one ticket (retry, `Stop` racing `SessionEnd`) | Server books once, answers `200 { duplicate: true }` |
|
|
1192
1393
|
| Replay off (`BIR_REPLAY=0`, `bir replay off`, or never installed with `--replay`) | Exactly v1: stop recording on a match, do nothing else, and no prompt waits for the match |
|
|
1193
1394
|
|
|
@@ -1256,6 +1457,11 @@ POST /proxy/result { workId: string, result?: unknown, error?: string }
|
|
|
1256
1457
|
→ 200 { ok: true }
|
|
1257
1458
|
```
|
|
1258
1459
|
|
|
1460
|
+
The hook routes ([BaseInstRunner.md](BaseInstRunner.md), Phase 5) carry more for a host that is not
|
|
1461
|
+
Claude Code: optional fields that its adapter adds to `/session/prompt`, `/tool/pre`, `/tool/post`
|
|
1462
|
+
and `/session/stop`, and `hostFields` in `/health`. §7.4 names them, and says what they mean for a
|
|
1463
|
+
plan.
|
|
1464
|
+
|
|
1259
1465
|
### 16.2 The direct executor
|
|
1260
1466
|
|
|
1261
1467
|
```ts
|
|
@@ -1358,7 +1564,11 @@ same parameters from the same prompt:
|
|
|
1358
1564
|
live call, recent results) and a bearer token, `derived` as the service answered. This is the
|
|
1359
1565
|
ordinary path: nearly every session runs inside Claude Code on a subscription and has no key of
|
|
1360
1566
|
its own (segmented.md R-PARAM-5). The answer is re-normalised here against this runner's own copy
|
|
1361
|
-
of the schema, so the gate reads a `missing` computed from what will actually run.
|
|
1567
|
+
of the schema, so the gate reads a `missing` computed from what will actually run. The service as
|
|
1568
|
+
built (not deployed yet) cuts the prompt it reads new values from at 20,000 characters, and keeps
|
|
1569
|
+
the last five tool results. A plan started at the prompt sends no tool results, in any host: the
|
|
1570
|
+
transcript does not hold the new prompt yet, so its results would be an earlier prompt's, read as
|
|
1571
|
+
this task's (segmented.md R-PARAM-2a). A plan started at a tool call sends this prompt's.
|
|
1362
1572
|
- **Neither → return every key's recorded `sampleValue`**, `costUsd: 0`, `derived: false`. Settings
|
|
1363
1573
|
stand on their samples; a scenario with a target declines rather than run on a stale one.
|
|
1364
1574
|
- With an `apiKey`, `POST https://api.anthropic.com/v1/messages` with the same extraction prompt,
|
|
@@ -1488,7 +1698,9 @@ is refused.
|
|
|
1488
1698
|
`bir` invocations (and a plain `cd`), are host housekeeping, like `ToolSearch`
|
|
1489
1699
|
([housekeeping.ts](../src/record/housekeeping.ts)). They are never recorded as steps, and never
|
|
1490
1700
|
count as divergence from a plan. A session that fixes a plan must not become a plan that edits
|
|
1491
|
-
plans when it is replayed.
|
|
1701
|
+
plans when it is replayed. In a session of another host (§7.4) only the `bir` server's own tools
|
|
1702
|
+
are housekeeping: a tool the application names `Bash`, `ToolSearch` or `TodoWrite` is its own work,
|
|
1703
|
+
and is recorded like any other. The operator's walk-through is
|
|
1492
1704
|
[calculatedReplayGuide.md](calculatedReplayGuide.md) §9.2.
|
|
1493
1705
|
|
|
1494
1706
|
`--dry` maps to BaseIn's `POST /scenarios/:id/replay`, which evaluates the stored logic against the
|