@basein/runner 0.3.9 → 0.3.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +46 -5
- package/dist/auth/client.d.ts +15 -0
- package/dist/auth/client.js +34 -12
- package/dist/bin/bir-hooks.d.ts +1 -1
- package/dist/bin/bir-hooks.js +1 -1
- package/dist/bin/bir.d.ts +2 -0
- package/dist/bin/bir.js +60 -6
- package/dist/bin/investigate.d.ts +5 -0
- package/dist/bin/investigate.js +117 -15
- package/dist/bin/setup.d.ts +12 -0
- package/dist/bin/setup.js +185 -56
- package/dist/bin/update.d.ts +81 -0
- package/dist/bin/update.js +194 -0
- package/dist/config/adapters/claude-code.d.ts +17 -0
- package/dist/config/adapters/claude-code.js +28 -1
- package/dist/config/generate.d.ts +14 -0
- package/dist/control/daemon.d.ts +26 -1
- package/dist/control/daemon.js +53 -3
- package/dist/control/pin-guard.d.ts +6 -1
- package/dist/control/pin-guard.js +15 -1
- package/dist/control/recorder-keeper.js +2 -1
- package/dist/control/server.d.ts +200 -2
- package/dist/control/server.js +741 -48
- package/dist/control/transcript.d.ts +4 -0
- package/dist/control/transcript.js +4 -0
- package/dist/proxy/session.js +1 -0
- package/dist/record/housekeeping.d.ts +7 -1
- package/dist/record/housekeeping.js +11 -3
- package/dist/record/recorder.d.ts +34 -0
- package/dist/record/remote-recorder.d.ts +6 -0
- package/dist/record/remote-recorder.js +31 -4
- package/dist/record/switchable-recorder.js +1 -0
- package/dist/replay/controller.d.ts +132 -3
- package/dist/replay/controller.js +248 -24
- package/dist/replay/derive.js +2 -0
- package/dist/replay/handover.d.ts +13 -1
- package/dist/replay/handover.js +19 -0
- package/dist/replay/safety.d.ts +15 -4
- package/dist/replay/safety.js +19 -6
- package/dist/replay/source-run.js +2 -1
- package/dist/util/service-headers.d.ts +24 -0
- package/dist/util/service-headers.js +47 -0
- package/docs/BaseInstRunner.md +115 -1
- package/docs/calculatedReplay.md +209 -32
- package/docs/calculatedReplayGuide.md +133 -12
- package/docs/installRun.md +91 -10
- package/docs/quickstart.md +64 -9
- package/package.json +1 -1
- package/production.md +81 -17
- package/scripts/install.ps1 +25 -1
package/docs/calculatedReplay.md
CHANGED
|
@@ -45,7 +45,7 @@ this design has a ladder whose cheapest rung is free — and the more of a stack
|
|
|
45
45
|
|---|---|---|
|
|
46
46
|
| Runs MCP tools itself | No (`replay.md` §12: *"rrepeat never executes tools — not even MCP"*) | **Yes**, on the connection it already owns |
|
|
47
47
|
| Divergence fallback | A second Agent SDK session — the dominant cost | Direct `tools/call` through the proxies — **$0** |
|
|
48
|
-
| Reaches built-ins (`Bash`, `Read`) | Only by steering the live session | `Read`, `Glob`, `Grep`, `Edit`, `Write` and `Bash` run on the runner itself, jailed to the project (`BIR_BUILTIN_TOOLS`, `BIR_BASH_ALLOW`); any other built-in is steered |
|
|
48
|
+
| Reaches built-ins (`Bash`, `Read`) | Only by steering the live session | `Read`, `Glob`, `Grep`, `Edit`, `Write` and `Bash` run on the runner itself, jailed to the project (`BIR_BUILTIN_TOOLS`, `BIR_BASH_ALLOW`); any other built-in is steered. In a host that is not Claude Code a tool with one of these names is the application's own, and the runner runs none of them (§7.4) |
|
|
49
49
|
| Needs `@anthropic-ai/*` at runtime | Yes | No (§16.3) |
|
|
50
50
|
|
|
51
51
|
---
|
|
@@ -131,6 +131,10 @@ Server-side, and worth stating because the threshold in §4 is a judgement about
|
|
|
131
131
|
(`recordings/embeddings.ts`).
|
|
132
132
|
- Candidates are **the caller's own canonical runs only** (`listUserRunEmbeddings(userId)`);
|
|
133
133
|
duplicates and sub-threshold runs are excluded from the pool at record time.
|
|
134
|
+
- **Inside one host** — on the service as built, which is not deployed yet. A run created with
|
|
135
|
+
`framework` `langchain` or `langchain:<scope>` (§7.4) is matched only against recordings and
|
|
136
|
+
plans of that same text. Everything else (`baseinstrunner`, `custom`, the built-ins) is Claude
|
|
137
|
+
Code's pool, as before.
|
|
134
138
|
- The match is the single best cosine ≥ `SIMILARITY_THRESHOLD` (default **0.9**).
|
|
135
139
|
- A matched run only carries a `scenarioId` when its scenario exists **and** is `state: 'ready'`.
|
|
136
140
|
|
|
@@ -191,7 +195,7 @@ Every gate that declines logs **one** `replay.decision` line carrying the reason
|
|
|
191
195
|
| # | Gate | Declines when | Outcome floor |
|
|
192
196
|
|---|---|---|---|
|
|
193
197
|
| 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched) |
|
|
194
|
-
| 2 | **Replay enabled** | `BIR_REPLAY === "0"`, or, with it unset, the project's stored switch is off (`bir replay off`, or a `bir install` without `--replay`) | `not_steered` |
|
|
198
|
+
| 2 | **Replay enabled** | `BIR_REPLAY === "0"`, or, with it unset, the project's stored switch is off (`bir replay off`, or a `bir install` without `--replay`). In a session of another host, also when the run's adapter asked for no plan with its prompt (`host_records_only`, §7.4) | `not_steered` |
|
|
195
199
|
| 3 | **Scenario ready** | `scenario == null`, `state !== "ready"`, or `steps.length === 0` | `not_steered` |
|
|
196
200
|
| 4 | ~~**Similarity**~~ | Retired: a match is the decision to use the scenario (below) | — |
|
|
197
201
|
| 5 | **Tool coverage** | No step is executable anywhere (§5 → `mode: "none"`) | `not_steered` |
|
|
@@ -204,6 +208,9 @@ inside the prompt's budget (§10) is dropped (`match_timeout`, `arm_timeout`), a
|
|
|
204
208
|
after the prompt was answered is retired unannounced (`match_late`, `arm_unannounced`). A `direct`
|
|
205
209
|
plan is declined when no running `bir-scenario` has said it serves `run_scenario`
|
|
206
210
|
(`no_delivery_tool`, §6.3). In plan mode (`permission_mode: "plan"`) nothing is announced at all.
|
|
211
|
+
In a session of another host (§7.4) a plan that armed is taken back, before any step of it runs,
|
|
212
|
+
when it names a tool the agent that asked does not hold (`host_tool_missing`): it is booked like
|
|
213
|
+
any gate's decline, `not_steered`, with no step marked failed.
|
|
207
214
|
|
|
208
215
|
Gate 4 is retired. Up to 0.3.6 it declined a match under `BIR_MIN_STEER_SIMILARITY` (0.92),
|
|
209
216
|
above the server's 0.9, on the argument that the server only decided *"do not record this again"*
|
|
@@ -277,6 +284,12 @@ A **mixed** scenario in `steer` mode still gets direct execution for its wrapped
|
|
|
277
284
|
diverges (§8), so coverage is not all-or-nothing at the divergence boundary — only at the arm-time
|
|
278
285
|
mode choice.
|
|
279
286
|
|
|
287
|
+
**None of this applies in a session of another host.** When a session's requests name a host that
|
|
288
|
+
is not Claude Code (§7.4), a tool's name carries no Claude Code meaning: `Bash`, `Read` or an
|
|
289
|
+
`mcp__` prefix name the application's own tools, no proxy of this folder sees their calls, and the
|
|
290
|
+
runner runs none of them, whatever `BIR_BUILTIN_TOOLS` lists. Every step is `live`, so the mode is
|
|
291
|
+
always `steer`.
|
|
292
|
+
|
|
280
293
|
---
|
|
281
294
|
|
|
282
295
|
## 6. Mode A — direct execution
|
|
@@ -423,7 +436,9 @@ another way. Divergence (§8) is for steer plans only.
|
|
|
423
436
|
The mixed rung. Ported from RRepeat's `replay.md` §3/§12, which is the right design and is already
|
|
424
437
|
proven: the model chooses *which* tool and *when*; the control server overrides the *arguments* and
|
|
425
438
|
threads state between steps. The session never leaves the model's hands, so every
|
|
426
|
-
`tool_use`/`tool_result` in the transcript is genuine and continuation is automatic.
|
|
439
|
+
`tool_use`/`tool_result` in the transcript is genuine and continuation is automatic. That is Claude
|
|
440
|
+
Code. In a host that is not Claude Code the same mode is stepped through by the host's adapter, in
|
|
441
|
+
the model's place, and a call the model makes itself is never pinned (§7.4).
|
|
427
442
|
|
|
428
443
|
**Arm** at `UserPromptSubmit` — plan on `run.plan`, `run.planToolUse = new Set()`, directive via
|
|
429
444
|
`additionalContext`, **no `decision`** so the prompt still reaches the model:
|
|
@@ -589,15 +604,91 @@ In [`HOOK_ROUTES`](../src/config/adapters/claude-code.ts) neither `PreToolUse` n
|
|
|
589
604
|
carries `async: true`, so this already holds — **and must never be relaxed**. Add it to the comment
|
|
590
605
|
block there, next to the `SessionEnd` note that records the same class of lesson.
|
|
591
606
|
|
|
607
|
+
### 7.4 A host that is not Claude Code
|
|
608
|
+
|
|
609
|
+
Everything above assumes Claude Code: its hooks report the turn, and its model makes the plan's
|
|
610
|
+
calls. The recorder's local protocol also serves a host that has neither. Today that is a
|
|
611
|
+
LangChain application: a Python program whose agent is built with LangChain's `create_agent` and
|
|
612
|
+
the package `langchain-basic2202`, a LangChain middleware that talks to the recorder over that
|
|
613
|
+
protocol. The package is *the adapter* below, and it is not on PyPI yet. Claude Code's hooks are
|
|
614
|
+
unchanged, and every new field is optional. A session whose requests carry
|
|
615
|
+
`host: { app, version?, adapter?, scope? }` with an `app` other than `claude-code` is a session of
|
|
616
|
+
another host ([`control/server.ts`](../src/control/server.ts)). The fields named below are what
|
|
617
|
+
such an adapter adds to the hook routes of the control plane
|
|
618
|
+
([BaseInstRunner.md](BaseInstRunner.md), Phase 5). `hostFields` in `/health` tells an adapter how
|
|
619
|
+
much of this a recorder understands: with 1 it records, and from 2 on it may also ask for a plan
|
|
620
|
+
and step through it.
|
|
621
|
+
|
|
622
|
+
For such a session:
|
|
623
|
+
|
|
624
|
+
- **The run names its host.** It is created with `framework` = the app, or `app:scope`
|
|
625
|
+
(`langchain`, `langchain:billing`), and `metadata.host`. A Claude Code run is created as before
|
|
626
|
+
(`framework: "baseinstrunner"`).
|
|
627
|
+
- **Only `/session/prompt` opens a run.** Any other hook that finds no open run opens none and
|
|
628
|
+
records nothing: it is answered `{}`, or `{ noPlan: true }` when it asked for a step.
|
|
629
|
+
- **A plan starts only for a prompt that said `replay: true`.** Otherwise the match is declined at
|
|
630
|
+
gate 2 with `host_records_only` (§4). The prompt's answer carries the armed plan as data too
|
|
631
|
+
(`plan: { scenarioId, mode, steps: [{ stepIndex, toolName }], stopAt? }`): the directive beside
|
|
632
|
+
it is a text written for a model.
|
|
633
|
+
- **The mode is always `steer`, and the runner runs no step itself.** The host runs its own tools.
|
|
634
|
+
Tool names carry no Claude Code meaning: `Bash`, `Read`, `TodoWrite`, `ToolSearch`… are the
|
|
635
|
+
application's own tools, and nothing is skipped as housekeeping except the `bir` server's own
|
|
636
|
+
tools. So there is no direct mode there, and no divergence recovery (§8): nothing can be executed
|
|
637
|
+
here in the host's place.
|
|
638
|
+
- **The adapter asks for each step**, with a `/tool/pre` that says `scripted: true`, and the host
|
|
639
|
+
runs its own tool. The answer is the step's arguments (`hookSpecificOutput.updatedInput`), or
|
|
640
|
+
`{ noPlan: true }` when the recorder holds no live plan, or `{ declined: "host_tool_missing" }`
|
|
641
|
+
when the plan names a tool that is not in `tools` (the agent's tools, sent with the first such
|
|
642
|
+
request of a run), or a refusal whose reason is the hand-over note when the plan stopped in that
|
|
643
|
+
very request. The step's output is threaded from the `/tool/post` that follows: every step is
|
|
644
|
+
`live` (§7.2).
|
|
645
|
+
- **The model's own call is never pinned.** A `/tool/pre` without `scripted` is the model's own
|
|
646
|
+
call, and it is never given the plan's arguments. When a plan is live it is handed over (§8): the
|
|
647
|
+
call runs as the model wrote it, and the hand-over note comes with that call's `/tool/post`.
|
|
648
|
+
- **The host may stop a step itself**, and says so with the step's `/tool/post`.
|
|
649
|
+
`host_stop: "rejected"` means a person said no, or answered in the tool's place, or a call limit
|
|
650
|
+
blocked the call; `"edited"` means a person changed the call. Either way the step's verdict is
|
|
651
|
+
`skipped`, not `failed`, nothing is counted against the plan, and the plan hands over (§8). §11.1
|
|
652
|
+
has what the report says.
|
|
653
|
+
- **There is no transcript.** What Claude Code's transcript gives comes with the requests instead:
|
|
654
|
+
with a `/tool/pre`, what the model wrote and reasoned before the call (`context`, `thinking`);
|
|
655
|
+
with `/session/stop`, the answer and the tokens (`final_answer`, `usage`). The cost comes from
|
|
656
|
+
that `usage` (§11.4). The conversation's earlier tool results do not come with the prompt: a new
|
|
657
|
+
prompt's values are read from the prompt alone, here as in Claude Code (§16.3), and a
|
|
658
|
+
`recent_results` an adapter still sends is not read.
|
|
659
|
+
- **A run can end without an ordinary stop.** With `/session/stop`, `failed` closes the run with
|
|
660
|
+
that text as its `error` (cleaned of secrets, cut) and no final answer, and `incomplete` marks
|
|
661
|
+
the finish `incomplete: { lostSteps, lossy }`. A run sealed after two hours of silence is closed
|
|
662
|
+
with `error: "turn ended early: idle"`. A run that is closed while its plan is still live has the
|
|
663
|
+
plan handed over first, so its report says what happened (§11.1). On the service as built (not
|
|
664
|
+
deployed yet), a run with an `error` or with that mark is never picked for automatic calculation
|
|
665
|
+
and no longer catches its prompt, unless its owner preferred it or it already has a plan. Only a
|
|
666
|
+
run of another host sends either today: a Claude Code run that was interrupted or lost steps is
|
|
667
|
+
finished as before.
|
|
668
|
+
- **A value that a plan reads from the prompt is checked against the host's own sinks** before it
|
|
669
|
+
goes into a tool: the `sinks` the host declared with its prompt (§13.2, mitigation 6).
|
|
670
|
+
- **The switch is the same.** `bir setup --host langchain` stores replay on for the folder
|
|
671
|
+
(`--no-replay` stores it off), and `bir replay on` and `bir replay off` work there as anywhere.
|
|
672
|
+
- **The audit lines name the host**: `run.start … host=langchain`,
|
|
673
|
+
`plan.armed … mode=steer host=langchain`, and `run.finish … ended=<reason>` when the run did not
|
|
674
|
+
end with an ordinary stop. `run.incomplete` and `run.host_failed` are what the host said with its
|
|
675
|
+
stop.
|
|
676
|
+
|
|
677
|
+
The how-to for a LangChain developer — setting up the folder, the two lines of Python, what to do
|
|
678
|
+
when something is wrong — is `langChainGuide.md` in the root of the runner's repository (it is not
|
|
679
|
+
in the npm package), and is not repeated here. The design guide with every decision is
|
|
680
|
+
`integratioLangChane.md` in the service's repository.
|
|
681
|
+
|
|
592
682
|
---
|
|
593
683
|
|
|
594
684
|
## 8. Divergence
|
|
595
685
|
|
|
596
686
|
In a steer plan, the model called a tool the plan did not expect, or the call had no `tool_use_id`.
|
|
597
|
-
(A direct plan never diverges, §6.4.
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
687
|
+
(A direct plan never diverges, §6.4. Nor does a plan in a host that is not Claude Code, §7.4: only
|
|
688
|
+
the host can run its tools, so the plan is handed over and nothing is executed in its place. A
|
|
689
|
+
`toolInputLogic` that throws is not a divergence either: re-executing would start at that same step
|
|
690
|
+
and throw again, so the step is recorded as failed and the task is handed to the model, with a note
|
|
691
|
+
in place of the call.) Three tiers, tried in order. Here is where the §0 asymmetry pays.
|
|
601
692
|
|
|
602
693
|
**Tier 1 — direct-execute the remainder (new; the common case).**
|
|
603
694
|
Take every not-yet-run step, execute it, and hand the composed result back. This happens inside the
|
|
@@ -636,6 +727,12 @@ about it:
|
|
|
636
727
|
- beside a pin still in flight: *"… was not run by the scenario: another of its calls was still in
|
|
637
728
|
flight, and a scenario runs one call at a time."*, and the pin itself *"was started and has not
|
|
638
729
|
reported back"*.
|
|
730
|
+
- stopped by its host, in a session of another host (§7.4): *"… was not run by the host. The result
|
|
731
|
+
above was written by the host, not by the tool. Do not run the step again unless the user asks."*
|
|
732
|
+
(`host_rejected`), or *"… ran with arguments a person chose, not the scenario's. Its result above
|
|
733
|
+
is real and its side effects are in place. Do not run it again."* (`host_edited`). When no step
|
|
734
|
+
ran before it, the note says *"No step of the scenario ran as it was planned."* instead of *"No
|
|
735
|
+
step … ran"*. The report's `fallbackKind` stays `step_failed`.
|
|
639
736
|
|
|
640
737
|
It ends with the remaining planned steps and the parameters derived for this request, within 4 000
|
|
641
738
|
characters.
|
|
@@ -781,8 +878,8 @@ The hook is on the user's critical path. Every await here is a stall the user wa
|
|
|
781
878
|
|
|
782
879
|
| Moment | Route | Hook timeout | Answered by | What may be awaited | Budget |
|
|
783
880
|
|---|---|---|---|---|---|
|
|
784
|
-
| `UserPromptSubmit` | `/session/prompt` | 10 s | 7 s | the match; for a scenario with a target, its derivation; a steer plan's argument preview | match `BIR_MATCH_BUDGET_MS` = **3000**; match and arming together ≤ **6 s**; derivation ≤ `BIR_HOOK_DERIVE_BUDGET_MS` = **
|
|
785
|
-
| `PreToolUse` | `/tool/pre` | 15 s | 12 s | `plan.ready()` (derivation); on a divergence, the remaining steps | derivation ≤ min(`BIR_DERIVE_BUDGET_MS`, `BIR_HOOK_DERIVE_BUDGET_MS`) = **
|
|
881
|
+
| `UserPromptSubmit` | `/session/prompt` | 10 s | 7 s | the match; for a scenario with a target, its derivation; a steer plan's argument preview | match `BIR_MATCH_BUDGET_MS` = **3000**; match and arming together ≤ **6 s**; derivation ≤ `BIR_HOOK_DERIVE_BUDGET_MS` = **8000**, cut short by the 6 s |
|
|
882
|
+
| `PreToolUse` | `/tool/pre` | 15 s | 12 s | `plan.ready()` (derivation); on a divergence, the remaining steps | derivation ≤ min(`BIR_DERIVE_BUDGET_MS`, `BIR_HOOK_DERIVE_BUDGET_MS`) = **8000**; all replay work ≤ `BIR_PRETOOL_REPLAY_BUDGET_MS` = **9000** (held at ≤ 10 000) |
|
|
786
883
|
| `run_scenario` | `/scenario/run` | `bir-scenario`'s own, **290 s** (`BIR_SCENARIO_RUN_TIMEOUT_MS`) | — | `ready()` + N steps | derivation `BIR_DERIVE_BUDGET_MS` = **8000**; plan `BIR_REPLAY_PLAN_BUDGET_MS` = **100000** (`BIR_REPLAY_BUDGET_MS` overrides it where set); per step `BIR_STEP_TIMEOUT_MS` = **60000**, clipped to what the plan has left |
|
|
787
884
|
| `PostToolUse` | `/tool/post` | 5 s | 2 s | nothing | — |
|
|
788
885
|
| `Stop` | `/session/stop` | 5 s | 2 s | nothing (report is queued) | — |
|
|
@@ -863,6 +960,25 @@ beside a pinned one hands over as `fell_back` or `failed` (§7). A direct run ca
|
|
|
863
960
|
(Esc, or `bir-scenario`'s own timeout) books `fell_back` or `failed` with `measured: false`, never
|
|
864
961
|
`diverged`: that turn's cost says nothing about what the task costs.
|
|
865
962
|
|
|
963
|
+
A host that is not Claude Code (§7.4) adds three endings, and none of them is counted against the
|
|
964
|
+
plan. The report says so with two fields: `measured`, and `baselineEligible: false`, which keeps
|
|
965
|
+
the turn's cost out of the baseline samples (absent, the turn is eligible).
|
|
966
|
+
|
|
967
|
+
- **A person's "no".** A step the host did not run (`host_stop: "rejected"`: a person said no, or
|
|
968
|
+
answered in the tool's place, or a call limit blocked the call) is recorded `skipped`, never
|
|
969
|
+
`failed`, and the plan hands over: `fell_back` when a step ran before it, `failed` when none did.
|
|
970
|
+
The report says `measured: false` and `baselineEligible: false`: the task ended early, so the
|
|
971
|
+
turn's cost is no measure of it.
|
|
972
|
+
- **A changed call.** A step that ran with arguments a person chose (`"edited"`) is `skipped` too,
|
|
973
|
+
and the plan hands over the same way. The cost stays measured, with `baselineEligible: false`.
|
|
974
|
+
- **A run closed with its plan live.** The plan is handed over first, so the report says
|
|
975
|
+
`fell_back` (a step ran) or `failed` (none ran), never `not_steered`, with `measured: false` and
|
|
976
|
+
`baselineEligible: false`. A run that ended early for any other reason (it failed, or went
|
|
977
|
+
silent) reports the same two fields.
|
|
978
|
+
|
|
979
|
+
A plan taken back before any step ran (`host_tool_missing`, §4) is a decline like any other:
|
|
980
|
+
`not_steered`.
|
|
981
|
+
|
|
866
982
|
Direct mode reports `steered_full`. BaseIn's zod enum
|
|
867
983
|
(mirrored in [`replay/types.ts`](../src/replay/types.ts)) has no `direct` member, and `steered_full`
|
|
868
984
|
means exactly what happened: every planned step executed under the scenario's control and the
|
|
@@ -887,6 +1003,11 @@ POST /scenarios/:scenarioId/executions
|
|
|
887
1003
|
An **unredeemed** ticket leaves no execution row at all. That asymmetry is intentional and worth
|
|
888
1004
|
preserving: silence loses a saving; it never invents one.
|
|
889
1005
|
|
|
1006
|
+
**How late a report may arrive.** The deployed service redeems a ticket for one hour after its plan
|
|
1007
|
+
was handed out, and answers a later report `409 invalid_ticket`. The service as built (not deployed
|
|
1008
|
+
yet) redeems it for 24 hours. It also answers a report whose write failed `503 report_not_booked`,
|
|
1009
|
+
and the recorder sends that report once more.
|
|
1010
|
+
|
|
890
1011
|
### 11.2a What each step did (the service's errorshandling.md)
|
|
891
1012
|
|
|
892
1013
|
The last five fields are the diagnosis, and none of the money depends on them. The service's
|
|
@@ -922,14 +1043,19 @@ a replay that sent 3600.
|
|
|
922
1043
|
are read on a page, never parsed, and with its marker a cut payload still fits the 4,000
|
|
923
1044
|
characters the service keeps of each — so the page shows this cut and its sizes, not a second one.
|
|
924
1045
|
The recorder redacts once more as it sends.
|
|
925
|
-
- **Inside one small request.** The service reads `POST /scenarios/:id/executions` with
|
|
926
|
-
JSON parser (only `/recordings` has the 5 MB one) and answers a larger body 413, which
|
|
927
|
-
the turn's cost with it. So the recorder holds the whole report to 96 KiB
|
|
928
|
-
it writes the costs and the verdicts first, then adds payloads while they
|
|
929
|
-
first, then every step's input, then the outputs, in the order the steps
|
|
930
|
-
not fit is left out, and its step keeps its verdict.
|
|
1046
|
+
- **Inside one small request.** The deployed service reads `POST /scenarios/:id/executions` with
|
|
1047
|
+
its 100 kB JSON parser (only `/recordings` has the 5 MB one) and answers a larger body 413, which
|
|
1048
|
+
would lose the turn's cost with it. So the recorder holds the whole report to 96 KiB
|
|
1049
|
+
(`REPORT_MAX_BYTES`): it writes the costs and the verdicts first, then adds payloads while they
|
|
1050
|
+
fit — the failed steps' first, then every step's input, then the outputs, in the order the steps
|
|
1051
|
+
ran. A payload that does not fit is left out, and its step keeps its verdict. The service as
|
|
1052
|
+
built (not deployed yet) accepts 1 MB on this route and on `/scenarios/:id/derive`, and cuts a
|
|
1053
|
+
report's `prompt` at 10,000 characters where the deployed one refuses the whole report with 400.
|
|
1054
|
+
The recorder keeps its report under 100 kB all the same: older services have that limit.
|
|
931
1055
|
- **Absent where there was none.** A step whose input logic threw has no input. A `skipped` step
|
|
932
|
-
has no output, and neither has a call the host says failed without a result.
|
|
1056
|
+
has no output, and neither has a call the host says failed without a result. The exception is a
|
|
1057
|
+
step its host stopped itself (§7.4): its verdict keeps what the host handed back, and after a
|
|
1058
|
+
changed call it has no input, because only the host knows the arguments that call ran with.
|
|
933
1059
|
- **Only for the reported scenario's own steps.** A step inside a called segment is reported with
|
|
934
1060
|
its verdict and its `scenarioId`, without payloads: the console still files verdicts by step
|
|
935
1061
|
number alone (the service's plan-services.md, W1.9), and a called segment numbers its steps from
|
|
@@ -938,9 +1064,11 @@ a replay that sent 3600.
|
|
|
938
1064
|
`ok` and `recorded` are passes (`recorded` with an asterisk: the values came from the source run,
|
|
939
1065
|
which happens only with `BIR_REPLAY_RECORDED_OUTPUTS=1` and in `bir replay`). `skipped` produced no
|
|
940
1066
|
result here: its call could not be sent (a built-in outside the session, no proxy for its server,
|
|
941
|
-
only another open session's proxy), or it was sent and timed out or was refused.
|
|
942
|
-
a
|
|
943
|
-
|
|
1067
|
+
only another open session's proxy), or it was sent and timed out or was refused. A step that its
|
|
1068
|
+
host stopped itself, in a host that is not Claude Code, is `skipped` as well: a person refused it,
|
|
1069
|
+
or changed its call (§7.4). The plan stops at a `skipped` step.
|
|
1070
|
+
**`failed` is the only one that means the scenario is broken** rather than the environment — it is
|
|
1071
|
+
the scenario's own logic throwing, and it is what `Recalculate` fixes.
|
|
944
1072
|
|
|
945
1073
|
Three rules this must keep:
|
|
946
1074
|
|
|
@@ -969,7 +1097,7 @@ service's matching `202 { recorded: false }`): it now *is* worth sending when a
|
|
|
969
1097
|
| Component | Source | Notes |
|
|
970
1098
|
|---|---|---|
|
|
971
1099
|
| `deriveCostUsd` | our own Haiku call's `usage`, priced locally | the only tokens replay itself spends |
|
|
972
|
-
| `sessionCostUsd` | transcript usage **delta** for this run (§11.4) | the live turn: prompt, tool calls, answer |
|
|
1100
|
+
| `sessionCostUsd` | transcript usage **delta** for this run (§11.4); in a host that is not Claude Code, the `usage` its adapter sends with `/session/stop` | the live turn: prompt, tool calls, answer |
|
|
973
1101
|
| `fallbackCostUsd` | **0.00** | direct execution spends no tokens (§8) |
|
|
974
1102
|
|
|
975
1103
|
> ### ⚠ Verify the pricing table before copying it
|
|
@@ -1010,9 +1138,23 @@ rather than to a negative cost. A run with **no mark** (one opened lazily by a p
|
|
|
1010
1138
|
prompt) reports `measured: false` — better to say "unmeasured" than to hand the ledger an inflated
|
|
1011
1139
|
baseline.
|
|
1012
1140
|
|
|
1013
|
-
`measured` is `true` only when a mark existed **and**
|
|
1014
|
-
|
|
1015
|
-
|
|
1141
|
+
`measured` is `true` only when a mark existed, tokens were found since it, **and** every model of
|
|
1142
|
+
the turn has a price in the runner's table (`priceUsage` and `hasModelPricing`,
|
|
1143
|
+
[`control/server.ts`](../src/control/server.ts)). A model outside the table used to cost $0, and
|
|
1144
|
+
that zero was called measured: a saving nobody made. Now such a turn is not measured, in every
|
|
1145
|
+
host. A `run.unmeasured` line says why, with `why=`, when the run had no mark or when a model has
|
|
1146
|
+
no price (then with `models=`). A turn whose transcript held no tokens since the mark is not
|
|
1147
|
+
measured either, and writes no such line. No price table and no `PRICING_VERSION` changed with this.
|
|
1148
|
+
|
|
1149
|
+
The server judges a report again. It stores one whose `pricingVersion` differs from its own as
|
|
1150
|
+
unmeasured rather than trusting it. As built (not deployed yet) it does the same with a report that
|
|
1151
|
+
says `measured: true` with a session cost of $0, and a finish with a cost of 0 stores no cost.
|
|
1152
|
+
|
|
1153
|
+
**A host that is not Claude Code has no transcript, and no mark** (§7.4). Its cost comes from the
|
|
1154
|
+
`usage` its adapter sends with `/session/stop`, and is measured only when at least one entry has
|
|
1155
|
+
tokens and every model in the list has a price; with no usage at all, the line says
|
|
1156
|
+
`why="the host sent no usage"`. Such a host hands over one number for the whole run, so with more
|
|
1157
|
+
than one plan in one run none of them is measured.
|
|
1016
1158
|
|
|
1017
1159
|
---
|
|
1018
1160
|
|
|
@@ -1049,10 +1191,11 @@ bir replay --scenario scn_… --prompt "…" --dry
|
|
|
1049
1191
|
|
|
1050
1192
|
Runs the same plan through the same executor, against the same live proxies, printing the per-step
|
|
1051
1193
|
input/output and the response model. It is how you test a scenario without a session, how a Tier 2
|
|
1052
|
-
or
|
|
1053
|
-
|
|
1054
|
-
|
|
1055
|
-
direct-run ceiling
|
|
1194
|
+
client — or one that is neither Claude Code nor a host whose adapter steps through plans (§7.4) —
|
|
1195
|
+
gets any replay at all, and the first thing to reach for when a steered turn behaves oddly. The
|
|
1196
|
+
operator named the scenario, so it keeps its own contract: a step it cannot send stands on its
|
|
1197
|
+
recorded output, the allowlist does not gate it, and it is not held to the 270 s direct-run ceiling
|
|
1198
|
+
(§10).
|
|
1056
1199
|
|
|
1057
1200
|
---
|
|
1058
1201
|
|
|
@@ -1090,6 +1233,15 @@ What actually constrains it:
|
|
|
1090
1233
|
and a `bir install` without it stores replay off in a project that never asked for it. A
|
|
1091
1234
|
compromised or misconfigured `BIR_AUTH_URL` therefore cannot start executing logic on a machine
|
|
1092
1235
|
whose operator never asked for replay.
|
|
1236
|
+
- **A small environment in an application's folder.** A recorder started for a folder that is set
|
|
1237
|
+
up for another host and not for Claude Code (`bir setup --host langchain`, `bir up`) gets a small
|
|
1238
|
+
environment, not the whole environment of the window that started it: `PATH`, the home and temp
|
|
1239
|
+
variables, what the system needs to start a process, the proxy and certificate variables, and
|
|
1240
|
+
every `BIR_*` variable (`smallEnv`, [`control/daemon.ts`](../src/control/daemon.ts)). Step code
|
|
1241
|
+
runs inside the recorder with no sandbox, so a model key or a database password of the
|
|
1242
|
+
application must not reach it. A folder that is set up for Claude Code too keeps the whole
|
|
1243
|
+
environment, because Claude Code's own steps may need it. Nothing changes for a Claude Code
|
|
1244
|
+
project.
|
|
1093
1245
|
|
|
1094
1246
|
If that is not enough for a deployment, the mitigation is a real one and is out of scope here:
|
|
1095
1247
|
evaluate the logic in a separate `node:vm` context with a frozen, minimal global. Filed as §20 question 3.
|
|
@@ -1152,6 +1304,9 @@ This is the sharpest edge in the whole design, and it must not be buried.
|
|
|
1152
1304
|
answered `allow` while the runner has announced a direct plan, so it brings no surprise prompt
|
|
1153
1305
|
(and no denial under `claude -p`); the user's own deny or ask rules for it still apply.
|
|
1154
1306
|
- **Plan mode** (`permission_mode: "plan"`) gets none of it: no directive, no pin, no `allow`.
|
|
1307
|
+
- **A host that is not Claude Code** (§7.4) is neither mode as written here: the runner runs no
|
|
1308
|
+
step. It hands the host's adapter each step's arguments, the host runs its own tool, and the
|
|
1309
|
+
adapter reports a step that a person refused or changed (`host_stop`).
|
|
1155
1310
|
|
|
1156
1311
|
So a scenario is, in effect, a **pre-approved list of tool calls with computed arguments**. That is
|
|
1157
1312
|
precisely what makes it valuable, and precisely what makes it dangerous. Six mitigations, all
|
|
@@ -1159,7 +1314,8 @@ required:
|
|
|
1159
1314
|
|
|
1160
1315
|
1. `--replay` is a distinct flag: `bir setup` passes it, and a `bir install` without it leaves
|
|
1161
1316
|
replay off in a project that never asked for it. `bir replay off` or `BIR_REPLAY=0` turns it off
|
|
1162
|
-
once installed,
|
|
1317
|
+
once installed — the recorder reads either when it starts, so `bir up --restart` applies it — and
|
|
1318
|
+
`bir off` takes the whole runner out of the way.
|
|
1163
1319
|
2. `BIR_REPLAY_ALLOW_SERVERS` — a comma-separated allowlist of the server keys bir may call by
|
|
1164
1320
|
itself: **direct** runs and divergence recovery (§8), though not `bir replay`, which an operator
|
|
1165
1321
|
runs by hand. Unset means *all wrapped servers*; the guide (§5 of `calculatedReplayGuide.md`)
|
|
@@ -1178,7 +1334,11 @@ required:
|
|
|
1178
1334
|
have), or the plan hands over; found at arm time, it declines as `missing_target`. A plan whose
|
|
1179
1335
|
only such steps write files (`Edit`, `MultiEdit`, `Write`, `NotebookEdit`) rejects only line
|
|
1180
1336
|
breaks and NUL. Plans with a step whose input is run do not send recent tool results to
|
|
1181
|
-
derivation, and the derivation prompt is redacted before it leaves the machine.
|
|
1337
|
+
derivation, and the derivation prompt is redacted before it leaves the machine. In a session of
|
|
1338
|
+
another host (§7.4) the tool names are the host's own, and Claude Code's lists of built-ins are
|
|
1339
|
+
not read: a value that a plan reads from the prompt is checked before it goes into a tool the
|
|
1340
|
+
host declared in `sinks` with its prompt (`command`, `write`), or whose plain name has a command
|
|
1341
|
+
word (sql, query, exec, shell, command, script, eval).
|
|
1182
1342
|
|
|
1183
1343
|
### 13.3 The control plane
|
|
1184
1344
|
|
|
@@ -1222,7 +1382,13 @@ fail because of BaseInstRunner.**
|
|
|
1222
1382
|
| `run_scenario` pending in two sessions and the request names neither | Nothing runs (`replay.run_ambiguous`); each model does its own task |
|
|
1223
1383
|
| The runner is off (`bir off`, or `BIR_OFF=1` for a session) | Hooks answer `{}` and nothing arms; the `bir` server lists no tools |
|
|
1224
1384
|
| Plan mode | Nothing is announced, pinned or approved |
|
|
1225
|
-
|
|
|
1385
|
+
| Another host (§7.4): the run's adapter asked for no plan with its prompt | Declined at gate 2 (`host_records_only`), `not_steered`; the model does the task |
|
|
1386
|
+
| Another host: the plan names a tool the agent that asked does not hold | Taken back before any step runs (`host_tool_missing`), `not_steered`; the model does the task |
|
|
1387
|
+
| Another host: the model makes a call of its own while the plan is live | The plan is handed over. The call runs as the model wrote it, never with the plan's arguments, and the note comes with its `/tool/post` |
|
|
1388
|
+
| Another host: a person refuses a step, or changes its call | The step is `skipped`, never `failed`, and the plan hands over; `baselineEligible: false`, and after a refusal `measured: false` (§11.1) |
|
|
1389
|
+
| Another host: the run fails, goes silent for two hours, or is closed with its plan still live | A live plan is handed over first: `fell_back` or `failed`. The report says `measured: false` and `baselineEligible: false`. A run that ended early is finished with an `error` and no final answer |
|
|
1390
|
+
| The service answers `429`: an account over one of its limits (built on the service, not deployed yet) | A create, a step or a report is sent once more, at most 2 s later, then dropped (`recorder.send_failed` with `HTTP 429`). A dropped step is a hole in its recording, and a run whose create got the 429 runs without its plan. A 429 that the second try did not cure counts as the service being unwell: for 5 to 60 s no prompt of that recorder waits for a match. An intent probe over its own limit is answered `200` with no match, never `429` |
|
|
1391
|
+
| Ticket redemption fails (offline, 409) | Logged; the turn is unaffected. An unredeemed ticket books nothing. A `503 report_not_booked` (the service as built; not deployed yet) is sent once more |
|
|
1226
1392
|
| Two reports for one ticket (retry, `Stop` racing `SessionEnd`) | Server books once, answers `200 { duplicate: true }` |
|
|
1227
1393
|
| Replay off (`BIR_REPLAY=0`, `bir replay off`, or never installed with `--replay`) | Exactly v1: stop recording on a match, do nothing else, and no prompt waits for the match |
|
|
1228
1394
|
|
|
@@ -1291,6 +1457,11 @@ POST /proxy/result { workId: string, result?: unknown, error?: string }
|
|
|
1291
1457
|
→ 200 { ok: true }
|
|
1292
1458
|
```
|
|
1293
1459
|
|
|
1460
|
+
The hook routes ([BaseInstRunner.md](BaseInstRunner.md), Phase 5) carry more for a host that is not
|
|
1461
|
+
Claude Code: optional fields that its adapter adds to `/session/prompt`, `/tool/pre`, `/tool/post`
|
|
1462
|
+
and `/session/stop`, and `hostFields` in `/health`. §7.4 names them, and says what they mean for a
|
|
1463
|
+
plan.
|
|
1464
|
+
|
|
1294
1465
|
### 16.2 The direct executor
|
|
1295
1466
|
|
|
1296
1467
|
```ts
|
|
@@ -1393,7 +1564,11 @@ same parameters from the same prompt:
|
|
|
1393
1564
|
live call, recent results) and a bearer token, `derived` as the service answered. This is the
|
|
1394
1565
|
ordinary path: nearly every session runs inside Claude Code on a subscription and has no key of
|
|
1395
1566
|
its own (segmented.md R-PARAM-5). The answer is re-normalised here against this runner's own copy
|
|
1396
|
-
of the schema, so the gate reads a `missing` computed from what will actually run.
|
|
1567
|
+
of the schema, so the gate reads a `missing` computed from what will actually run. The service as
|
|
1568
|
+
built (not deployed yet) cuts the prompt it reads new values from at 20,000 characters, and keeps
|
|
1569
|
+
the last five tool results. A plan started at the prompt sends no tool results, in any host: the
|
|
1570
|
+
transcript does not hold the new prompt yet, so its results would be an earlier prompt's, read as
|
|
1571
|
+
this task's (segmented.md R-PARAM-2a). A plan started at a tool call sends this prompt's.
|
|
1397
1572
|
- **Neither → return every key's recorded `sampleValue`**, `costUsd: 0`, `derived: false`. Settings
|
|
1398
1573
|
stand on their samples; a scenario with a target declines rather than run on a stale one.
|
|
1399
1574
|
- With an `apiKey`, `POST https://api.anthropic.com/v1/messages` with the same extraction prompt,
|
|
@@ -1523,7 +1698,9 @@ is refused.
|
|
|
1523
1698
|
`bir` invocations (and a plain `cd`), are host housekeeping, like `ToolSearch`
|
|
1524
1699
|
([housekeeping.ts](../src/record/housekeeping.ts)). They are never recorded as steps, and never
|
|
1525
1700
|
count as divergence from a plan. A session that fixes a plan must not become a plan that edits
|
|
1526
|
-
plans when it is replayed.
|
|
1701
|
+
plans when it is replayed. In a session of another host (§7.4) only the `bir` server's own tools
|
|
1702
|
+
are housekeeping: a tool the application names `Bash`, `ToolSearch` or `TodoWrite` is its own work,
|
|
1703
|
+
and is recorded like any other. The operator's walk-through is
|
|
1527
1704
|
[calculatedReplayGuide.md](calculatedReplayGuide.md) §9.2.
|
|
1528
1705
|
|
|
1529
1706
|
`--dry` maps to BaseIn's `POST /scenarios/:id/replay`, which evaluates the stored logic against the
|