@basein/runner 0.3.5 → 0.3.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -4
- package/dist/bin/bir-hooks.d.ts +0 -1
- package/dist/bin/bir-hooks.js +0 -1
- package/dist/bin/bir.js +2 -4
- package/dist/bin/investigate.js +4 -2
- package/dist/config/generate.d.ts +0 -2
- package/dist/control/recorder-main.d.ts +1 -1
- package/dist/control/recorder-main.js +4 -14
- package/dist/control/server.js +1 -2
- package/dist/replay/controller.d.ts +1 -10
- package/dist/replay/controller.js +8 -20
- package/docs/calculatedReplay.md +13 -9
- package/docs/calculatedReplayGuide.md +20 -24
- package/docs/installRun.md +2 -2
- package/docs/t-bench.md +2 -3
- package/package.json +1 -1
- package/production.md +1 -3
package/README.md
CHANGED
|
@@ -303,9 +303,14 @@ does a plain `bir install` where replay was never set up. `bir install --replay`
|
|
|
303
303
|
never turns back on a `bir replay off` of yours. The switches are kept per
|
|
304
304
|
project in `~/.baseinstrunner/installed.json`, so a recorder the hook starts — with Claude
|
|
305
305
|
Code's environment, not your terminal's — keeps them. A recorder reads them when
|
|
306
|
-
it starts: `bir up --restart` applies a change. `BIR_REPLAY
|
|
307
|
-
`BIR_REPLAY_ALLOW_SERVERS`
|
|
308
|
-
|
|
306
|
+
it starts: `bir up --restart` applies a change. `BIR_REPLAY` and
|
|
307
|
+
`BIR_REPLAY_ALLOW_SERVERS` in the environment still override them, and
|
|
308
|
+
`bir doctor` says which source is in effect.
|
|
309
|
+
|
|
310
|
+
Every match is used. When the service matches a prompt to a recording with a
|
|
311
|
+
ready scenario, bir runs the scenario; there is no second, stricter score on
|
|
312
|
+
this side. (Up to 0.3.6 bir declined a match under 0.92, and
|
|
313
|
+
`BIR_MIN_STEER_SIMILARITY` set that bar; it is ignored now.)
|
|
309
314
|
|
|
310
315
|
Two switches, not one, because of what replay is: **a scenario is a pre-approved
|
|
311
316
|
list of tool calls with computed arguments.** A steered step is answered
|
|
@@ -435,7 +440,6 @@ what to do when they register, so there is one switch and no second copy of it:
|
|
|
435
440
|
| `BIR_REPLAY_ALLOW_SERVERS` | Server keys bir may call unattended: **direct** execution, and a steered plan's recovery. Unset means the list `bir replay allow` stored for the project, else every wrapped server; narrowing it is the recommendation. |
|
|
436
441
|
| `BIR_BUILTIN_TOOLS` | Built-in tools bir may execute itself during a replay — in the session's own project directory, and confined to it. Default `Read,Glob,Grep` (read-only). Add `Edit,Write,Bash` only deliberately: a runner-side step never reaches Claude Code's permission prompts. `none` turns it off, and a built-in step hands the turn back as before. |
|
|
437
442
|
| `BIR_BASH_ALLOW` | Command prefixes a runner-side `Bash` step may run (`*` for all). Unset, `Bash` never executes on the runner, even when `BIR_BUILTIN_TOOLS` lists it. A plan's Bash steps share a working directory, as Claude Code's do (variables and functions do not persist, also as in Claude Code). |
|
|
438
|
-
| `BIR_MIN_STEER_SIMILARITY` | Minimum match similarity to replay (default `0.92`, above the service's own `0.9` detection threshold). |
|
|
439
443
|
| `BIR_DERIVE_API_KEY` | **Optional.** Working out what a new request is about — which fleet, which file, which date — is done for you by the service on its own key, as long as you are signed in. Set this (an Anthropic key) only to keep that reading on your machine, on your key. Signed out *and* unset, a scenario whose values change between requests is declined rather than replayed on stale ones. |
|
|
440
444
|
| `ANTHROPIC_API_KEY` | Not used by the background recorder, which has Claude Code's environment and would bill your own key — unless `BIR_USE_ANTHROPIC_API_KEY=1`. A `bir-hooks` run by hand in a terminal uses that terminal's key. |
|
|
441
445
|
| `BIR_DERIVE_MODEL` | Derivation model, when you set a key of your own (default `claude-haiku-4-5-20251001`). |
|
package/dist/bin/bir-hooks.d.ts
CHANGED
|
@@ -73,7 +73,6 @@
|
|
|
73
73
|
* BIR_BASH_ALLOW command prefixes a runner-side Bash step may run
|
|
74
74
|
* (`*` for all). Unset, Bash never executes on the
|
|
75
75
|
* runner even when BIR_BUILTIN_TOOLS lists it
|
|
76
|
-
* BIR_MIN_STEER_SIMILARITY minimum match similarity to replay (default 0.92)
|
|
77
76
|
* BIR_DERIVE_API_KEY OPTIONAL. Derivation — reading what this turn acts
|
|
78
77
|
* on — is done by the service for a signed-in
|
|
79
78
|
* runner. Set this (an Anthropic key) only to keep
|
package/dist/bin/bir-hooks.js
CHANGED
|
@@ -73,7 +73,6 @@
|
|
|
73
73
|
* BIR_BASH_ALLOW command prefixes a runner-side Bash step may run
|
|
74
74
|
* (`*` for all). Unset, Bash never executes on the
|
|
75
75
|
* runner even when BIR_BUILTIN_TOOLS lists it
|
|
76
|
-
* BIR_MIN_STEER_SIMILARITY minimum match similarity to replay (default 0.92)
|
|
77
76
|
* BIR_DERIVE_API_KEY OPTIONAL. Derivation — reading what this turn acts
|
|
78
77
|
* on — is done by the service for a signed-in
|
|
79
78
|
* runner. Set this (an Anthropic key) only to keep
|
package/dist/bin/bir.js
CHANGED
|
@@ -3292,8 +3292,7 @@ async function doctor(args) {
|
|
|
3292
3292
|
// reading a log file.
|
|
3293
3293
|
const replay = health?.replay;
|
|
3294
3294
|
if (replay?.enabled) {
|
|
3295
|
-
out(`Replay : ON servers=${replay.allowServers?.join(",") || "(all wrapped)"}
|
|
3296
|
-
`minSimilarity=${replay.minSimilarity ?? "?"}` +
|
|
3295
|
+
out(`Replay : ON servers=${replay.allowServers?.join(",") || "(all wrapped)"}` +
|
|
3297
3296
|
(replay.source ? ` source=${replay.source}` : ""));
|
|
3298
3297
|
// An older control server sends only `deriveKey`; read it as the two states
|
|
3299
3298
|
// it could describe then.
|
|
@@ -3527,8 +3526,7 @@ function replayPolicyCommand(args) {
|
|
|
3527
3526
|
? "off (not asked for at install: --no-replay, `bir uninstall --replay`, or no --replay) — `bir replay on` turns it on"
|
|
3528
3527
|
: "off"}`);
|
|
3529
3528
|
out(` direct servers ${policy?.allowServers?.length ? policy.allowServers.join(", ") : "(all wrapped)"}`);
|
|
3530
|
-
|
|
3531
|
-
const overrides = ["BIR_REPLAY", "BIR_REPLAY_ALLOW_SERVERS", "BIR_MIN_STEER_SIMILARITY"].filter((k) => process.env[k] !== undefined);
|
|
3529
|
+
const overrides = ["BIR_REPLAY", "BIR_REPLAY_ALLOW_SERVERS"].filter((k) => process.env[k] !== undefined);
|
|
3532
3530
|
if (overrides.length) {
|
|
3533
3531
|
out(` (set in this shell's environment, which a recorder started from it would prefer: ${overrides.join(", ")})`);
|
|
3534
3532
|
}
|
package/dist/bin/investigate.js
CHANGED
|
@@ -170,9 +170,11 @@ export const DECLINES = {
|
|
|
170
170
|
cause: "the matched recording had no ready scenario at that moment.",
|
|
171
171
|
fix: "wait for the calculation, or start one with `bir scenario calc <runId>`, then run the prompt again.",
|
|
172
172
|
},
|
|
173
|
+
// Written only by runners up to 0.3.6, which re-scored a match against a
|
|
174
|
+
// bar of their own; kept so their log lines still read.
|
|
173
175
|
similarity: {
|
|
174
|
-
cause: "
|
|
175
|
-
fix: "
|
|
176
|
+
cause: "an older runner used a scenario only when the prompt scored 0.92 or more, above the service's own bar for a match (0.9, or a checked match under it).",
|
|
177
|
+
fix: "update the runner (`npm install -g @basein/runner`) and restart it (`bir up --restart`): every match is used now.",
|
|
176
178
|
},
|
|
177
179
|
coverage: {
|
|
178
180
|
cause: "no step of the scenario could run anywhere: its tools are neither wrapped MCP servers registered in this session nor built-ins.",
|
|
@@ -412,8 +412,6 @@ export interface ReplayPolicy {
|
|
|
412
412
|
disabledBy?: "install";
|
|
413
413
|
/** Server keys eligible for direct execution. Absent means every wrapped one. */
|
|
414
414
|
allowServers?: string[];
|
|
415
|
-
/** `BIR_MIN_STEER_SIMILARITY`'s stored twin. */
|
|
416
|
-
minSimilarity?: number;
|
|
417
415
|
}
|
|
418
416
|
/** What `bir install` decided for one project directory. */
|
|
419
417
|
export interface ProjectRecord {
|
|
@@ -52,7 +52,7 @@ export declare function derivationKey(daemon?: boolean): {
|
|
|
52
52
|
apiKey?: string;
|
|
53
53
|
source?: string;
|
|
54
54
|
};
|
|
55
|
-
/** Where the
|
|
55
|
+
/** Where the two replay switches came from, for `/health` and `bir doctor`. */
|
|
56
56
|
export type ReplaySource = "env" | "sidecar" | "default";
|
|
57
57
|
/** What a recorder still serves (K4, `ControlServer.activity()`). */
|
|
58
58
|
export interface RecorderActivity {
|
|
@@ -155,11 +155,9 @@ export function derivationKey(daemon = process.env[DAEMON_ENV] === "1") {
|
|
|
155
155
|
/**
|
|
156
156
|
* Assemble the replay configuration (docs/calculatedReplay.md §5.3 of the guide).
|
|
157
157
|
*
|
|
158
|
-
*
|
|
159
|
-
* (
|
|
160
|
-
*
|
|
161
|
-
* which is a much stronger claim, and a 0.90-similar prompt usually covers only
|
|
162
|
-
* part of the live task.
|
|
158
|
+
* There is no steer threshold: a match the service hands out is steered
|
|
159
|
+
* (controller.ts, gate 4). A recorder started with `BIR_MIN_STEER_SIMILARITY`
|
|
160
|
+
* or `BIR_MIN_SEGMENT_STEER_SIMILARITY` set ignores them.
|
|
163
161
|
*/
|
|
164
162
|
function buildReplayOptions(keeper, policy, key) {
|
|
165
163
|
// The environment still wins — an operator at a terminal is the most
|
|
@@ -168,8 +166,7 @@ function buildReplayOptions(keeper, policy, key) {
|
|
|
168
166
|
// switches the project stored instead of forgetting them (generate.ts).
|
|
169
167
|
const envReplay = process.env.BIR_REPLAY;
|
|
170
168
|
const envAllow = process.env.BIR_REPLAY_ALLOW_SERVERS;
|
|
171
|
-
const
|
|
172
|
-
const fromEnv = envReplay !== undefined || envAllow !== undefined || Number.isFinite(envMin);
|
|
169
|
+
const fromEnv = envReplay !== undefined || envAllow !== undefined;
|
|
173
170
|
const source = fromEnv ? "env" : policy ? "sidecar" : "default";
|
|
174
171
|
const enabled = envReplay !== undefined ? envReplay !== "0" : policy?.enabled !== false;
|
|
175
172
|
const allowServers = envAllow !== undefined
|
|
@@ -177,7 +174,6 @@ function buildReplayOptions(keeper, policy, key) {
|
|
|
177
174
|
: policy?.allowServers
|
|
178
175
|
? parseAllowList(policy.allowServers.join(","))
|
|
179
176
|
: undefined;
|
|
180
|
-
const minSimilarity = Number.isFinite(envMin) ? envMin : (policy?.minSimilarity ?? 0.92);
|
|
181
177
|
// Which built-ins the runner may execute itself (replay/builtins.ts):
|
|
182
178
|
// read-only by default, `BIR_BUILTIN_TOOLS` to change, `BIR_BASH_ALLOW`
|
|
183
179
|
// before a `Bash` step may run at all. Environment-only knobs, like
|
|
@@ -186,7 +182,6 @@ function buildReplayOptions(keeper, policy, key) {
|
|
|
186
182
|
const bashAllow = parseBashAllow(process.env.BIR_BASH_ALLOW);
|
|
187
183
|
const opts = {
|
|
188
184
|
enabled,
|
|
189
|
-
minSimilarity,
|
|
190
185
|
allowServers,
|
|
191
186
|
builtinTools,
|
|
192
187
|
bashAllow,
|
|
@@ -222,10 +217,6 @@ function buildReplayOptions(keeper, policy, key) {
|
|
|
222
217
|
// Observe-only until an operator has read the logs and turned it on
|
|
223
218
|
// (segmented.md R-OUT-10, R-LIFE-7). Arming is never the way to find out.
|
|
224
219
|
segmentArm: process.env.BIR_SEGMENT_ARM === "1",
|
|
225
|
-
minSegmentSimilarity: (() => {
|
|
226
|
-
const n = Number(process.env.BIR_MIN_SEGMENT_STEER_SIMILARITY);
|
|
227
|
-
return Number.isFinite(n) ? n : 0.95;
|
|
228
|
-
})(),
|
|
229
220
|
// Both read live, never captured (C5). The recorder signs in after it
|
|
230
221
|
// listens, refreshes the token as the session outlives it, and may sign
|
|
231
222
|
// in or out while it runs; replay reads whatever holds *now*. The old
|
|
@@ -244,7 +235,6 @@ function buildReplayOptions(keeper, policy, key) {
|
|
|
244
235
|
// and nothing else: an ordinary wrapped call keeps its permission prompt
|
|
245
236
|
// (see the correlation line under `control.listening`).
|
|
246
237
|
logLine("replay.enabled", {
|
|
247
|
-
minSimilarity: opts.minSimilarity,
|
|
248
238
|
allowServers: allowServers ? [...allowServers].join(",") : "(all wrapped)",
|
|
249
239
|
// Which built-ins may run on the runner itself, and whether Bash may at
|
|
250
240
|
// all — loud for the same reason as the rest of this line.
|
package/dist/control/server.js
CHANGED
|
@@ -404,7 +404,7 @@ export class ControlServer {
|
|
|
404
404
|
this.token = opts.token ?? randomBytes(32).toString("hex");
|
|
405
405
|
for (const name of opts.wrappedServers ?? [])
|
|
406
406
|
this.wrapped.add(name);
|
|
407
|
-
this.replay = new ReplayController(opts.replay ?? { enabled: false
|
|
407
|
+
this.replay = new ReplayController(opts.replay ?? { enabled: false });
|
|
408
408
|
this.deriveRecentResults = opts.deriveRecentResults ?? 5;
|
|
409
409
|
this.intentSources = opts.intentSources ?? new Set(["text", "thinking", "tool"]);
|
|
410
410
|
this.timings = { ...DEFAULT_SESSION_TIMINGS, ...(opts.sessionTimings ?? {}) };
|
|
@@ -579,7 +579,6 @@ export class ControlServer {
|
|
|
579
579
|
replay: {
|
|
580
580
|
enabled: this.replay.enabled,
|
|
581
581
|
source: this.opts.replaySource ?? null,
|
|
582
|
-
minSimilarity: this.opts.replay?.minSimilarity ?? null,
|
|
583
582
|
allowServers: this.opts.replay?.allowServers ? [...this.opts.replay.allowServers] : null,
|
|
584
583
|
deriveKey: Boolean(this.opts.replay?.apiKey ?? process.env.ANTHROPIC_API_KEY),
|
|
585
584
|
// Who reads the turn for its parameters (segmented.md R-PARAM-5). The
|
|
@@ -180,15 +180,6 @@ export declare const RUN_NO_PLAN = "no_plan";
|
|
|
180
180
|
export interface ReplayOptions {
|
|
181
181
|
/** On unless `BIR_REPLAY=0`. Everything here is inert when false. */
|
|
182
182
|
enabled: boolean;
|
|
183
|
-
/** Minimum similarity to steer — deliberately above the service's detection threshold. */
|
|
184
|
-
minSimilarity: number;
|
|
185
|
-
/**
|
|
186
|
-
* Minimum similarity to steer a **segment** (`BIR_MIN_SEGMENT_STEER_SIMILARITY`,
|
|
187
|
-
* segmented.md R-OUT-11). The same number as the server's clear band, so a
|
|
188
|
-
* clear hand-out is always accepted and no score falls between "clear" and
|
|
189
|
-
* "accepted".
|
|
190
|
-
*/
|
|
191
|
-
minSegmentSimilarity?: number;
|
|
192
183
|
/**
|
|
193
184
|
* `BIR_SEGMENT_ARM`. Off means observe-only: the runner asks for no segment
|
|
194
185
|
* and arms none, and only logs what would have armed (R-OUT-10).
|
|
@@ -439,7 +430,7 @@ export interface ReplayState {
|
|
|
439
430
|
* Why a match did not arm. One code per gate of the ladder, so the caller can
|
|
440
431
|
* act on a decline without parsing the sentence that explains it.
|
|
441
432
|
*/
|
|
442
|
-
export type DeclineCode = "replay_disabled" | "not_ready" | "flatten_failed" | "
|
|
433
|
+
export type DeclineCode = "replay_disabled" | "not_ready" | "flatten_failed" | "known_bad_first_step" | "nondeterministic_first_step" | "unusable_first_step" | "coverage" | "missing_target" | "no_derive_key";
|
|
443
434
|
/** What `PreToolUse` should do about this call. */
|
|
444
435
|
export type PreToolAction =
|
|
445
436
|
/** Pin the arguments and let the real tool run. */
|
|
@@ -333,14 +333,7 @@ export class ReplayController {
|
|
|
333
333
|
verified: match.verified,
|
|
334
334
|
sharedWith: [],
|
|
335
335
|
};
|
|
336
|
-
// A segment is judged against its own gate, and a `same` verdict stands in
|
|
337
|
-
// for the score (segmented.md R-OUT-11): the question was asked *because*
|
|
338
|
-
// the score sat below the threshold, so a numeric gate would decline every
|
|
339
|
-
// verified hand-out.
|
|
340
336
|
const isSegment = state.kind === "segment";
|
|
341
|
-
const threshold = isSegment
|
|
342
|
-
? (this.opts.minSegmentSimilarity ?? 0.95)
|
|
343
|
-
: this.opts.minSimilarity;
|
|
344
337
|
const decline = (why, code) => {
|
|
345
338
|
state.declined = code;
|
|
346
339
|
logLine("replay.decision", {
|
|
@@ -349,7 +342,6 @@ export class ReplayController {
|
|
|
349
342
|
run: match.runId,
|
|
350
343
|
scenario: match.scenarioId ?? undefined,
|
|
351
344
|
similarity: match.similarity.toFixed(3),
|
|
352
|
-
threshold,
|
|
353
345
|
kind: state.kind,
|
|
354
346
|
key: state.key,
|
|
355
347
|
verified: state.verified?.verdict,
|
|
@@ -371,18 +363,14 @@ export class ReplayController {
|
|
|
371
363
|
if (!match.scenarioId || !isReadyScenario(scenario)) {
|
|
372
364
|
return decline(match.scenarioId ? "scenario is not ready" : "no scenario for the matched run", "not_ready");
|
|
373
365
|
}
|
|
374
|
-
// Gate 4 — similarity.
|
|
375
|
-
//
|
|
376
|
-
//
|
|
377
|
-
//
|
|
378
|
-
//
|
|
379
|
-
//
|
|
380
|
-
//
|
|
381
|
-
|
|
382
|
-
const verifiedOk = isSegment && match.verified?.verdict === "same";
|
|
383
|
-
if (!scoreOk && !verifiedOk) {
|
|
384
|
-
return decline("similarity below threshold", "similarity");
|
|
385
|
-
}
|
|
366
|
+
// Gate 4 — similarity — is gone. A hand-out is the service's decision that
|
|
367
|
+
// this scenario does what the prompt asks: its match threshold, the
|
|
368
|
+
// same-request question under it, the intent pass, a segment's clear band
|
|
369
|
+
// or `same` verdict. Scoring it again here declined matches the service
|
|
370
|
+
// had made (a checked 0.85, a task line at 0.91), so every match now
|
|
371
|
+
// steers. What a plan cannot do — a step that keeps failing, a tool
|
|
372
|
+
// nobody wrapped, a target the prompt never named — the gates below
|
|
373
|
+
// still stop.
|
|
386
374
|
// Flatten the chain before the remaining gates, so every one of them judges
|
|
387
375
|
// the steps that will *really* run — the caller's own and, in order, those
|
|
388
376
|
// of every segment it calls (segmented.md R-CALL-29, R-CALL-30).
|
package/docs/calculatedReplay.md
CHANGED
|
@@ -193,7 +193,7 @@ Every gate that declines logs **one** `replay.decision` line carrying the reason
|
|
|
193
193
|
| 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched) |
|
|
194
194
|
| 2 | **Replay enabled** | `BIR_REPLAY === "0"`, or, with it unset, the project's stored switch is off (`bir replay off`, or a `bir install` without `--replay`) | `not_steered` |
|
|
195
195
|
| 3 | **Scenario ready** | `scenario == null`, `state !== "ready"`, or `steps.length === 0` | `not_steered` |
|
|
196
|
-
| 4 |
|
|
196
|
+
| 4 | ~~**Similarity**~~ | Retired: a match is the decision to use the scenario (below) | — |
|
|
197
197
|
| 5 | **Tool coverage** | No step is executable anywhere (§5 → `mode: "none"`) | `not_steered` |
|
|
198
198
|
|
|
199
199
|
Past the table, a plan can still be refused before the model hears of it, each refusal with its own
|
|
@@ -205,12 +205,16 @@ after the prompt was answered is retired unannounced (`match_late`, `arm_unannou
|
|
|
205
205
|
plan is declined when no running `bir-scenario` has said it serves `run_scenario`
|
|
206
206
|
(`no_delivery_tool`, §6.3). In plan mode (`permission_mode: "plan"`) nothing is announced at all.
|
|
207
207
|
|
|
208
|
-
Gate 4
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
208
|
+
Gate 4 is retired. Up to 0.3.6 it declined a match under `BIR_MIN_STEER_SIMILARITY` (0.92),
|
|
209
|
+
above the server's 0.9, on the argument that the server only decided *"do not record this again"*
|
|
210
|
+
while steering claims *"do not think about this again."* The server's match stopped being detection
|
|
211
|
+
alone: it hands a scenario over only when it may run, with a ticket, and it decides the match with
|
|
212
|
+
more than one score — a same-request question under its threshold, and a second pass on the
|
|
213
|
+
prompt's task line. Re-scoring that on this side declined matches the server had made (a checked
|
|
214
|
+
0.85, a task line at 0.91). **A match is the decision to use the scenario**, so every hand-out
|
|
215
|
+
steers; what a plan cannot do — a parked first step, a tool nothing reaches, a target the prompt
|
|
216
|
+
never named — the gates after it still stop. `BIR_MIN_STEER_SIMILARITY` and
|
|
217
|
+
`BIR_MIN_SEGMENT_STEER_SIMILARITY` are ignored.
|
|
214
218
|
|
|
215
219
|
`not_steered` as the floor is not a failure code. It is the **control group**: the prompt matched,
|
|
216
220
|
the agent ran the task the ordinary way, and the cost of doing so is a fresh measurement of what this
|
|
@@ -1486,7 +1490,7 @@ plans when it is replayed. The operator's walk-through is
|
|
|
1486
1490
|
source run's *recorded* outputs — no real tools, no side effects, one Haiku call. Without `--dry`,
|
|
1487
1491
|
`bir replay` runs the same plan through the same executor against the live proxies. `bir doctor`
|
|
1488
1492
|
gains a `replay` block: on/off, the effective `BIR_REPLAY_ALLOW_SERVERS` allowlist,
|
|
1489
|
-
|
|
1493
|
+
and who derives — this machine, the service, or nobody, which is the
|
|
1490
1494
|
state in which a scenario with a target never runs (§13.2, mitigation 3).
|
|
1491
1495
|
|
|
1492
1496
|
---
|
|
@@ -1627,7 +1631,7 @@ changes the corpus, the threshold and every stored vector. §20 question 1.
|
|
|
1627
1631
|
and argument objects the bodies rely on today.
|
|
1628
1632
|
4. **Should a declined match still arm a *dry* plan?** Deriving parameters and computing the first
|
|
1629
1633
|
step's input without executing anything would make `bir doctor` able to say *"this scenario would
|
|
1630
|
-
have run these three calls"* — excellent for
|
|
1634
|
+
have run these three calls"* — excellent for fixing the plan behind a decline, at the price of a
|
|
1631
1635
|
Haiku call on turns that replay nothing.
|
|
1632
1636
|
5. **Should `scenario_steps` carry `recorded_output`?** §8.1 works around its absence with one lazy
|
|
1633
1637
|
`GET /recordings/runs/:runId`. A column would make it a field read, would let a scenario outlive
|
|
@@ -300,7 +300,7 @@ bir doctor
|
|
|
300
300
|
```
|
|
301
301
|
|
|
302
302
|
```
|
|
303
|
-
Replay : ON servers=chrome-devtools,fleet-api
|
|
303
|
+
Replay : ON servers=chrome-devtools,fleet-api source=sidecar
|
|
304
304
|
derive=the service (no key needed here)
|
|
305
305
|
Wrapped in config : chrome-devtools, fleet-api
|
|
306
306
|
Registered proxies: chrome-devtools, fleet-api
|
|
@@ -324,7 +324,7 @@ uses, with the account it is signed in as. The two normally agree.
|
|
|
324
324
|
|
|
325
325
|
### 5.3 Environment
|
|
326
326
|
|
|
327
|
-
The first
|
|
327
|
+
The first two switches are stored per project, and that is where to set them:
|
|
328
328
|
|
|
329
329
|
```bash
|
|
330
330
|
bir replay allow chrome-devtools,fleet-api # the servers bir may call by itself; `allow all` clears it
|
|
@@ -344,7 +344,6 @@ so there is exactly one switch and it cannot get out of step with itself.
|
|
|
344
344
|
|---|---|---|
|
|
345
345
|
| `BIR_REPLAY` | *(the stored switch)* | `0` disables replay and any other value turns it on, whatever `bir replay on` or `off` or the install stored. Unset, the stored switch decides: on after `bir setup` or `bir install --replay`, off after a plain `bir install`. Nothing below matters while replay is off |
|
|
346
346
|
| `BIR_REPLAY_ALLOW_SERVERS` | *(the stored list, else all wrapped)* | Comma-separated server keys bir may call by itself: **direct** execution and divergence recovery. Overrides `bir replay allow` |
|
|
347
|
-
| `BIR_MIN_STEER_SIMILARITY` | `0.92` | Below this a match is detected but not replayed (§8) |
|
|
348
347
|
| `BIR_DERIVE_API_KEY` | *(unset)* | **Not required.** Derivation — reading what this request acts on — is done by the service on its key for a signed-in runner. Set this to keep the reading on this machine instead: the prompt then never leaves it, and it is one round trip faster. Signed out *and* unset, only a scenario with nothing to work out replays. A recorder running in the background (started by the SessionStart hook, `bir up` or `bir setup`) ignores `ANTHROPIC_API_KEY` (usually Claude Code's own, and billed) unless `BIR_USE_ANTHROPIC_API_KEY=1`; a `bir-hooks` you run in a terminal uses it |
|
|
349
348
|
| `BIR_DERIVE_MODEL` | `claude-haiku-4-5-20251001` | The derivation model, when this machine does the reading |
|
|
350
349
|
| `BIR_MATCH_BUDGET_MS` | `3000` | How long the prompt hook waits for a match before giving up. After a match slower than `BIR_MATCH_SLOW_MS` (`2000`), prompts stop waiting for one for 5 minutes, doubling up to 30 while the service stays slow |
|
|
@@ -515,8 +514,8 @@ a turn cut short says nothing about what the task costs.
|
|
|
515
514
|
|
|
516
515
|
`not_steered` is not an error, and it is not wasted. A saving is `baseline − actual`, and the baseline
|
|
517
516
|
is *what this task costs when the agent does it*. Every declined match is a fresh measurement of
|
|
518
|
-
exactly that. A deployment that
|
|
519
|
-
|
|
517
|
+
exactly that. A deployment that seldom declines measures its baseline mostly from the recording
|
|
518
|
+
itself; `bir replay off` for a few runs of the prompt, then `bir replay on`, gives it more.
|
|
520
519
|
|
|
521
520
|
An **unredeemed** ticket books nothing at all. Silence loses a saving; it never invents one.
|
|
522
521
|
|
|
@@ -524,25 +523,22 @@ An **unredeemed** ticket books nothing at all. Silence loses a saving; it never
|
|
|
524
523
|
|
|
525
524
|
## 8. Tuning
|
|
526
525
|
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
If a specific pair of prompts *should* match and does not, the fix is usually upstream of the
|
|
544
|
-
threshold: the two prompts genuinely mean different things to the embedding model. Re-record with a
|
|
545
|
-
first prompt that states the task plainly.
|
|
526
|
+
There is no threshold on this side. **Every match steers**: a match is the service's decision that
|
|
527
|
+
the scenario does what the prompt asks, and bir does not score it again. (Up to 0.3.6 it did:
|
|
528
|
+
`BIR_MIN_STEER_SIMILARITY`, default 0.92, declined matches the service had made. It is ignored now.)
|
|
529
|
+
|
|
530
|
+
Whether a prompt matches is decided by the service, in three steps. A prompt at 0.9 or more against
|
|
531
|
+
a recording matches (`SIMILARITY_THRESHOLD`). From 0.7 (`SIMILARITY_CHECK_FLOOR`) up to that, a quick
|
|
532
|
+
model check asks whether the two prompts are the same request, values included, and only a yes
|
|
533
|
+
matches. A prompt that missed both is matched once more on its task line (`PROMPT_INTENT_THRESHOLD`,
|
|
534
|
+
0.9) — the way the same task with a different value finds its scenario.
|
|
535
|
+
|
|
536
|
+
- A replay that answered the wrong question → switch that recording off in the console (its
|
|
537
|
+
scenario then never runs), or `bir replay off` in the project, and re-read the scenario's
|
|
538
|
+
`intent`. On a service you run yourself, raise `SIMILARITY_THRESHOLD` or `SIMILARITY_CHECK_FLOOR`.
|
|
539
|
+
- A prompt you *would* have wanted replayed that did not match → the fix is usually upstream of any
|
|
540
|
+
threshold: the two prompts genuinely mean different things to the embedding model. Re-record with
|
|
541
|
+
a first prompt that states the task plainly.
|
|
546
542
|
|
|
547
543
|
---
|
|
548
544
|
|
package/docs/installRun.md
CHANGED
|
@@ -571,8 +571,8 @@ Fleet switches on `bir setup`: `--allow-servers a,b` stores the servers replay
|
|
|
571
571
|
may call unattended for this project; `bir replay allow|on|off|status` changes
|
|
572
572
|
it later. Both live in the project record (§4.3), so a recorder the hook starts
|
|
573
573
|
— with Claude Code's environment, not the operator's terminal — keeps them;
|
|
574
|
-
`BIR_REPLAY
|
|
575
|
-
|
|
574
|
+
`BIR_REPLAY` and `BIR_REPLAY_ALLOW_SERVERS` in the recorder's environment still
|
|
575
|
+
override, and `bir doctor` says which source is in
|
|
576
576
|
effect. `--no-replay` adds no scenario server (one an earlier setup added stays
|
|
577
577
|
until `bir uninstall --replay`) and stores replay off (`bir replay on` turns it
|
|
578
578
|
on); `--no-daemon` wires everything and starts
|
package/docs/t-bench.md
CHANGED
|
@@ -357,9 +357,8 @@ Per arm, per domain: reward (mean), pass^k across trials, `$`/episode and `$`/tu
|
|
|
357
357
|
`--output-format json`'s `total_cost_usd`, wall-clock per episode, and turn count.
|
|
358
358
|
|
|
359
359
|
Per replayed turn, from the audit log and the execution ledger: outcome
|
|
360
|
-
(`steered_full` / `diverged` / `fell_back` / `not_steered` / `failed`), match similarity
|
|
361
|
-
|
|
362
|
-
`sessionCostUsd`, and the per-step verdicts.
|
|
360
|
+
(`steered_full` / `diverged` / `fell_back` / `not_steered` / `failed`), match similarity, direct
|
|
361
|
+
vs steered mode, `deriveCostUsd` against `sessionCostUsd`, and the per-step verdicts.
|
|
363
362
|
|
|
364
363
|
Two τ-specific splits worth reporting that MCPMark cannot give us:
|
|
365
364
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@basein/runner",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.7",
|
|
4
4
|
"description": "A recording MCP proxy: sits between any MCP client and its MCP servers, executes each call on the client's behalf, and records the run as a reusable BaseIn scenario.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/production.md
CHANGED
|
@@ -847,7 +847,7 @@ Truncation is always marked with the original size.
|
|
|
847
847
|
| `matchGate` | `{holding, holdMs, until}`: whether prompts have stopped waiting on the match because the service is slow |
|
|
848
848
|
| `correlation` | `auto`, `allow` or `ask`, or `fingerprint` when correlation is off: who approves wrapped calls |
|
|
849
849
|
| `deliveryTool`, `scenarioServers[]` | whether a live `bir-scenario` has said hello (`{pid, tools, off, version, ageMs}`). An empty list means no direct plan is announced. |
|
|
850
|
-
| `replay` | `{enabled, source,
|
|
850
|
+
| `replay` | `{enabled, source, allowServers, deriveKey, deriveVia, pollingProxies}`. `deriveVia` is `service`, `key` or `samples`; `samples` means no scenario with a target can run. |
|
|
851
851
|
| `registeredProxies[]`, `wrappedServers` | proxies that registered (`serverName`, `pid`, `cwd`, `version`) and the wrapped server names |
|
|
852
852
|
| `hooksVersion`, `pid`, `cwd`, `url` | the recorder's version and identity |
|
|
853
853
|
| `segmentArm` | whether service-found sub-tasks may run mid-task (default: observe only) |
|
|
@@ -981,7 +981,6 @@ Defaults were checked in `src/`.
|
|
|
981
981
|
| `BIR_DERIVE_MODEL` | recorder | `claude-haiku-4-5-20251001` | Model for derivation on this machine. |
|
|
982
982
|
| `BIR_REPLAY` | recorder | on, or the project's stored switch | `0` turns replay off. Any other value forces it on over the stored switch. |
|
|
983
983
|
| `BIR_REPLAY_ALLOW_SERVERS` | recorder | all wrapped servers | Servers that direct runs and divergence recovery may call unattended (comma-separated). |
|
|
984
|
-
| `BIR_MIN_STEER_SIMILARITY` | recorder | 0.92 | Minimum match similarity to replay. |
|
|
985
984
|
| `BIR_MATCH_BUDGET_MS` | recorder | 3000 | Longest a prompt waits for the match. |
|
|
986
985
|
| `BIR_MATCH_SLOW_MS` | recorder | 2000 | A match slower than this, or one that timed out, stops prompts waiting for 5 min, doubling to 30 min. |
|
|
987
986
|
| `BIR_DERIVE_BUDGET_MS` | recorder | 8000 | Derivation wait for `/scenario/run`. Hooks use the smaller of this and `BIR_HOOK_DERIVE_BUDGET_MS`. |
|
|
@@ -1003,7 +1002,6 @@ Defaults were checked in `src/`.
|
|
|
1003
1002
|
| `BIR_INTENT_ARM` | recorder | unset (follows `BIR_SEGMENT_ARM`) | `1`: a probe may arm a plan in its own PreToolUse, and that hook waits for it. `0`: never. |
|
|
1004
1003
|
| `BIR_INTENT_ARM_BUDGET_MS` | recorder | 500 | How long that PreToolUse waits for the probe. |
|
|
1005
1004
|
| `BIR_SEGMENT_ARM` | recorder | unset (observe only) | `1`: recurring sub-tasks the service found may run mid-task. |
|
|
1006
|
-
| `BIR_MIN_SEGMENT_STEER_SIMILARITY` | recorder | 0.95 | Minimum similarity to steer a segment. |
|
|
1007
1005
|
| `BIR_CONTROL_URL` | proxy, scenario | written into wrapped entries | The project's control server. Together with `BIR_CONTROL_TOKEN` it is the SDK channel and wins over discovery. |
|
|
1008
1006
|
| `BIR_CONTROL_TOKEN` | proxy, scenario | unset | Bearer token for the SDK channel. |
|
|
1009
1007
|
| `BIR_PROXY_STAY_UP` | proxy | unset | `1`: keep answering errors after the upstream dies (0.2.12). |
|