@basein/runner 0.3.7 → 0.3.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -6
- package/dist/bin/bir-hooks.d.ts +8 -8
- package/dist/bin/bir-hooks.js +8 -8
- package/dist/bin/investigate.js +20 -6
- package/dist/control/recorder-main.js +4 -4
- package/dist/control/server.js +7 -4
- package/dist/control/transcript.d.ts +9 -0
- package/dist/control/transcript.js +26 -7
- package/dist/record/remote-recorder.d.ts +9 -1
- package/dist/record/remote-recorder.js +77 -10
- package/dist/record/truncate.d.ts +11 -0
- package/dist/record/truncate.js +11 -0
- package/dist/replay/builtins.d.ts +21 -16
- package/dist/replay/builtins.js +38 -21
- package/dist/replay/controller.d.ts +22 -9
- package/dist/replay/controller.js +103 -12
- package/dist/replay/coverage.d.ts +8 -7
- package/dist/replay/coverage.js +8 -7
- package/dist/replay/plan.d.ts +6 -0
- package/dist/replay/plan.js +13 -3
- package/dist/replay/types.d.ts +16 -0
- package/docs/calculatedReplay.md +48 -8
- package/docs/calculatedReplayGuide.md +11 -5
- package/package.json +1 -1
- package/production.md +3 -2
package/README.md
CHANGED
|
@@ -319,15 +319,19 @@ prompt you would have seen — only a call that plausibly is the step, though (t
|
|
|
319
319
|
same command, the same file, the same kind of arguments); any other call runs as
|
|
320
320
|
written and is asked about as usual. A *directly executed* step never reaches the
|
|
321
321
|
permission system at all — it is dispatched to a `bir-proxy` and called on a
|
|
322
|
-
connection that is already open
|
|
322
|
+
connection that is already open, or, for Claude Code's own `Read`, `Glob`,
|
|
323
|
+
`Grep`, `Edit`, `Write` and `Bash`, run by the recorder itself in the project
|
|
324
|
+
folder — and `run_scenario`, the call that starts the
|
|
323
325
|
run, is answered `allow` while the recorder has announced the plan. In plan mode
|
|
324
326
|
nothing is announced and nothing is approved. Set `BIR_REPLAY_ALLOW_SERVERS` to
|
|
325
327
|
the servers you are content to have called unattended (it also covers a steered
|
|
326
|
-
plan's recovery), and
|
|
328
|
+
plan's recovery), `BIR_BUILTIN_TOOLS` and `BIR_BASH_ALLOW` to the built-ins and
|
|
329
|
+
commands, and read [the guide's §5.1](docs/calculatedReplayGuide.md)
|
|
327
330
|
before enabling it anywhere that matters.
|
|
328
331
|
|
|
329
|
-
What it buys: on a scenario whose every step is a wrapped MCP tool
|
|
330
|
-
sequence runs through connections the proxies already
|
|
332
|
+
What it buys: on a scenario whose every step is a wrapped MCP tool or one of
|
|
333
|
+
those built-ins, the whole sequence runs through connections the proxies already
|
|
334
|
+
hold, or on the recorder itself — **zero model tokens**
|
|
331
335
|
— and the model's only job is one tool call that reads the results. Such a plan
|
|
332
336
|
is announced only while a `bir` server has told the recorder it is connected,
|
|
333
337
|
and runs only through `run_scenario`: if the model calls anything else first,
|
|
@@ -438,8 +442,8 @@ what to do when they register, so there is one switch and no second copy of it:
|
|
|
438
442
|
|---|---|
|
|
439
443
|
| `BIR_REPLAY` | `0` turns replay off and any other value on, over the project's stored switch (`bir replay on` / `off`; a first plain `bir install` stores off). Nothing below matters while it is off. |
|
|
440
444
|
| `BIR_REPLAY_ALLOW_SERVERS` | Server keys bir may call unattended: **direct** execution, and a steered plan's recovery. Unset means the list `bir replay allow` stored for the project, else every wrapped server; narrowing it is the recommendation. |
|
|
441
|
-
| `BIR_BUILTIN_TOOLS` | Built-in tools bir may execute itself during a replay — in the session's own project directory, and confined to it. Default `Read,Glob,Grep` (read-only
|
|
442
|
-
| `BIR_BASH_ALLOW` | Command prefixes a runner-side `Bash` step may run (`*`
|
|
445
|
+
| `BIR_BUILTIN_TOOLS` | Built-in tools bir may execute itself during a replay — in the session's own project directory, and confined to it. Default all six: `Read,Glob,Grep,Edit,Write,Bash` (up to 0.3.7 only the read-only `Read,Glob,Grep`). A runner-side step never reaches Claude Code's permission prompts, so leave off the list any tool you want the model to call itself. `none` turns it off, and built-in steps are steered in the session. |
|
|
446
|
+
| `BIR_BASH_ALLOW` | Command prefixes a runner-side `Bash` step may run. Unset means any command (`*`); a list such as `git,npm test` narrows it; `none` allows no command, and `Bash` steps are then steered in the session. A plan's Bash steps share a working directory, as Claude Code's do (variables and functions do not persist, also as in Claude Code). |
|
|
443
447
|
| `BIR_DERIVE_API_KEY` | **Optional.** Working out what a new request is about — which fleet, which file, which date — is done for you by the service on its own key, as long as you are signed in. Set this (an Anthropic key) only to keep that reading on your machine, on your key. Signed out *and* unset, a scenario whose values change between requests is declined rather than replayed on stale ones. |
|
|
444
448
|
| `ANTHROPIC_API_KEY` | Not used by the background recorder, which has Claude Code's environment and would bill your own key — unless `BIR_USE_ANTHROPIC_API_KEY=1`. A `bir-hooks` run by hand in a terminal uses that terminal's key. |
|
|
445
449
|
| `BIR_DERIVE_MODEL` | Derivation model, when you set a key of your own (default `claude-haiku-4-5-20251001`). |
|
package/dist/bin/bir-hooks.d.ts
CHANGED
|
@@ -64,15 +64,15 @@
|
|
|
64
64
|
* BIR_BUILTIN_TOOLS comma-separated built-ins the runner may execute
|
|
65
65
|
* itself during a replay, in the session's own
|
|
66
66
|
* project directory and confined to it
|
|
67
|
-
* (replay/builtins.ts). Default
|
|
68
|
-
*
|
|
69
|
-
*
|
|
70
|
-
*
|
|
71
|
-
*
|
|
72
|
-
*
|
|
67
|
+
* (replay/builtins.ts). Default all six:
|
|
68
|
+
* Read,Glob,Grep,Edit,Write,Bash. A runner-side
|
|
69
|
+
* step never reaches the host's permission
|
|
70
|
+
* prompts; Read,Glob,Grep is the read-only set
|
|
71
|
+
* (the default up to 0.3.7). `none` turns
|
|
72
|
+
* built-ins off, and their steps are steered
|
|
73
73
|
* BIR_BASH_ALLOW command prefixes a runner-side Bash step may run
|
|
74
|
-
* (`*` for all). Unset
|
|
75
|
-
*
|
|
74
|
+
* (`*` for all). Unset means `*`; `none` means no
|
|
75
|
+
* command, and Bash steps are steered
|
|
76
76
|
* BIR_DERIVE_API_KEY OPTIONAL. Derivation — reading what this turn acts
|
|
77
77
|
* on — is done by the service for a signed-in
|
|
78
78
|
* runner. Set this (an Anthropic key) only to keep
|
package/dist/bin/bir-hooks.js
CHANGED
|
@@ -64,15 +64,15 @@
|
|
|
64
64
|
* BIR_BUILTIN_TOOLS comma-separated built-ins the runner may execute
|
|
65
65
|
* itself during a replay, in the session's own
|
|
66
66
|
* project directory and confined to it
|
|
67
|
-
* (replay/builtins.ts). Default
|
|
68
|
-
*
|
|
69
|
-
*
|
|
70
|
-
*
|
|
71
|
-
*
|
|
72
|
-
*
|
|
67
|
+
* (replay/builtins.ts). Default all six:
|
|
68
|
+
* Read,Glob,Grep,Edit,Write,Bash. A runner-side
|
|
69
|
+
* step never reaches the host's permission
|
|
70
|
+
* prompts; Read,Glob,Grep is the read-only set
|
|
71
|
+
* (the default up to 0.3.7). `none` turns
|
|
72
|
+
* built-ins off, and their steps are steered
|
|
73
73
|
* BIR_BASH_ALLOW command prefixes a runner-side Bash step may run
|
|
74
|
-
* (`*` for all). Unset
|
|
75
|
-
*
|
|
74
|
+
* (`*` for all). Unset means `*`; `none` means no
|
|
75
|
+
* command, and Bash steps are steered
|
|
76
76
|
* BIR_DERIVE_API_KEY OPTIONAL. Derivation — reading what this turn acts
|
|
77
77
|
* on — is done by the service for a signed-in
|
|
78
78
|
* runner. Set this (an Anthropic key) only to keep
|
package/dist/bin/investigate.js
CHANGED
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
* declined and what mode armed; the service knows the baseline, the ledger and
|
|
18
18
|
* whether a run ever became a recording. Both lists are printed, worst first.
|
|
19
19
|
*/
|
|
20
|
+
import { EXECUTABLE_BUILTINS } from "../replay/builtins.js";
|
|
20
21
|
import { ReadClock, jsonText, readWithin, serviceProblem, underMcp, unreachable } from "./service-read.js";
|
|
21
22
|
const num = (v) => {
|
|
22
23
|
if (v === undefined || v === null || v === "")
|
|
@@ -267,12 +268,25 @@ export function localFindings(turn) {
|
|
|
267
268
|
.filter((t) => Boolean(t));
|
|
268
269
|
if (turn.plan.mode === "steer" && live.length > 0) {
|
|
269
270
|
const lowSaving = savedPct(turn) != null && savedPct(turn) < LOW_SAVING_PCT;
|
|
271
|
+
const names = [...new Set(live)];
|
|
272
|
+
// One of the runner's own built-ins steered only because it was off the
|
|
273
|
+
// list (or the turn ran on a runner older than 0.3.8); anything else can
|
|
274
|
+
// run directly only behind a wrapped MCP server.
|
|
275
|
+
const offList = names.filter((n) => EXECUTABLE_BUILTINS.has(n));
|
|
276
|
+
const elsewhere = names.filter((n) => !EXECUTABLE_BUILTINS.has(n));
|
|
277
|
+
const fixes = [];
|
|
278
|
+
if (offList.length > 0) {
|
|
279
|
+
fixes.push(`The runner runs ${offList.join(", ")} itself when ${offList.length === 1 ? "it is" : "they are"} on \`BIR_BUILTIN_TOOLS\` (unset means all six, from runner 0.3.8) and, for Bash, \`BIR_BASH_ALLOW\` is not \`none\`: fix those where the recorder starts, then \`bir up --restart\`.`);
|
|
280
|
+
}
|
|
281
|
+
if (elsewhere.length > 0) {
|
|
282
|
+
fixes.push(`Put the data ${elsewhere.join(", ")} ${elsewhere.length === 1 ? "reaches" : "reach"} behind an MCP server, wrap it (\`bir install --server <name>\`), and record the prompt again so its steps are MCP calls.`);
|
|
283
|
+
}
|
|
270
284
|
out.push({
|
|
271
285
|
code: "steer_mode",
|
|
272
286
|
severity: lowSaving ? "warn" : "info",
|
|
273
|
-
title: `The plan armed in steer mode: ${live.length} of ${coverage.length} steps are reachable only inside the session (${
|
|
274
|
-
cause: "In steer mode the runner pins each call's inputs and auto-approves it, but the model still emits every tool call, reads every result and writes the answer — so the whole live turn is still paid for. Direct mode, where the model makes no tool call, needs every step to be a wrapped MCP server tool.",
|
|
275
|
-
fix: "
|
|
287
|
+
title: `The plan armed in steer mode: ${live.length} of ${coverage.length} steps are reachable only inside the session (${names.join(", ")}).`,
|
|
288
|
+
cause: "In steer mode the runner pins each call's inputs and auto-approves it, but the model still emits every tool call, reads every result and writes the answer — so the whole live turn is still paid for. Direct mode, where the model makes no tool call, needs every step to be a wrapped MCP server tool or a built-in the runner runs itself (Read, Glob, Grep, Edit, Write, Bash).",
|
|
289
|
+
fix: fixes.join(" "),
|
|
276
290
|
evidence: { coverage, tools },
|
|
277
291
|
});
|
|
278
292
|
}
|
|
@@ -280,8 +294,8 @@ export function localFindings(turn) {
|
|
|
280
294
|
out.push({
|
|
281
295
|
code: "direct_mode",
|
|
282
296
|
severity: "info",
|
|
283
|
-
title: `The plan armed in direct mode: ${turn.plan.steps ?? coverage.length} steps ran on the proxies, no model in the loop.`,
|
|
284
|
-
cause: "Every step is a wrapped MCP tool this session has a proxy for.",
|
|
297
|
+
title: `The plan armed in direct mode: ${turn.plan.steps ?? coverage.length} steps ran on the proxies and the runner, no model in the loop.`,
|
|
298
|
+
cause: "Every step is a wrapped MCP tool this session has a proxy for, or a built-in the runner runs itself.",
|
|
285
299
|
fix: "",
|
|
286
300
|
});
|
|
287
301
|
}
|
|
@@ -332,7 +346,7 @@ export function localFindings(turn) {
|
|
|
332
346
|
title: `Saved ${pct(p)}: ${usd(r.savedUsd ?? 0)} of a ${usd((r.savedUsd ?? 0) + cost)} baseline.`,
|
|
333
347
|
cause: `The live session cost ${usd(r.session)} of that baseline${turn.plan?.mode === "steer" ? " — steer mode pays for the whole turn." : "."}`,
|
|
334
348
|
fix: turn.plan?.mode === "steer"
|
|
335
|
-
? "Direct mode is the fix
|
|
349
|
+
? "Direct mode is the fix (see steer_mode for what kept this plan out of it)."
|
|
336
350
|
: "Compare the baseline with a few unsteered runs (BIR_REPLAY=0); a single cheap source run makes any replay look poor.",
|
|
337
351
|
evidence: { ...r, costUsd: cost, savedPct: p },
|
|
338
352
|
});
|
|
@@ -175,9 +175,9 @@ function buildReplayOptions(keeper, policy, key) {
|
|
|
175
175
|
? parseAllowList(policy.allowServers.join(","))
|
|
176
176
|
: undefined;
|
|
177
177
|
// Which built-ins the runner may execute itself (replay/builtins.ts):
|
|
178
|
-
//
|
|
179
|
-
//
|
|
180
|
-
// `BIR_REPLAY_RECORDED_OUTPUTS`.
|
|
178
|
+
// all six by default, `BIR_BUILTIN_TOOLS` to narrow them, `BIR_BASH_ALLOW`
|
|
179
|
+
// to narrow the commands a `Bash` step may run. Environment-only knobs,
|
|
180
|
+
// like `BIR_REPLAY_RECORDED_OUTPUTS`.
|
|
181
181
|
const builtinTools = parseBuiltinTools(process.env.BIR_BUILTIN_TOOLS);
|
|
182
182
|
const bashAllow = parseBashAllow(process.env.BIR_BASH_ALLOW);
|
|
183
183
|
const opts = {
|
|
@@ -239,7 +239,7 @@ function buildReplayOptions(keeper, policy, key) {
|
|
|
239
239
|
// Which built-ins may run on the runner itself, and whether Bash may at
|
|
240
240
|
// all — loud for the same reason as the rest of this line.
|
|
241
241
|
builtins: builtinTools.size > 0 ? [...builtinTools].join(",") : "(none)",
|
|
242
|
-
bashAllow: builtinTools.has("Bash") ? (bashAllow ? bashAllow.join(",") : "(
|
|
242
|
+
bashAllow: builtinTools.has("Bash") ? (bashAllow.length > 0 ? bashAllow.join(",") : "(none — Bash steps are steered)") : undefined,
|
|
243
243
|
// Who reads what this turn acts on (segmented.md R-PARAM-5). Named at
|
|
244
244
|
// startup because the third state is a replay that silently never runs.
|
|
245
245
|
derive: key.source ??
|
package/dist/control/server.js
CHANGED
|
@@ -31,7 +31,7 @@ import { BIR_CALL_ID, fingerprint, newCallId, parseQualifiedName, qualifyToolNam
|
|
|
31
31
|
import { StepIndexAllocator } from "./ordering.js";
|
|
32
32
|
import { pidAlive } from "./discovery.js";
|
|
33
33
|
import { pinCompatible } from "./pin-guard.js";
|
|
34
|
-
import { contextForToolUse, intentForToolUse, markTranscriptUsage, recentToolResults, settledLastAssistantText, usageSince, } from "./transcript.js";
|
|
34
|
+
import { contextForToolUse, intentForToolUse, markTranscriptUsage, recentToolResults, settledLastAssistantText, thinkingForToolUse, usageSince, } from "./transcript.js";
|
|
35
35
|
import { isHousekeepingCall } from "../record/housekeeping.js";
|
|
36
36
|
import { redact } from "../record/redact.js";
|
|
37
37
|
import { DEFAULT_MAX_STRING_BYTES, serializeCapped, truncateStrings } from "../record/truncate.js";
|
|
@@ -2346,15 +2346,18 @@ export class ControlServer {
|
|
|
2346
2346
|
const agentId = payload.agent_id ?? session.agentId;
|
|
2347
2347
|
// The real intent, read from every line sharing this message's id
|
|
2348
2348
|
// (segmented.md R-INTENT-3). `context` stays the *written* text only, with
|
|
2349
|
-
// its agent prefix
|
|
2350
|
-
// (R-INTENT-2), so it rides separately in `intent`.
|
|
2349
|
+
// its agent prefix, so thinking rides separately in `intent`.
|
|
2351
2350
|
const intent = intentForToolUse(session.transcriptPath, toolUseId, this.intentSources);
|
|
2352
2351
|
const reasoning = intent?.source === "text" ? intent.text : "";
|
|
2353
2352
|
// The prefix rides on whether there is a subagent, not on whether it said
|
|
2354
2353
|
// anything: a subagent's steps must be tellable apart even when it narrated
|
|
2355
2354
|
// nothing. What it never carries is thinking (R-INTENT-2).
|
|
2356
2355
|
const context = agentId ? `[agent ${agentId}] ${reasoning}`.trim() : reasoning;
|
|
2357
|
-
|
|
2356
|
+
// Sent whether or not there was written text: the service keeps it for the
|
|
2357
|
+
// recording page (R-INTENT-2, amended 2026-09-29), and still embeds the
|
|
2358
|
+
// written text first. The probe below reads `intent`, unchanged.
|
|
2359
|
+
const thinking = thinkingForToolUse(session.transcriptPath, toolUseId, this.intentSources);
|
|
2360
|
+
const thinkingIntent = thinking ? { text: thinking, source: "thinking" } : undefined;
|
|
2358
2361
|
// ── replay steering (docs/calculatedReplay.md §7) ────────────────────────
|
|
2359
2362
|
// Runs before anything else, because two of its four answers end the call.
|
|
2360
2363
|
// Not in plan mode (nothing is approved on the user's behalf there), not
|
|
@@ -78,6 +78,15 @@ export interface ToolUseIntent {
|
|
|
78
78
|
* next rung tried. It governs only what may be *sent*; it never stops a probe.
|
|
79
79
|
*/
|
|
80
80
|
export declare function intentForToolUse(transcriptPath: string | undefined, toolUseId: string, sources?: ReadonlySet<IntentSource>): ToolUseIntent | undefined;
|
|
81
|
+
/**
|
|
82
|
+
* The thinking `toolUseId` was chosen in, whether or not the agent also wrote
|
|
83
|
+
* text. The service keeps it with the step so the recording page can show it
|
|
84
|
+
* (segmented.md R-INTENT-2, amended 2026-09-29); {@link intentForToolUse}
|
|
85
|
+
* returns thinking only in place of missing text, which is what matching wants.
|
|
86
|
+
*
|
|
87
|
+
* `sources` is `BIR_INTENT_SOURCES`: without `thinking` in it, none is sent.
|
|
88
|
+
*/
|
|
89
|
+
export declare function thinkingForToolUse(transcriptPath: string | undefined, toolUseId: string, sources?: ReadonlySet<IntentSource>): string | undefined;
|
|
81
90
|
/** The assistant text that accompanied `toolUseId`, or "" when unavailable. */
|
|
82
91
|
export declare function contextForToolUse(transcriptPath: string | undefined, toolUseId: string): string;
|
|
83
92
|
/** The last assistant text in the transcript — the turn's final answer. */
|
|
@@ -92,8 +92,9 @@ const MAX_INDEXES = 16;
|
|
|
92
92
|
const MAX_MESSAGES = 128;
|
|
93
93
|
/**
|
|
94
94
|
* Characters of one message's text, and separately of its thinking, kept.
|
|
95
|
-
* Text becomes a step's `context`; thinking
|
|
96
|
-
*
|
|
95
|
+
* Text becomes a step's `context`; thinking goes with the step as its
|
|
96
|
+
* `intent`, capped again at 8 KB on the way out, and the intent probe sends
|
|
97
|
+
* 4,000 characters of either.
|
|
97
98
|
*/
|
|
98
99
|
const MAX_MESSAGE_CHARS = 16 * 1024;
|
|
99
100
|
/** `tool_use_id`s remembered, newest last. */
|
|
@@ -443,11 +444,7 @@ export function recentToolResults(transcriptPath, n = 5, maxBytes = 2048) {
|
|
|
443
444
|
* next rung tried. It governs only what may be *sent*; it never stops a probe.
|
|
444
445
|
*/
|
|
445
446
|
export function intentForToolUse(transcriptPath, toolUseId, sources = new Set(["text", "thinking", "tool"])) {
|
|
446
|
-
|
|
447
|
-
return undefined;
|
|
448
|
-
const idx = indexFor(transcriptPath);
|
|
449
|
-
const holder = idx?.toolUses.get(toolUseId);
|
|
450
|
-
const message = holder ? idx?.messages.get(holder.message) : undefined;
|
|
447
|
+
const message = messageFor(transcriptPath, toolUseId);
|
|
451
448
|
if (!message)
|
|
452
449
|
return undefined;
|
|
453
450
|
if (sources.has("text")) {
|
|
@@ -462,6 +459,28 @@ export function intentForToolUse(transcriptPath, toolUseId, sources = new Set(["
|
|
|
462
459
|
}
|
|
463
460
|
return undefined;
|
|
464
461
|
}
|
|
462
|
+
/**
|
|
463
|
+
* The thinking `toolUseId` was chosen in, whether or not the agent also wrote
|
|
464
|
+
* text. The service keeps it with the step so the recording page can show it
|
|
465
|
+
* (segmented.md R-INTENT-2, amended 2026-09-29); {@link intentForToolUse}
|
|
466
|
+
* returns thinking only in place of missing text, which is what matching wants.
|
|
467
|
+
*
|
|
468
|
+
* `sources` is `BIR_INTENT_SOURCES`: without `thinking` in it, none is sent.
|
|
469
|
+
*/
|
|
470
|
+
export function thinkingForToolUse(transcriptPath, toolUseId, sources = new Set(["text", "thinking", "tool"])) {
|
|
471
|
+
if (!sources.has("thinking"))
|
|
472
|
+
return undefined;
|
|
473
|
+
const thinking = messageFor(transcriptPath, toolUseId)?.thinkings.join("\n").trim();
|
|
474
|
+
return thinking || undefined;
|
|
475
|
+
}
|
|
476
|
+
/** The message that emitted `toolUseId`: its text and thinking, gathered across its lines. */
|
|
477
|
+
function messageFor(transcriptPath, toolUseId) {
|
|
478
|
+
if (!transcriptPath || !toolUseId)
|
|
479
|
+
return undefined;
|
|
480
|
+
const idx = indexFor(transcriptPath);
|
|
481
|
+
const holder = idx?.toolUses.get(toolUseId);
|
|
482
|
+
return holder ? idx?.messages.get(holder.message) : undefined;
|
|
483
|
+
}
|
|
465
484
|
/** The assistant text that accompanied `toolUseId`, or "" when unavailable. */
|
|
466
485
|
export function contextForToolUse(transcriptPath, toolUseId) {
|
|
467
486
|
if (!transcriptPath || !toolUseId)
|
|
@@ -62,13 +62,21 @@ export declare const TEXT_CAPS: {
|
|
|
62
62
|
readonly answer: number;
|
|
63
63
|
/** A step's written reasoning. */
|
|
64
64
|
readonly context: number;
|
|
65
|
-
/** A step's thinking text,
|
|
65
|
+
/** A step's thinking text: embedded, and kept for the recording page. */
|
|
66
66
|
readonly intent: number;
|
|
67
67
|
/** A tool's error text when it is not JSON. */
|
|
68
68
|
readonly toolError: number;
|
|
69
69
|
/** An execution report's error lines, rendered in a web page. */
|
|
70
70
|
readonly reportError: number;
|
|
71
71
|
};
|
|
72
|
+
/**
|
|
73
|
+
* The most an execution report's request may weigh. The service reads this
|
|
74
|
+
* route with its 100 kB JSON parser — only `/recordings` has the larger one —
|
|
75
|
+
* and a report it refuses for its size loses the turn's cost with it. So the
|
|
76
|
+
* steps' tool payloads get what is left of this after everything else in the
|
|
77
|
+
* report: the costs, the prompt, the verdicts.
|
|
78
|
+
*/
|
|
79
|
+
export declare const REPORT_MAX_BYTES: number;
|
|
72
80
|
/** Queued sends past which the oldest unsent step is dropped (`BIR_RECORDER_MAX_PENDING`). */
|
|
73
81
|
export declare const DEFAULT_MAX_PENDING_SENDS = 500;
|
|
74
82
|
/** Queued bytes past which the same happens (`BIR_RECORDER_MAX_PENDING_BYTES`). */
|
|
@@ -52,7 +52,7 @@ import { credentialsPath, describeBody, isAuthResponse, issuedFor, loadCredentia
|
|
|
52
52
|
import { errText, logDetail, logEvery, logLine } from "../util/log.js";
|
|
53
53
|
import { isUnhealthyStatus, serviceHealth, serviceTimeoutMs, timeoutSignal, } from "../util/service-health.js";
|
|
54
54
|
import { redactJsonText, redactText } from "./redact.js";
|
|
55
|
-
import { capJsonText, capText, DEFAULT_MAX_PAYLOAD_BYTES, preCut } from "./truncate.js";
|
|
55
|
+
import { capJsonText, capText, DEFAULT_MAX_PAYLOAD_BYTES, preCut, REPORT_PAYLOAD_MAX_BYTES, } from "./truncate.js";
|
|
56
56
|
/**
|
|
57
57
|
* Caps for the free text this recorder sends, in UTF-8 bytes. Tool payloads
|
|
58
58
|
* arrive capped at 64 KiB by their producers; text never passed through one,
|
|
@@ -66,13 +66,21 @@ export const TEXT_CAPS = {
|
|
|
66
66
|
answer: 64 * 1024,
|
|
67
67
|
/** A step's written reasoning. */
|
|
68
68
|
context: 8 * 1024,
|
|
69
|
-
/** A step's thinking text,
|
|
69
|
+
/** A step's thinking text: embedded, and kept for the recording page. */
|
|
70
70
|
intent: 8 * 1024,
|
|
71
71
|
/** A tool's error text when it is not JSON. */
|
|
72
72
|
toolError: 8 * 1024,
|
|
73
73
|
/** An execution report's error lines, rendered in a web page. */
|
|
74
74
|
reportError: 2 * 1024,
|
|
75
75
|
};
|
|
76
|
+
/**
|
|
77
|
+
* The most an execution report's request may weigh. The service reads this
|
|
78
|
+
* route with its 100 kB JSON parser — only `/recordings` has the larger one —
|
|
79
|
+
* and a report it refuses for its size loses the turn's cost with it. So the
|
|
80
|
+
* steps' tool payloads get what is left of this after everything else in the
|
|
81
|
+
* report: the costs, the prompt, the verdicts.
|
|
82
|
+
*/
|
|
83
|
+
export const REPORT_MAX_BYTES = 96 * 1024;
|
|
76
84
|
/** Queued sends past which the oldest unsent step is dropped (`BIR_RECORDER_MAX_PENDING`). */
|
|
77
85
|
export const DEFAULT_MAX_PENDING_SENDS = 500;
|
|
78
86
|
/** Queued bytes past which the same happens (`BIR_RECORDER_MAX_PENDING_BYTES`). */
|
|
@@ -177,6 +185,61 @@ function sizeOf(...parts) {
|
|
|
177
185
|
n += p?.text.length ?? 0;
|
|
178
186
|
return n;
|
|
179
187
|
}
|
|
188
|
+
function pendStep(step) {
|
|
189
|
+
return {
|
|
190
|
+
step,
|
|
191
|
+
toolInput: pend(step.toolInput, REPORT_PAYLOAD_MAX_BYTES, true),
|
|
192
|
+
toolOutput: pend(step.toolOutput, REPORT_PAYLOAD_MAX_BYTES, true),
|
|
193
|
+
};
|
|
194
|
+
}
|
|
195
|
+
/**
|
|
196
|
+
* An execution report's request body: `fields`, and its steps as sent — each
|
|
197
|
+
* error and each tool payload capped and redacted, like every other field here.
|
|
198
|
+
*
|
|
199
|
+
* The payloads are what makes a report heavy, so they are added last and only
|
|
200
|
+
* while the whole body stays within {@link REPORT_MAX_BYTES}: the failed
|
|
201
|
+
* steps' first — theirs are the ones somebody will read — then every step's
|
|
202
|
+
* input, then the outputs, each in the order the steps ran. A payload that
|
|
203
|
+
* does not fit is left out; its step keeps its verdict, and the costs the
|
|
204
|
+
* report exists for are never what gives way.
|
|
205
|
+
*/
|
|
206
|
+
function reportBody(fields, steps) {
|
|
207
|
+
if (!steps)
|
|
208
|
+
return JSON.stringify(fields);
|
|
209
|
+
const sent = steps.map(({ step }) => ({
|
|
210
|
+
...step,
|
|
211
|
+
error: typeof step.error === "string"
|
|
212
|
+
? capText(step.error, TEXT_CAPS.reportError, redactText)
|
|
213
|
+
: step.error,
|
|
214
|
+
toolInput: undefined,
|
|
215
|
+
toolOutput: undefined,
|
|
216
|
+
}));
|
|
217
|
+
const bare = Buffer.byteLength(JSON.stringify({ ...fields, steps: sent }), "utf8");
|
|
218
|
+
let left = REPORT_MAX_BYTES - bare;
|
|
219
|
+
const add = (i, key) => {
|
|
220
|
+
if (sent[i][key] !== undefined)
|
|
221
|
+
return;
|
|
222
|
+
const text = out(steps[i][key]);
|
|
223
|
+
if (text === undefined)
|
|
224
|
+
return;
|
|
225
|
+
// The value as a JSON string, and `,"<key>":` in front of it.
|
|
226
|
+
const size = Buffer.byteLength(JSON.stringify(text), "utf8") + key.length + 4;
|
|
227
|
+
if (size > left)
|
|
228
|
+
return;
|
|
229
|
+
sent[i][key] = text;
|
|
230
|
+
left -= size;
|
|
231
|
+
};
|
|
232
|
+
const all = sent.map((_, i) => i);
|
|
233
|
+
for (const i of all.filter((n) => sent[n].status === "failed")) {
|
|
234
|
+
add(i, "toolInput");
|
|
235
|
+
add(i, "toolOutput");
|
|
236
|
+
}
|
|
237
|
+
for (const i of all)
|
|
238
|
+
add(i, "toolInput");
|
|
239
|
+
for (const i of all)
|
|
240
|
+
add(i, "toolOutput");
|
|
241
|
+
return JSON.stringify({ ...fields, steps: sent });
|
|
242
|
+
}
|
|
180
243
|
/** A tool error is JSON when a producer serialized it, and prose when it did not. */
|
|
181
244
|
function pendError(text) {
|
|
182
245
|
if (typeof text !== "string")
|
|
@@ -609,12 +672,21 @@ export class RemoteRecorder {
|
|
|
609
672
|
const costUsd = report.deriveCostUsd + report.sessionCostUsd + report.fallbackCostUsd;
|
|
610
673
|
const prompt = pend(report.prompt, TEXT_CAPS.prompt);
|
|
611
674
|
const error = pend(report.error, TEXT_CAPS.reportError);
|
|
675
|
+
const steps = report.steps?.map(pendStep);
|
|
612
676
|
this.enqueue({
|
|
613
677
|
lane: EXECUTIONS,
|
|
614
678
|
kind: "report",
|
|
615
|
-
bytes: sizeOf(prompt, error) + 1024,
|
|
679
|
+
bytes: sizeOf(prompt, error, ...(steps ?? []).flatMap((s) => [s.toolInput, s.toolOutput])) + 1024,
|
|
616
680
|
send: async () => {
|
|
617
|
-
const body = await this.request("POST", `/scenarios/${report.scenarioId}/executions`,
|
|
681
|
+
const body = await this.request("POST", `/scenarios/${report.scenarioId}/executions`,
|
|
682
|
+
// Per-step verdicts and the headline failure (errorshandling.md) ride
|
|
683
|
+
// beside the costs. All optional on the service side, so an older
|
|
684
|
+
// service ignores them rather than rejecting the report — and the
|
|
685
|
+
// costs still book. Errors are shown on a web page, so they are
|
|
686
|
+
// capped and redacted like any text; so is what each step sent and
|
|
687
|
+
// received, which the page shows under the verdict in place of the
|
|
688
|
+
// recording's sample (`reportBody`).
|
|
689
|
+
reportBody({
|
|
618
690
|
ticket: report.ticket,
|
|
619
691
|
outcome: report.outcome,
|
|
620
692
|
costUsd,
|
|
@@ -627,11 +699,6 @@ export class RemoteRecorder {
|
|
|
627
699
|
measured: report.measured,
|
|
628
700
|
pricingVersion: report.pricingVersion,
|
|
629
701
|
prompt: out(prompt),
|
|
630
|
-
// Per-step verdicts and the headline failure (errorshandling.md). All
|
|
631
|
-
// optional on the service side, so an older service ignores them rather
|
|
632
|
-
// than rejecting the report — and the costs still book. Errors are
|
|
633
|
-
// shown on a web page, so they are capped and redacted like any text.
|
|
634
|
-
steps: report.steps?.map((s) => typeof s.error === "string" ? { ...s, error: capText(s.error, TEXT_CAPS.reportError, redactText) } : s),
|
|
635
702
|
error: out(error),
|
|
636
703
|
errorStage: report.errorStage,
|
|
637
704
|
errorStepIndex: report.errorStepIndex,
|
|
@@ -643,7 +710,7 @@ export class RemoteRecorder {
|
|
|
643
710
|
// Why nothing ran, when nothing ran. Additive: an older service drops
|
|
644
711
|
// the field and books the cost exactly as it always did.
|
|
645
712
|
declined: report.declined,
|
|
646
|
-
}), { retryTransient: Boolean(report.ticket) });
|
|
713
|
+
}, steps), { retryTransient: Boolean(report.ticket) });
|
|
647
714
|
const r = (body ?? {});
|
|
648
715
|
const failed = report.steps?.filter((s) => s.status === "failed").length ?? 0;
|
|
649
716
|
logLine("execution.reported", {
|
|
@@ -20,6 +20,17 @@
|
|
|
20
20
|
export declare const DEFAULT_MAX_PAYLOAD_BYTES: number;
|
|
21
21
|
/** 8 KiB per individual string leaf. */
|
|
22
22
|
export declare const DEFAULT_MAX_STRING_BYTES: number;
|
|
23
|
+
/**
|
|
24
|
+
* What one tool payload may take inside an execution report. A report carries
|
|
25
|
+
* every step of a replay in one small request, so a step's input or output
|
|
26
|
+
* gets far less there than the 64 KiB a recorded step has to itself.
|
|
27
|
+
*
|
|
28
|
+
* 3,800 bytes, not a round 4 KiB: the service keeps 4,000 characters of each,
|
|
29
|
+
* and a payload cut here still fits that with its marker. So the scenario's
|
|
30
|
+
* page shows this cut, which says how large the payload was, and never a
|
|
31
|
+
* second one made on top of it.
|
|
32
|
+
*/
|
|
33
|
+
export declare const REPORT_PAYLOAD_MAX_BYTES = 3800;
|
|
23
34
|
export interface TruncateOptions {
|
|
24
35
|
maxPayloadBytes?: number;
|
|
25
36
|
maxStringBytes?: number;
|
package/dist/record/truncate.js
CHANGED
|
@@ -20,6 +20,17 @@
|
|
|
20
20
|
export const DEFAULT_MAX_PAYLOAD_BYTES = 64 * 1024;
|
|
21
21
|
/** 8 KiB per individual string leaf. */
|
|
22
22
|
export const DEFAULT_MAX_STRING_BYTES = 8 * 1024;
|
|
23
|
+
/**
|
|
24
|
+
* What one tool payload may take inside an execution report. A report carries
|
|
25
|
+
* every step of a replay in one small request, so a step's input or output
|
|
26
|
+
* gets far less there than the 64 KiB a recorded step has to itself.
|
|
27
|
+
*
|
|
28
|
+
* 3,800 bytes, not a round 4 KiB: the service keeps 4,000 characters of each,
|
|
29
|
+
* and a payload cut here still fits that with its marker. So the scenario's
|
|
30
|
+
* page shows this cut, which says how large the payload was, and never a
|
|
31
|
+
* second one made on top of it.
|
|
32
|
+
*/
|
|
33
|
+
export const REPORT_PAYLOAD_MAX_BYTES = 3_800;
|
|
23
34
|
const utf8 = (s) => Buffer.byteLength(s, "utf8");
|
|
24
35
|
/** The marker {@link truncateString} appends, so a second cut can report the first size. */
|
|
25
36
|
const MARKER = /… \[truncated \d+ of (\d+) bytes\]$/;
|
|
@@ -10,18 +10,21 @@
|
|
|
10
10
|
*
|
|
11
11
|
* THE RULES, because this bypasses the host's permission prompts entirely:
|
|
12
12
|
*
|
|
13
|
-
* - **
|
|
14
|
-
* `BIR_BUILTIN_TOOLS` ({@link parseBuiltinTools}),
|
|
15
|
-
*
|
|
16
|
-
*
|
|
17
|
-
*
|
|
13
|
+
* - **All six by default, each a switch.** Which built-ins may run at all is
|
|
14
|
+
* `BIR_BUILTIN_TOOLS` ({@link parseBuiltinTools}); unset, it is every one
|
|
15
|
+
* this module implements (up to 0.3.7 it was the read-only set,
|
|
16
|
+
* `Read,Glob,Grep`). The permission story is the one a steered step
|
|
17
|
+
* already has — its pinned call is answered `allow` too. What running it
|
|
18
|
+
* here changes is that no model sees the call before it runs, so an
|
|
19
|
+
* operator who wants one in the loop takes the tool off the list.
|
|
18
20
|
* - **Path jail.** Every path is resolved against, and confined to, the
|
|
19
21
|
* session's project directory ({@link jailed}), symlinks included. The
|
|
20
22
|
* daemon's own cwd is never used: it is `~/.baseinstrunner`, not the
|
|
21
23
|
* project (§5.1 of the change plan).
|
|
22
|
-
* - **Bash runs
|
|
23
|
-
* match one of its prefixes ({@link bashAllowed});
|
|
24
|
-
*
|
|
24
|
+
* - **Bash runs any command unless `BIR_BASH_ALLOW` narrows it.** Set, a
|
|
25
|
+
* command must match one of its prefixes ({@link bashAllowed}); `none`
|
|
26
|
+
* allows no command, and then `Bash` counts as off the list, so its steps
|
|
27
|
+
* are steered in the session (controller.ts).
|
|
25
28
|
* - **Same failure contract as an MCP step.** A step that *cannot* run here
|
|
26
29
|
* throws {@link BuiltinNotRunnable} (→ `not_run`: the session can still do
|
|
27
30
|
* it, or a recorded output may stand in under
|
|
@@ -36,14 +39,15 @@
|
|
|
36
39
|
* best effort and may drift from a future host's, the same caveat the
|
|
37
40
|
* hook-view threading fallback already carries ("shapes may differ").
|
|
38
41
|
*/
|
|
42
|
+
/** The built-ins {@link runBuiltin} actually implements. */
|
|
43
|
+
export declare const EXECUTABLE_BUILTINS: ReadonlySet<string>;
|
|
39
44
|
/**
|
|
40
45
|
* The built-ins bir may stand in for or execute when `BIR_BUILTIN_TOOLS` is
|
|
41
|
-
* unset:
|
|
42
|
-
*
|
|
46
|
+
* unset: every one it implements, so a chain of built-ins arms `direct` just
|
|
47
|
+
* as a chain of wrapped MCP calls does. Up to 0.3.7 this was the read-only
|
|
48
|
+
* set (`Read,Glob,Grep`).
|
|
43
49
|
*/
|
|
44
50
|
export declare const DEFAULT_BUILTIN_TOOLS: readonly string[];
|
|
45
|
-
/** The built-ins {@link runBuiltin} actually implements. */
|
|
46
|
-
export declare const EXECUTABLE_BUILTINS: ReadonlySet<string>;
|
|
47
51
|
/**
|
|
48
52
|
* Parse `BIR_BUILTIN_TOOLS`. Unset means {@link DEFAULT_BUILTIN_TOOLS}; set,
|
|
49
53
|
* it is the whole list — `none` (or an empty value) turns built-ins off
|
|
@@ -54,10 +58,11 @@ export declare const EXECUTABLE_BUILTINS: ReadonlySet<string>;
|
|
|
54
58
|
export declare function parseBuiltinTools(value: string | undefined): ReadonlySet<string>;
|
|
55
59
|
/**
|
|
56
60
|
* Parse `BIR_BASH_ALLOW`: comma-separated command prefixes, `*` for
|
|
57
|
-
* everything.
|
|
58
|
-
*
|
|
61
|
+
* everything. Unset means everything, because `Bash` is on the default list.
|
|
62
|
+
* Set, it is the whole list, as `BIR_BUILTIN_TOOLS` is: `none` (or an empty
|
|
63
|
+
* value) allows no command.
|
|
59
64
|
*/
|
|
60
|
-
export declare function parseBashAllow(value: string | undefined): readonly string[]
|
|
65
|
+
export declare function parseBashAllow(value: string | undefined): readonly string[];
|
|
61
66
|
/**
|
|
62
67
|
* Whether `command` matches the `BIR_BASH_ALLOW` prefixes. A prefix that ends
|
|
63
68
|
* mid-word must end the command's word too: `git` allows `git status` and
|
|
@@ -92,7 +97,7 @@ export interface BuiltinContext {
|
|
|
92
97
|
cwd: string;
|
|
93
98
|
/** The step's clock, the same budget an MCP dispatch would get. */
|
|
94
99
|
timeoutMs?: number;
|
|
95
|
-
/** Parsed `BIR_BASH_ALLOW` ({@link parseBashAllow}). */
|
|
100
|
+
/** Parsed `BIR_BASH_ALLOW` ({@link parseBashAllow}). Absent or empty, no command runs. */
|
|
96
101
|
bashAllow?: readonly string[];
|
|
97
102
|
/**
|
|
98
103
|
* Where this plan's *previous* `Bash` step left its shell — the host's own
|