@basein/runner 0.3.10 → 0.3.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/dist/bin/bir.js +4 -0
- package/dist/bin/investigate.js +8 -3
- package/dist/config/adapters/claude-code.d.ts +1 -1
- package/dist/config/adapters/claude-code.js +2 -2
- package/dist/control/server.d.ts +37 -1
- package/dist/control/server.js +121 -41
- package/dist/record/recorder.d.ts +25 -0
- package/dist/record/recorder.js +3 -0
- package/dist/record/remote-recorder.d.ts +30 -5
- package/dist/record/remote-recorder.js +51 -6
- package/dist/record/switchable-recorder.js +2 -0
- package/dist/replay/controller.d.ts +25 -1
- package/dist/replay/controller.js +30 -12
- package/dist/replay/derive.d.ts +12 -0
- package/dist/replay/derive.js +25 -0
- package/dist/replay/types.d.ts +7 -0
- package/docs/calculatedReplay.md +19 -6
- package/docs/calculatedReplayGuide.md +1 -1
- package/package.json +1 -1
- package/production.md +6 -6
package/README.md
CHANGED
|
@@ -488,7 +488,7 @@ what to do when they register, so there is one switch and no second copy of it:
|
|
|
488
488
|
| `ANTHROPIC_API_KEY` | Not used by the background recorder, which has Claude Code's environment and would bill your own key — unless `BIR_USE_ANTHROPIC_API_KEY=1`. A `bir-hooks` run by hand in a terminal uses that terminal's key. |
|
|
489
489
|
| `BIR_DERIVE_MODEL` | Derivation model, when you set a key of your own (default `claude-haiku-4-5-20251001`). |
|
|
490
490
|
| `BIR_MATCH_BUDGET_MS` | How long the prompt hook waits for a match (default `3000`). After a match slower than `BIR_MATCH_SLOW_MS` (default `2000`), or one that timed out, prompts stop waiting for 5 min, doubling up to 30 min; a fast match ends that. |
|
|
491
|
-
| `BIR_DERIVE_BUDGET_MS` / `BIR_HOOK_DERIVE_BUDGET_MS` | How long a direct run, and a hook, waits for parameters (default `8000` / `8000`; a hook waits the smaller, and the prompt hook never past its own
|
|
491
|
+
| `BIR_DERIVE_BUDGET_MS` / `BIR_HOOK_DERIVE_BUDGET_MS` | How long a direct run, and a hook, waits for parameters (default `8000` / `8000`; a hook waits the smaller, and the prompt hook never past its own 7.5 s). |
|
|
492
492
|
| `BIR_PRETOOL_REPLAY_BUDGET_MS` | Everything replay does inside one `PreToolUse` (default `9000`, at most `10000`). |
|
|
493
493
|
| `BIR_REPLAY_BUDGET_MS` / `BIR_STEP_TIMEOUT_MS` | Whole-plan and per-step ceilings for direct execution (default `100000` / `60000`; `BIR_REPLAY_PLAN_BUDGET_MS` also sets the first). A direct run's parameter wait plus its plan are held under 270 s. |
|
|
494
494
|
| `BIR_LOGIC_TIMEOUT_MS` | How long one scenario logic body may run before it is stopped and the plan hands over (default `1000`, at most `10000`). |
|
|
@@ -525,7 +525,7 @@ The governing rule: **a host session never fails because of BaseInstRunner.**
|
|
|
525
525
|
| No recorder running at session start | the SessionStart hook starts one in the background; it always exits 0 and never holds the start longer than about 11.5 s |
|
|
526
526
|
| Control server unreachable at startup | steps buffered ≤ 10 s, then Tier 2 (passthrough: nothing); the proxy keeps looking every 15 s and binds when one answers |
|
|
527
527
|
| Control server dies mid-session | the proxy records to a Tier 2 run of its own (flagged lossy), starts the recorder again in the background (not after `bir down`), and binds to it once it answers |
|
|
528
|
-
| A hook the recorder cannot answer in time | every route has a hard deadline (prompt
|
|
528
|
+
| A hook the recorder cannot answer in time | every route has a hard deadline (prompt 8.5 s, `PreToolUse` 12 s, the others 1–2 s) and answers `{}`; so does a hook with a stale token or a malformed body |
|
|
529
529
|
| BaseIn service slow or down | hooks stop waiting on it after one failure (a cool-down of 5 s, doubling to 60 s); sends go on in the background, retried once only when they cannot have arrived (429, 503, a refused connection); past 500 queued sends or 32 MiB the oldest steps are dropped |
|
|
530
530
|
| BaseIn service answers `429`: the account is over one of its limits | the send — a run's create, a step, a report — goes once more, at most 2 s later, and is then dropped, with `recorder.send_failed … HTTP 429` on the log (one line per run). A dropped step is a hole in its recording; a run whose create was refused runs without its plan; a `429` the second try did not cure starts the cool-down above. The limits are settings of the service, per account — by default 120 runs and 3,000 steps a minute (a tool call is two steps), 10 title requests a minute, and 20 Calculate, Prefer and Recalculate requests in any one hour — and are built there, not deployed yet |
|
|
531
531
|
| Upstream fails to start, or exits | calls in flight are answered (`MCP server "<name>" is not available`, `isError: true` for a `tools/call`), then `bir-proxy` exits with the server's own status, so Claude Code shows the server as failed, as it would unwrapped. `BIR_PROXY_STAY_UP=1` keeps the proxy up answering that instead |
|
package/dist/bin/bir.js
CHANGED
|
@@ -68,6 +68,7 @@ import { SCENARIO_EDIT_SUBCOMMANDS, scenarioEditCommand } from "./scenario-edit.
|
|
|
68
68
|
import { loadCredentials } from "../auth/client.js";
|
|
69
69
|
import { packageVersion } from "../util/version.js";
|
|
70
70
|
import { runnerOutdated, serviceHeaders } from "../util/service-headers.js";
|
|
71
|
+
import { localDay } from "../replay/derive.js";
|
|
71
72
|
const VERSION = packageVersion();
|
|
72
73
|
/**
|
|
73
74
|
* The npx SessionStart hook's own budget, passed as `ensure --budget-ms <n>`
|
|
@@ -3630,6 +3631,9 @@ async function replayCommand(args) {
|
|
|
3630
3631
|
if (args.dry) {
|
|
3631
3632
|
const { status, body } = await service("POST", `/scenarios/${scenarioId}/replay`, {
|
|
3632
3633
|
prompt: args.prompt,
|
|
3634
|
+
// The day on this machine, as a session's reading sends it: a dry
|
|
3635
|
+
// replay works a value like "tomorrow" out from the same day.
|
|
3636
|
+
today: localDay(),
|
|
3633
3637
|
});
|
|
3634
3638
|
if (status !== 200) {
|
|
3635
3639
|
out(`Dry replay failed (HTTP ${status}): ${JSON.stringify(body)}`);
|
package/dist/bin/investigate.js
CHANGED
|
@@ -214,8 +214,13 @@ export const DECLINES = {
|
|
|
214
214
|
// Not a gate of the ladder: the control server's own verdict, when arming
|
|
215
215
|
// outlived the hook that would have announced the plan.
|
|
216
216
|
arm_timeout: {
|
|
217
|
-
cause: "the plan was not ready before the hook that would have announced it had to answer: the prompt hook waits
|
|
218
|
-
fix: "run the prompt again. If it keeps happening the reading is slow: the recorder's log (`bir status` names the file) shows when its `replay.derived` line landed. BIR_HOOK_DERIVE_BUDGET_MS does not help here, because the hook's own limit was reached first. To read values on this machine instead of asking the service, set BIR_DERIVE_API_KEY where Claude Code starts.",
|
|
217
|
+
cause: "the plan was not ready before the hook that would have announced it had to answer: the prompt hook waits 7.5 s in all, match included, and a tool call that starts a plan mid-task waits 1.5 s. What arming waits for is the reading of the prompt's values (the derivation). A plan that was not announced is never run.",
|
|
218
|
+
fix: "run the prompt again. If it keeps happening the reading is slow: the recorder's log (`bir status` names the file) shows when its `replay.derived` line landed, and a dry replay shows what the service's reading took, under `reading` (`bir scenario replay <scnId> --prompt \"…\" --dry --json`). BIR_HOOK_DERIVE_BUDGET_MS does not help here, because the hook's own limit was reached first. To read values on this machine instead of asking the service, set BIR_DERIVE_API_KEY where Claude Code starts.",
|
|
219
|
+
},
|
|
220
|
+
// Nor is this one: the match itself was late, so nothing was armed at all.
|
|
221
|
+
match_late: {
|
|
222
|
+
cause: "the service's answer to the match landed too late: after the prompt hook had stopped waiting for it (3 s, BIR_MATCH_BUDGET_MS); while prompts were not waiting on the match at all — after one slow answer they stop waiting for a few minutes (`replay.match_gate state=holding`); or with the hook not waiting from the start, because a failure to reach the service was still on record (a `service.degraded` line, perhaps hours before, and no `service.recovered` until after this prompt) or the match request's first attempt failed. The agent did the task itself.",
|
|
223
|
+
fix: "nothing in the prompt has to change: run it again a few minutes later. A `service.recovered` line just after this turn's `run.start` means an old failure was still on record, and the next prompt waits again. If it keeps happening without one, the service answers matches slowly: `bir doctor` shows its state, and `/health.matchGate` when prompts wait again.",
|
|
219
224
|
},
|
|
220
225
|
runner_update_required: {
|
|
221
226
|
cause: "the service matched the prompt and handed over no plan: this runner is older than its plans need.",
|
|
@@ -287,7 +292,7 @@ function readingDecline(code, why) {
|
|
|
287
292
|
return {
|
|
288
293
|
cause: `the plan needs a target, a value read from the prompt, and that reading (the derivation) had not answered when the runner stopped waiting for it: its wait is ${waitedMs} ms, and none while the service is cooling down after a slow or failed call. ` +
|
|
289
294
|
"Whether the prompt names the target was never learned. A replay never takes a target from the recording.",
|
|
290
|
-
fix: "nothing in the prompt needs to change: run it again. If it keeps happening the reading is slower than the wait, and the recorder's log (`bir status` names the file) has its `replay.derived` line after this decline. Give it longer: set BIR_HOOK_DERIVE_BUDGET_MS where the recorder starts, then `bir up --restart`. The prompt hook stops
|
|
295
|
+
fix: "nothing in the prompt needs to change: run it again. If it keeps happening the reading is slower than the wait, and the recorder's log (`bir status` names the file) has its `replay.derived` line after this decline. Give it longer: set BIR_HOOK_DERIVE_BUDGET_MS where the recorder starts, then `bir up --restart`. The prompt hook stops 7.5 s after the prompt arrived, whatever is set.",
|
|
291
296
|
};
|
|
292
297
|
}
|
|
293
298
|
// Gate 7 writes `<reason>: target <name>`, and `no_derive_key` itself as the
|
|
@@ -182,7 +182,7 @@ export declare function setLocalEntries(claudeJson: string, keys: string[], name
|
|
|
182
182
|
* to its `timeout`, so each number here is how long a slow or blackholed
|
|
183
183
|
* service could make a person wait — at every prompt, every tool call, every
|
|
184
184
|
* `/exit`. The control server keeps its own per-route deadlines well inside
|
|
185
|
-
* them (prompt ≤
|
|
185
|
+
* them (prompt ≤ 8.5 s, tool/pre ≤ 12 s, tool/post and stop ≤ 2 s, end ≤ 1 s,
|
|
186
186
|
* start ≤ 2 s) and answers `{}` on overrun, so these are the second line of
|
|
187
187
|
* defence, not a budget to spend. SessionEnd matters twice: Claude Code's own
|
|
188
188
|
* default for it is 1.5 s, and whatever is written here RAISES that — the 30 it
|
|
@@ -448,7 +448,7 @@ export function setLocalEntries(claudeJson, keys, name, entry, shouldTouch = ()
|
|
|
448
448
|
* to its `timeout`, so each number here is how long a slow or blackholed
|
|
449
449
|
* service could make a person wait — at every prompt, every tool call, every
|
|
450
450
|
* `/exit`. The control server keeps its own per-route deadlines well inside
|
|
451
|
-
* them (prompt ≤
|
|
451
|
+
* them (prompt ≤ 8.5 s, tool/pre ≤ 12 s, tool/post and stop ≤ 2 s, end ≤ 1 s,
|
|
452
452
|
* start ≤ 2 s) and answers `{}` on overrun, so these are the second line of
|
|
453
453
|
* defence, not a budget to spend. SessionEnd matters twice: Claude Code's own
|
|
454
454
|
* default for it is 1.5 s, and whatever is written here RAISES that — the 30 it
|
|
@@ -460,7 +460,7 @@ export const HOOK_ROUTES = [
|
|
|
460
460
|
{ event: "SessionStart", route: "/session/start", matcher: "", timeout: 5 },
|
|
461
461
|
// Awaits the similar-prompt match and the derivation before it can return a
|
|
462
462
|
// steering directive (docs/calculatedReplay.md §10); the server caps the
|
|
463
|
-
// whole route at
|
|
463
|
+
// whole route at 8.5 s.
|
|
464
464
|
//
|
|
465
465
|
// It must NEVER become `async`. An async hook is fire-and-forget, so it cannot
|
|
466
466
|
// return `additionalContext` at all — replay would silently never arm.
|
package/dist/control/server.d.ts
CHANGED
|
@@ -582,6 +582,12 @@ export declare class ControlServer {
|
|
|
582
582
|
* plan. Memoized, because `ensureRun` is reached from five different hooks and
|
|
583
583
|
* arming twice would run the scenario twice.
|
|
584
584
|
*
|
|
585
|
+
* Two answers are followed. `pending` is what the prompt hook may wait for,
|
|
586
|
+
* and the recorder answers it null at once while a failure is on record or
|
|
587
|
+
* once the create's first attempt failed. `settled` is the service's own
|
|
588
|
+
* answer, whenever it lands: the hit that stops the recording, and a ticket
|
|
589
|
+
* the turn owes a report even when nobody waited for it.
|
|
590
|
+
*
|
|
585
591
|
* Resolves with the steering directive to inject, or undefined.
|
|
586
592
|
*/
|
|
587
593
|
private watchForMatch;
|
|
@@ -593,6 +599,11 @@ export declare class ControlServer {
|
|
|
593
599
|
* match. A match that misses its budget, or a plan still deriving when the
|
|
594
600
|
* budget runs out, is never delivered: the turn runs as an ordinary one, and
|
|
595
601
|
* one `replay.decision` line says why (docs/calculatedReplay.md §10).
|
|
602
|
+
*
|
|
603
|
+
* `pending` is what the prompt may wait for; `settled` is the service's own
|
|
604
|
+
* answer, whenever it lands (see {@link watchForMatch}). Every way of going
|
|
605
|
+
* ahead without a match follows `settled`, so a hand-out nobody waited for
|
|
606
|
+
* still has its ticket reported ({@link followLateMatch}).
|
|
596
607
|
*/
|
|
597
608
|
private armFromMatch;
|
|
598
609
|
/**
|
|
@@ -607,6 +618,25 @@ export declare class ControlServer {
|
|
|
607
618
|
* not several, so the hold doubles at most once per hold (on its probe).
|
|
608
619
|
*/
|
|
609
620
|
private noteMatchLatency;
|
|
621
|
+
/**
|
|
622
|
+
* The prompt went ahead without the match: when the service's own answer
|
|
623
|
+
* lands, a hit is a hand-out nobody used. It is said once, as `match_late`
|
|
624
|
+
* with `why`, and its ticket is carried to the turn's report
|
|
625
|
+
* ({@link noteLateMatch}). A miss says nothing: the service has no ticket out.
|
|
626
|
+
*/
|
|
627
|
+
private followLateMatch;
|
|
628
|
+
/**
|
|
629
|
+
* A match that landed after its prompt had been answered, or while prompts
|
|
630
|
+
* were not waiting on it. Nothing is armed for it, and nothing derived. But
|
|
631
|
+
* the service issued a ticket with that hand-out, and without a report under
|
|
632
|
+
* it the console can only say the turn never reported — for a turn the agent
|
|
633
|
+
* simply did itself. So the turn carries the ticket to its report, as a
|
|
634
|
+
* decline with its own code, like a plan that was only late ({@link shelve}).
|
|
635
|
+
*
|
|
636
|
+
* Only a hand-out with a ticket: a match the service handed nothing for has
|
|
637
|
+
* its reason on the service already. And only while the turn is open.
|
|
638
|
+
*/
|
|
639
|
+
private noteLateMatch;
|
|
610
640
|
/**
|
|
611
641
|
* {@link armFromMatch} with replay off: nothing can be announced, and no hook
|
|
612
642
|
* waits for the answer (`onPrompt` has already returned). So there is no
|
|
@@ -614,7 +644,8 @@ export declare class ControlServer {
|
|
|
614
644
|
* had, and must not mark the service slow (that would open the process-wide
|
|
615
645
|
* cool-down, and `/health` would report a degradation the user never had).
|
|
616
646
|
* A hit still books its decline — the ledger's baseline sample — whenever
|
|
617
|
-
* it lands, as long as the turn is still open to carry it.
|
|
647
|
+
* it lands, as long as the turn is still open to carry it. So `settled` is
|
|
648
|
+
* the service's own answer, not the one a hook may have been given at once.
|
|
618
649
|
*/
|
|
619
650
|
private noteDisabledMatch;
|
|
620
651
|
/**
|
|
@@ -712,6 +743,11 @@ export declare class ControlServer {
|
|
|
712
743
|
* told of it, so it must never steer or diverge a call: it is kept only as
|
|
713
744
|
* what it now is — a matched turn the agent does itself, which still owes the
|
|
714
745
|
* ledger its baseline sample (§11.1) — exactly like a declined match.
|
|
746
|
+
*
|
|
747
|
+
* `why` is the code the report carries when the plan was only late. Without
|
|
748
|
+
* it the service got a matched turn that was not steered and no reason, and
|
|
749
|
+
* its console could only say that the runner had given none. A plan a gate
|
|
750
|
+
* declined keeps the gate's own code.
|
|
715
751
|
*/
|
|
716
752
|
private shelve;
|
|
717
753
|
/**
|
package/dist/control/server.js
CHANGED
|
@@ -36,7 +36,7 @@ import { isHousekeepingCall } from "../record/housekeeping.js";
|
|
|
36
36
|
import { redact, redactText } from "../record/redact.js";
|
|
37
37
|
import { DEFAULT_MAX_STRING_BYTES, serializeCapped, truncateString, truncateStrings, } from "../record/truncate.js";
|
|
38
38
|
import { StepQueue } from "../record/queue.js";
|
|
39
|
-
import { NullRecorder, isIntentMatcher, isMatchAware, isRecorderHealthReporter, isRunCreationAware, isScenarioReporter, } from "../record/recorder.js";
|
|
39
|
+
import { NullRecorder, isIntentMatcher, isMatchAware, isRecorderHealthReporter, isRunCreationAware, isScenarioReporter, isServiceProber, } from "../record/recorder.js";
|
|
40
40
|
import { ReplayController, POLL_HOLD_MS, } from "../replay/controller.js";
|
|
41
41
|
import { calculateCostUsd, hasModelPricing } from "../replay/pricing.js";
|
|
42
42
|
import { logDetail, logLine, logOnce, errText } from "../util/log.js";
|
|
@@ -64,7 +64,9 @@ const PROXY_REPORT_GRACE_MS = 1_500;
|
|
|
64
64
|
*/
|
|
65
65
|
const HOOK_DEADLINE_MS = {
|
|
66
66
|
"/session/start": 2_000,
|
|
67
|
-
|
|
67
|
+
// A second past {@link PROMPT_WAIT_MS}, and a second and a half inside the
|
|
68
|
+
// 10 s Claude Code gives the hook.
|
|
69
|
+
"/session/prompt": 8_500,
|
|
68
70
|
"/tool/pre": 12_000,
|
|
69
71
|
"/tool/post": 2_000,
|
|
70
72
|
"/session/agent": 2_000,
|
|
@@ -79,10 +81,19 @@ const PREVIEW_RESERVE_MS = 500;
|
|
|
79
81
|
/**
|
|
80
82
|
* What `UserPromptSubmit` may spend on replay: the similar-prompt match plus
|
|
81
83
|
* any derivation a matched scenario with a target needs before its directive
|
|
82
|
-
* can go out (segmented.md R-PARAM-4). Below the route's
|
|
83
|
-
* decision to steer is always taken here and never by the guard.
|
|
84
|
+
* can go out (segmented.md R-PARAM-4). Below the route's 8.5 s deadline, so
|
|
85
|
+
* the decision to steer is always taken here and never by the guard.
|
|
86
|
+
*
|
|
87
|
+
* LONGER THAN THE SERVICE MAY TAKE TO READ, ON PURPOSE. The service gives a
|
|
88
|
+
* reading 6 s (its RUNNER_DERIVE_TIMEOUT_MS) and answers either way, so with
|
|
89
|
+
* the match and the round trip an answer can be 6.5 s after the prompt. At
|
|
90
|
+
* 6 s this wait ended before that answer could arrive: on 2026-10-08 a
|
|
91
|
+
* reading that took 6.0 s armed its plan 0.3 s after the prompt had been
|
|
92
|
+
* answered, and a good 20-step plan was thrown away (`arm_timeout`) for a
|
|
93
|
+
* turn the agent then spent 39 s on. The wait is only ever this long while a
|
|
94
|
+
* matched plan's values are still being read.
|
|
84
95
|
*/
|
|
85
|
-
const PROMPT_WAIT_MS =
|
|
96
|
+
const PROMPT_WAIT_MS = 7_500;
|
|
86
97
|
/**
|
|
87
98
|
* How long a `PreToolUse` waits for the intent probe when that probe may arm a
|
|
88
99
|
* plan in the call's place (`BIR_INTENT_ARM_BUDGET_MS`). A clear match or a
|
|
@@ -1629,6 +1640,12 @@ export class ControlServer {
|
|
|
1629
1640
|
* plan. Memoized, because `ensureRun` is reached from five different hooks and
|
|
1630
1641
|
* arming twice would run the scenario twice.
|
|
1631
1642
|
*
|
|
1643
|
+
* Two answers are followed. `pending` is what the prompt hook may wait for,
|
|
1644
|
+
* and the recorder answers it null at once while a failure is on record or
|
|
1645
|
+
* once the create's first attempt failed. `settled` is the service's own
|
|
1646
|
+
* answer, whenever it lands: the hit that stops the recording, and a ticket
|
|
1647
|
+
* the turn owes a report even when nobody waited for it.
|
|
1648
|
+
*
|
|
1632
1649
|
* Resolves with the steering directive to inject, or undefined.
|
|
1633
1650
|
*/
|
|
1634
1651
|
watchForMatch(session, run, prompt = "") {
|
|
@@ -1639,7 +1656,10 @@ export class ControlServer {
|
|
|
1639
1656
|
const runId = run.runId;
|
|
1640
1657
|
const startedAt = Date.now();
|
|
1641
1658
|
const pending = this.recorder.getMatch(runId).catch(() => null);
|
|
1642
|
-
|
|
1659
|
+
const settled = this.recorder.settledMatch
|
|
1660
|
+
? this.recorder.settledMatch(runId).catch(() => null)
|
|
1661
|
+
: pending;
|
|
1662
|
+
void settled.then((match) => {
|
|
1643
1663
|
if (!match || run.runId !== runId)
|
|
1644
1664
|
return;
|
|
1645
1665
|
run.recording = false;
|
|
@@ -1651,7 +1671,7 @@ export class ControlServer {
|
|
|
1651
1671
|
why: "similar prompt — the service kept its own run; not recording this one",
|
|
1652
1672
|
});
|
|
1653
1673
|
});
|
|
1654
|
-
run.matchWatch = this.armFromMatch(session, run, pending, prompt, startedAt).catch((err) => {
|
|
1674
|
+
run.matchWatch = this.armFromMatch(session, run, pending, settled, prompt, startedAt).catch((err) => {
|
|
1655
1675
|
// Replay is an optimisation; a broken one must never cost a turn.
|
|
1656
1676
|
logLine("replay.arm_failed", { run: run.runId, error: errText(err) });
|
|
1657
1677
|
return undefined;
|
|
@@ -1666,12 +1686,17 @@ export class ControlServer {
|
|
|
1666
1686
|
* match. A match that misses its budget, or a plan still deriving when the
|
|
1667
1687
|
* budget runs out, is never delivered: the turn runs as an ordinary one, and
|
|
1668
1688
|
* one `replay.decision` line says why (docs/calculatedReplay.md §10).
|
|
1689
|
+
*
|
|
1690
|
+
* `pending` is what the prompt may wait for; `settled` is the service's own
|
|
1691
|
+
* answer, whenever it lands (see {@link watchForMatch}). Every way of going
|
|
1692
|
+
* ahead without a match follows `settled`, so a hand-out nobody waited for
|
|
1693
|
+
* still has its ticket reported ({@link followLateMatch}).
|
|
1669
1694
|
*/
|
|
1670
|
-
async armFromMatch(session, run, pending, prompt, startedAt) {
|
|
1695
|
+
async armFromMatch(session, run, pending, settled, prompt, startedAt) {
|
|
1671
1696
|
// Off for the recorder, or — in a session of another host — not asked
|
|
1672
1697
|
// for by this run's adapter: either way nothing can be announced.
|
|
1673
1698
|
if (!this.replay.enabled || this.recordsOnly(session, run)) {
|
|
1674
|
-
return this.noteDisabledMatch(session, run,
|
|
1699
|
+
return this.noteDisabledMatch(session, run, settled, prompt);
|
|
1675
1700
|
}
|
|
1676
1701
|
const promptWaitMs = this.promptWaitMs();
|
|
1677
1702
|
const matchBudget = Math.min(this.replay.budgets.matchMs, promptWaitMs);
|
|
@@ -1688,18 +1713,7 @@ export class ControlServer {
|
|
|
1688
1713
|
run: run.runId,
|
|
1689
1714
|
why: "the service has been answering matches slowly — not waited on; this turn runs normally",
|
|
1690
1715
|
});
|
|
1691
|
-
|
|
1692
|
-
if (!late)
|
|
1693
|
-
return;
|
|
1694
|
-
logLine("replay.decision", {
|
|
1695
|
-
verdict: "no-steer",
|
|
1696
|
-
code: "match_late",
|
|
1697
|
-
run: run.runId,
|
|
1698
|
-
scenario: late.scenarioId ?? undefined,
|
|
1699
|
-
ms: Date.now() - startedAt,
|
|
1700
|
-
why: "the match landed while prompts were not waiting on it — not steered, and not recorded",
|
|
1701
|
-
});
|
|
1702
|
-
});
|
|
1716
|
+
this.followLateMatch(session, run, settled, startedAt, "the match landed while prompts were not waiting on it — not steered, and not recorded");
|
|
1703
1717
|
return undefined;
|
|
1704
1718
|
}
|
|
1705
1719
|
const cooling = !this.service.canWait();
|
|
@@ -1723,23 +1737,18 @@ export class ControlServer {
|
|
|
1723
1737
|
? "the service is failing — not waited on; this turn runs normally"
|
|
1724
1738
|
: "the match did not answer inside its budget; this turn runs normally",
|
|
1725
1739
|
});
|
|
1726
|
-
|
|
1727
|
-
if (!late)
|
|
1728
|
-
return;
|
|
1729
|
-
logLine("replay.decision", {
|
|
1730
|
-
verdict: "no-steer",
|
|
1731
|
-
code: "match_late",
|
|
1732
|
-
run: run.runId,
|
|
1733
|
-
scenario: late.scenarioId ?? undefined,
|
|
1734
|
-
ms: Date.now() - startedAt,
|
|
1735
|
-
why: "the match landed after the prompt was answered — not steered, and not recorded",
|
|
1736
|
-
});
|
|
1737
|
-
});
|
|
1740
|
+
this.followLateMatch(session, run, settled, startedAt, "the match landed after the prompt was answered — not steered, and not recorded");
|
|
1738
1741
|
}
|
|
1739
1742
|
return undefined;
|
|
1740
1743
|
}
|
|
1741
|
-
if (!match)
|
|
1744
|
+
if (!match) {
|
|
1745
|
+
// "No match" — which may have come without waiting: the recorder answers
|
|
1746
|
+
// the prompt so at once while a service failure is on record, however
|
|
1747
|
+
// old, and once the create's first attempt failed. The service may have
|
|
1748
|
+
// matched the prompt all the same, and issued a ticket.
|
|
1749
|
+
this.followLateMatch(session, run, settled, startedAt, "the match landed after the prompt had gone ahead without it (a service failure was still on record, or the create's first attempt failed) — not steered, and not recorded");
|
|
1742
1750
|
return undefined;
|
|
1751
|
+
}
|
|
1743
1752
|
// Awaited: a scenario with a target derives before the directive goes
|
|
1744
1753
|
// out, so a plan that cannot find its target is never delivered
|
|
1745
1754
|
// (segmented.md R-PARAM-4). Inside the prompt's own budget.
|
|
@@ -1768,12 +1777,12 @@ export class ControlServer {
|
|
|
1768
1777
|
});
|
|
1769
1778
|
}
|
|
1770
1779
|
if (state) {
|
|
1771
|
-
this.shelve(session, run, state);
|
|
1780
|
+
this.shelve(session, run, state, "arm_unannounced");
|
|
1772
1781
|
}
|
|
1773
1782
|
else {
|
|
1774
1783
|
void arming.then((late) => {
|
|
1775
1784
|
if (late)
|
|
1776
|
-
this.shelve(session, run, late);
|
|
1785
|
+
this.shelve(session, run, late, "arm_timeout");
|
|
1777
1786
|
});
|
|
1778
1787
|
}
|
|
1779
1788
|
return undefined;
|
|
@@ -1858,6 +1867,45 @@ export class ControlServer {
|
|
|
1858
1867
|
why: "the match answered fast again — prompts wait on it as before",
|
|
1859
1868
|
});
|
|
1860
1869
|
}
|
|
1870
|
+
/**
|
|
1871
|
+
* The prompt went ahead without the match: when the service's own answer
|
|
1872
|
+
* lands, a hit is a hand-out nobody used. It is said once, as `match_late`
|
|
1873
|
+
* with `why`, and its ticket is carried to the turn's report
|
|
1874
|
+
* ({@link noteLateMatch}). A miss says nothing: the service has no ticket out.
|
|
1875
|
+
*/
|
|
1876
|
+
followLateMatch(session, run, settled, startedAt, why) {
|
|
1877
|
+
void settled.then((late) => {
|
|
1878
|
+
if (!late)
|
|
1879
|
+
return;
|
|
1880
|
+
logLine("replay.decision", {
|
|
1881
|
+
verdict: "no-steer",
|
|
1882
|
+
code: "match_late",
|
|
1883
|
+
run: run.runId,
|
|
1884
|
+
scenario: late.scenarioId ?? undefined,
|
|
1885
|
+
ms: Date.now() - startedAt,
|
|
1886
|
+
why,
|
|
1887
|
+
});
|
|
1888
|
+
this.noteLateMatch(session, run, late);
|
|
1889
|
+
});
|
|
1890
|
+
}
|
|
1891
|
+
/**
|
|
1892
|
+
* A match that landed after its prompt had been answered, or while prompts
|
|
1893
|
+
* were not waiting on it. Nothing is armed for it, and nothing derived. But
|
|
1894
|
+
* the service issued a ticket with that hand-out, and without a report under
|
|
1895
|
+
* it the console can only say the turn never reported — for a turn the agent
|
|
1896
|
+
* simply did itself. So the turn carries the ticket to its report, as a
|
|
1897
|
+
* decline with its own code, like a plan that was only late ({@link shelve}).
|
|
1898
|
+
*
|
|
1899
|
+
* Only a hand-out with a ticket: a match the service handed nothing for has
|
|
1900
|
+
* its reason on the service already. And only while the turn is open.
|
|
1901
|
+
*/
|
|
1902
|
+
noteLateMatch(session, run, match) {
|
|
1903
|
+
if (run.finished || run.lateMatch || !match.scenarioId || !match.executionTicket)
|
|
1904
|
+
return;
|
|
1905
|
+
const state = this.replay.lateMatch(match, this.hostContext(session, run));
|
|
1906
|
+
state.sessionId = session.sessionId;
|
|
1907
|
+
run.lateMatch = state;
|
|
1908
|
+
}
|
|
1861
1909
|
/**
|
|
1862
1910
|
* {@link armFromMatch} with replay off: nothing can be announced, and no hook
|
|
1863
1911
|
* waits for the answer (`onPrompt` has already returned). So there is no
|
|
@@ -1865,10 +1913,11 @@ export class ControlServer {
|
|
|
1865
1913
|
* had, and must not mark the service slow (that would open the process-wide
|
|
1866
1914
|
* cool-down, and `/health` would report a degradation the user never had).
|
|
1867
1915
|
* A hit still books its decline — the ledger's baseline sample — whenever
|
|
1868
|
-
* it lands, as long as the turn is still open to carry it.
|
|
1916
|
+
* it lands, as long as the turn is still open to carry it. So `settled` is
|
|
1917
|
+
* the service's own answer, not the one a hook may have been given at once.
|
|
1869
1918
|
*/
|
|
1870
|
-
async noteDisabledMatch(session, run,
|
|
1871
|
-
const match = await
|
|
1919
|
+
async noteDisabledMatch(session, run, settled, prompt) {
|
|
1920
|
+
const match = await settled;
|
|
1872
1921
|
if (!match || run.finished)
|
|
1873
1922
|
return undefined;
|
|
1874
1923
|
// Declined at its first gate, before derivation: no transcript is read for it.
|
|
@@ -2128,7 +2177,7 @@ export class ControlServer {
|
|
|
2128
2177
|
scenario: state.scenarioId ?? undefined,
|
|
2129
2178
|
why: "the prompt was answered without this plan's directive — retired unannounced, so never run",
|
|
2130
2179
|
});
|
|
2131
|
-
this.shelve(session, run, state);
|
|
2180
|
+
this.shelve(session, run, state, "arm_unannounced");
|
|
2132
2181
|
}
|
|
2133
2182
|
}
|
|
2134
2183
|
/**
|
|
@@ -2136,14 +2185,20 @@ export class ControlServer {
|
|
|
2136
2185
|
* told of it, so it must never steer or diverge a call: it is kept only as
|
|
2137
2186
|
* what it now is — a matched turn the agent does itself, which still owes the
|
|
2138
2187
|
* ledger its baseline sample (§11.1) — exactly like a declined match.
|
|
2188
|
+
*
|
|
2189
|
+
* `why` is the code the report carries when the plan was only late. Without
|
|
2190
|
+
* it the service got a matched turn that was not steered and no reason, and
|
|
2191
|
+
* its console could only say that the runner had given none. A plan a gate
|
|
2192
|
+
* declined keeps the gate's own code.
|
|
2139
2193
|
*/
|
|
2140
|
-
shelve(session, run, late) {
|
|
2194
|
+
shelve(session, run, late, why) {
|
|
2141
2195
|
if (late.plan && !late.retired) {
|
|
2142
2196
|
logDetail("replay.shelved", {
|
|
2143
2197
|
run: run.runId,
|
|
2144
2198
|
scenario: late.scenarioId ?? undefined,
|
|
2145
2199
|
why: "armed without its directive reaching the model — retired, never steers",
|
|
2146
2200
|
});
|
|
2201
|
+
late.declined ??= why;
|
|
2147
2202
|
}
|
|
2148
2203
|
late.sessionId = session.sessionId;
|
|
2149
2204
|
late.plan = undefined;
|
|
@@ -2419,6 +2474,25 @@ export class ControlServer {
|
|
|
2419
2474
|
}
|
|
2420
2475
|
this.queue.push(() => recorder.reportExecution(report));
|
|
2421
2476
|
});
|
|
2477
|
+
// The prompt match that landed too late (see `noteLateMatch`). Not when a
|
|
2478
|
+
// plan of the same scenario armed later in the turn: that one has a ticket
|
|
2479
|
+
// of its own, and says what happened. The turn's cost is a sample of the
|
|
2480
|
+
// task's usual cost only when no plan ran in it at all.
|
|
2481
|
+
const late = run.lateMatch;
|
|
2482
|
+
if (late && !states.some((s) => s.scenarioId === late.scenarioId)) {
|
|
2483
|
+
const alone = states.length === 0;
|
|
2484
|
+
const report = this.replay.buildReport(late, {
|
|
2485
|
+
sessionCostUsd: alone ? cost.usd : 0,
|
|
2486
|
+
measured: alone && cost.measured,
|
|
2487
|
+
durationMs,
|
|
2488
|
+
prompt: run.input || undefined,
|
|
2489
|
+
siblings: [...states, late],
|
|
2490
|
+
runId: run.runId,
|
|
2491
|
+
cutShort: cutShort || undefined,
|
|
2492
|
+
});
|
|
2493
|
+
if (report)
|
|
2494
|
+
this.queue.push(() => recorder.reportExecution(report));
|
|
2495
|
+
}
|
|
2422
2496
|
}
|
|
2423
2497
|
/**
|
|
2424
2498
|
* Seal the session's run now, and drain it to the service in the background.
|
|
@@ -2496,6 +2570,12 @@ export class ControlServer {
|
|
|
2496
2570
|
cwd: session.cwd,
|
|
2497
2571
|
why: "run deferred to prompt",
|
|
2498
2572
|
});
|
|
2573
|
+
// The first prompt is seconds away, and its match is not waited for while
|
|
2574
|
+
// a service failure is on record — one from hours ago included, when the
|
|
2575
|
+
// machine slept or went offline and nothing was sent since. Find out now,
|
|
2576
|
+
// in the background, whether the service answers again.
|
|
2577
|
+
if (isServiceProber(this.recorder))
|
|
2578
|
+
this.recorder.probeIfFailing();
|
|
2499
2579
|
return {};
|
|
2500
2580
|
}
|
|
2501
2581
|
async onPrompt(payload) {
|
|
@@ -180,6 +180,16 @@ export interface RunMatch {
|
|
|
180
180
|
/** Optional capability: `null` means "fresh run, keep recording". */
|
|
181
181
|
export interface MatchAware {
|
|
182
182
|
getMatch(runId: string): Promise<RunMatch | null>;
|
|
183
|
+
/**
|
|
184
|
+
* The create call's own answer, whenever it lands. `getMatch` is what a hook
|
|
185
|
+
* may wait for, and a recorder may answer it null without waiting — while
|
|
186
|
+
* the service has a failure on record, or once the create's first attempt
|
|
187
|
+
* failed. The service can still match the prompt then, hand out the scenario
|
|
188
|
+
* and issue a ticket the turn owes a report; this is how the turn finds out.
|
|
189
|
+
* Never for a hook to wait on. Absent where `getMatch` always waits for the
|
|
190
|
+
* create's answer: it is the same answer then.
|
|
191
|
+
*/
|
|
192
|
+
settledMatch?(runId: string): Promise<RunMatch | null>;
|
|
183
193
|
}
|
|
184
194
|
export declare function isMatchAware(r: Recorder): r is Recorder & MatchAware;
|
|
185
195
|
/**
|
|
@@ -201,6 +211,21 @@ export interface RunCreationAware {
|
|
|
201
211
|
runCreated(runId: string): Promise<RunCreation>;
|
|
202
212
|
}
|
|
203
213
|
export declare function isRunCreationAware(r: Recorder): r is Recorder & RunCreationAware;
|
|
214
|
+
/**
|
|
215
|
+
* Optional capability: a prompt is on its way, so the service verdict it will
|
|
216
|
+
* be decided by should be current.
|
|
217
|
+
*
|
|
218
|
+
* A hook never waits on a service with a failure on record, and is never the
|
|
219
|
+
* request that finds out whether it answers again. After hours with nothing
|
|
220
|
+
* sent — a laptop asleep, a network that dropped — the failure is still on
|
|
221
|
+
* record at the next prompt, whose match is then not waited for however fast
|
|
222
|
+
* the service answers. Called when a session starts, before its first prompt:
|
|
223
|
+
* it checks in the background, and sends nothing while no failure is on record.
|
|
224
|
+
*/
|
|
225
|
+
export interface ServiceProber {
|
|
226
|
+
probeIfFailing(): void;
|
|
227
|
+
}
|
|
228
|
+
export declare function isServiceProber(r: Recorder): r is Recorder & ServiceProber;
|
|
204
229
|
/** What a ReAct iteration sends to find a scenario by its intent (fallbk.md §Runner 4). */
|
|
205
230
|
export interface IntentMatchRequest {
|
|
206
231
|
/**
|
package/dist/record/recorder.js
CHANGED
|
@@ -14,6 +14,9 @@ export function isMatchAware(r) {
|
|
|
14
14
|
export function isRunCreationAware(r) {
|
|
15
15
|
return typeof r.runCreated === "function";
|
|
16
16
|
}
|
|
17
|
+
export function isServiceProber(r) {
|
|
18
|
+
return typeof r.probeIfFailing === "function";
|
|
19
|
+
}
|
|
17
20
|
export function isIntentMatcher(r) {
|
|
18
21
|
return typeof r.matchIntent === "function";
|
|
19
22
|
}
|
|
@@ -43,8 +43,10 @@
|
|
|
43
43
|
* SIMILAR-PROMPT HITS. `POST /recordings/runs` may answer `{ id: null, matched }`,
|
|
44
44
|
* meaning the server recognised the prompt and did **not** create a run. Every
|
|
45
45
|
* later step post for that id would then 404, so callers must stop recording —
|
|
46
|
-
* {@link RemoteRecorder.
|
|
47
|
-
*
|
|
46
|
+
* {@link RemoteRecorder.settledMatch} is how they find out, whenever the answer
|
|
47
|
+
* lands ({@link RemoteRecorder.getMatch} is only what a hook may wait for) — and
|
|
48
|
+
* this recorder skips them itself too, for the caller that stopped waiting
|
|
49
|
+
* before the answer came.
|
|
48
50
|
*/
|
|
49
51
|
import { type AuthSession } from "../auth/client.js";
|
|
50
52
|
import { type ServiceHealth } from "../util/service-health.js";
|
|
@@ -187,11 +189,34 @@ export declare class RemoteRecorder implements Recorder, RecorderHealthReporter
|
|
|
187
189
|
matchIntent(req: IntentMatchRequest, opts?: IntentMatchOptions): Promise<IntentMatchAnswer | null>;
|
|
188
190
|
/**
|
|
189
191
|
* The similar-prompt match for a run's create call (null = fresh run). Like
|
|
190
|
-
* {@link runCreated}, answers null at once while
|
|
191
|
-
*
|
|
192
|
-
*
|
|
192
|
+
* {@link runCreated}, answers null at once while a failure is on record —
|
|
193
|
+
* in a cool-down, and after it until an answer clears it, however long ago
|
|
194
|
+
* it was — and as soon as the create's first attempt fails: the first prompt
|
|
195
|
+
* of an outage is not held for the retry's sleep (up to 2 s behind a
|
|
196
|
+
* Retry-After). The create goes on, and its own answer is
|
|
197
|
+
* {@link settledMatch}'s.
|
|
193
198
|
*/
|
|
194
199
|
getMatch(runId: string): Promise<RunMatch | null>;
|
|
200
|
+
/**
|
|
201
|
+
* The create's own match (null = fresh run), whenever it lands — after the
|
|
202
|
+
* cool-down and the retry, if it took them — however {@link getMatch}
|
|
203
|
+
* answered the hook. A hook told "no match" at once, because a failure was
|
|
204
|
+
* on record or the first attempt failed, may be about a prompt the service
|
|
205
|
+
* matched all the same: it then handed out the scenario and issued a ticket,
|
|
206
|
+
* and the turn owes that ticket a report. Never for a hook to wait on. Null
|
|
207
|
+
* for an id this recorder never opened.
|
|
208
|
+
*/
|
|
209
|
+
settledMatch(runId: string): Promise<RunMatch | null>;
|
|
210
|
+
/**
|
|
211
|
+
* A prompt is on its way (a session just started). While a failure is on
|
|
212
|
+
* record no hook waits on the service, and none is the request that finds
|
|
213
|
+
* out whether it answers again ({@link mayHotWait}). After hours with
|
|
214
|
+
* nothing sent, the failure is still on record at the first prompt, and its
|
|
215
|
+
* match is not waited for however fast the service answers — so find out
|
|
216
|
+
* now: one background `GET /health`, once the cool-down is over. Nothing
|
|
217
|
+
* while no failure is on record, during a cool-down, or with a probe out.
|
|
218
|
+
*/
|
|
219
|
+
probeIfFailing(): void;
|
|
195
220
|
recordToolSelected(runId: string, stepIndex: number, data: {
|
|
196
221
|
toolName: string;
|
|
197
222
|
toolInput: string;
|
|
@@ -43,8 +43,10 @@
|
|
|
43
43
|
* SIMILAR-PROMPT HITS. `POST /recordings/runs` may answer `{ id: null, matched }`,
|
|
44
44
|
* meaning the server recognised the prompt and did **not** create a run. Every
|
|
45
45
|
* later step post for that id would then 404, so callers must stop recording —
|
|
46
|
-
* {@link RemoteRecorder.
|
|
47
|
-
*
|
|
46
|
+
* {@link RemoteRecorder.settledMatch} is how they find out, whenever the answer
|
|
47
|
+
* lands ({@link RemoteRecorder.getMatch} is only what a hook may wait for) — and
|
|
48
|
+
* this recorder skips them itself too, for the caller that stopped waiting
|
|
49
|
+
* before the answer came.
|
|
48
50
|
*/
|
|
49
51
|
import { randomUUID } from "node:crypto";
|
|
50
52
|
import { statSync } from "node:fs";
|
|
@@ -479,7 +481,8 @@ export class RemoteRecorder {
|
|
|
479
481
|
}),
|
|
480
482
|
// The id is client-generated, so the service can tell a repeat. A
|
|
481
483
|
// hook waiting on this answer is let go before the retry: a match
|
|
482
|
-
// found by the retry
|
|
484
|
+
// found by the retry can no longer steer the turn, and only its
|
|
485
|
+
// ticket's report is left to it (`settledMatch`).
|
|
483
486
|
{ retryTransient: true, onRetry: () => entry.release() });
|
|
484
487
|
const b = body && typeof body === "object"
|
|
485
488
|
? body
|
|
@@ -579,9 +582,12 @@ export class RemoteRecorder {
|
|
|
579
582
|
}
|
|
580
583
|
/**
|
|
581
584
|
* The similar-prompt match for a run's create call (null = fresh run). Like
|
|
582
|
-
* {@link runCreated}, answers null at once while
|
|
583
|
-
*
|
|
584
|
-
*
|
|
585
|
+
* {@link runCreated}, answers null at once while a failure is on record —
|
|
586
|
+
* in a cool-down, and after it until an answer clears it, however long ago
|
|
587
|
+
* it was — and as soon as the create's first attempt fails: the first prompt
|
|
588
|
+
* of an outage is not held for the retry's sleep (up to 2 s behind a
|
|
589
|
+
* Retry-After). The create goes on, and its own answer is
|
|
590
|
+
* {@link settledMatch}'s.
|
|
585
591
|
*/
|
|
586
592
|
async getMatch(runId) {
|
|
587
593
|
const entry = this.runs.get(runId);
|
|
@@ -595,6 +601,39 @@ export class RemoteRecorder {
|
|
|
595
601
|
await entry.hot;
|
|
596
602
|
return entry.settled?.match ?? null;
|
|
597
603
|
}
|
|
604
|
+
/**
|
|
605
|
+
* The create's own match (null = fresh run), whenever it lands — after the
|
|
606
|
+
* cool-down and the retry, if it took them — however {@link getMatch}
|
|
607
|
+
* answered the hook. A hook told "no match" at once, because a failure was
|
|
608
|
+
* on record or the first attempt failed, may be about a prompt the service
|
|
609
|
+
* matched all the same: it then handed out the scenario and issued a ticket,
|
|
610
|
+
* and the turn owes that ticket a report. Never for a hook to wait on. Null
|
|
611
|
+
* for an id this recorder never opened.
|
|
612
|
+
*/
|
|
613
|
+
async settledMatch(runId) {
|
|
614
|
+
const entry = this.runs.get(runId);
|
|
615
|
+
if (!entry)
|
|
616
|
+
return null;
|
|
617
|
+
entry.lastUsed = Date.now();
|
|
618
|
+
if (entry.settled)
|
|
619
|
+
return entry.settled.match;
|
|
620
|
+
return (await entry.answer) ?? null;
|
|
621
|
+
}
|
|
622
|
+
/**
|
|
623
|
+
* A prompt is on its way (a session just started). While a failure is on
|
|
624
|
+
* record no hook waits on the service, and none is the request that finds
|
|
625
|
+
* out whether it answers again ({@link mayHotWait}). After hours with
|
|
626
|
+
* nothing sent, the failure is still on record at the first prompt, and its
|
|
627
|
+
* match is not waited for however fast the service answers — so find out
|
|
628
|
+
* now: one background `GET /health`, once the cool-down is over. Nothing
|
|
629
|
+
* while no failure is on record, during a cool-down, or with a probe out.
|
|
630
|
+
*/
|
|
631
|
+
probeIfFailing() {
|
|
632
|
+
this.checkCredentials();
|
|
633
|
+
if (this.state !== "recording" || this.healthyNow())
|
|
634
|
+
return;
|
|
635
|
+
this.probeSoon();
|
|
636
|
+
}
|
|
598
637
|
recordToolSelected(runId, stepIndex, data) {
|
|
599
638
|
const id = "step_" + randomUUID();
|
|
600
639
|
const toolInput = pend(data.toolInput, DEFAULT_MAX_PAYLOAD_BYTES, true);
|
|
@@ -796,6 +835,7 @@ export class RemoteRecorder {
|
|
|
796
835
|
newEntry() {
|
|
797
836
|
let resolveHot;
|
|
798
837
|
let resolveCreated;
|
|
838
|
+
let resolveAnswer;
|
|
799
839
|
const entry = {
|
|
800
840
|
userId: this.session.user.id,
|
|
801
841
|
hot: new Promise((r) => {
|
|
@@ -804,6 +844,9 @@ export class RemoteRecorder {
|
|
|
804
844
|
created: new Promise((r) => {
|
|
805
845
|
resolveCreated = r;
|
|
806
846
|
}),
|
|
847
|
+
answer: new Promise((r) => {
|
|
848
|
+
resolveAnswer = r;
|
|
849
|
+
}),
|
|
807
850
|
settle: (match, created) => {
|
|
808
851
|
if (entry.settled)
|
|
809
852
|
return;
|
|
@@ -811,9 +854,11 @@ export class RemoteRecorder {
|
|
|
811
854
|
entry.settledAt = Date.now();
|
|
812
855
|
resolveHot();
|
|
813
856
|
resolveCreated(created);
|
|
857
|
+
resolveAnswer(match);
|
|
814
858
|
// Callers that are waiting hold the promises themselves.
|
|
815
859
|
entry.hot = undefined;
|
|
816
860
|
entry.created = undefined;
|
|
861
|
+
entry.answer = undefined;
|
|
817
862
|
},
|
|
818
863
|
release: () => {
|
|
819
864
|
if (entry.settled || entry.released)
|
|
@@ -46,10 +46,12 @@ import { NullRecorder, } from "./recorder.js";
|
|
|
46
46
|
*/
|
|
47
47
|
const NEUTRAL = {
|
|
48
48
|
getMatch: () => Promise.resolve(null),
|
|
49
|
+
settledMatch: () => Promise.resolve(null),
|
|
49
50
|
runCreated: () => Promise.resolve({ created: true }),
|
|
50
51
|
matchIntent: () => Promise.resolve(null),
|
|
51
52
|
reportExecution: () => undefined,
|
|
52
53
|
backfillContext: () => undefined,
|
|
54
|
+
probeIfFailing: () => undefined,
|
|
53
55
|
};
|
|
54
56
|
/**
|
|
55
57
|
* `RemoteRecorder`'s private helpers. They are methods at runtime (TypeScript's
|
|
@@ -507,7 +507,20 @@ export type DeclineCode = "replay_disabled" | "not_ready" | "flatten_failed" | "
|
|
|
507
507
|
/** The session's host only records: its adapter asked for no plan with the prompt. */
|
|
508
508
|
| "host_records_only"
|
|
509
509
|
/** The plan names a tool the host's agent does not hold. */
|
|
510
|
-
| "host_tool_missing"
|
|
510
|
+
| "host_tool_missing"
|
|
511
|
+
/**
|
|
512
|
+
* No gate said no: the plan was good, and late. Not ready when the prompt
|
|
513
|
+
* had to be answered (`arm_timeout`), or ready just after it had been
|
|
514
|
+
* (`arm_unannounced`). Set by the control server, where a prompt's wait
|
|
515
|
+
* ends, so the report can say why a matched turn was not steered.
|
|
516
|
+
*/
|
|
517
|
+
| "arm_timeout" | "arm_unannounced"
|
|
518
|
+
/**
|
|
519
|
+
* Nothing was armed at all: the service's answer to the match landed after
|
|
520
|
+
* the prompt had been answered, or while prompts were not waiting on it
|
|
521
|
+
* (the match gate). Set by the control server, like the two above.
|
|
522
|
+
*/
|
|
523
|
+
| "match_late";
|
|
511
524
|
/** What `PreToolUse` should do about this call. */
|
|
512
525
|
export type PreToolAction =
|
|
513
526
|
/** Pin the arguments and let the real tool run. */
|
|
@@ -580,6 +593,17 @@ export declare class ReplayController {
|
|
|
580
593
|
* covers both "no match" and "too slow", which are the same thing from here.
|
|
581
594
|
*/
|
|
582
595
|
awaitMatch(pending: Promise<unknown>): Promise<RunMatchLike | null>;
|
|
596
|
+
/** A matched state before any gate has judged it: nothing planned, nothing run. */
|
|
597
|
+
private blankState;
|
|
598
|
+
/**
|
|
599
|
+
* The state of a prompt match that landed too late to be used
|
|
600
|
+
* (`match_late`). No gate is run and nothing is derived — a plan could no
|
|
601
|
+
* longer be announced, and a reading nobody waits for is money spent on
|
|
602
|
+
* nothing. It exists for the one thing the hand-out still owes: its ticket's
|
|
603
|
+
* report, with the reason, so the service does not show a turn the agent did
|
|
604
|
+
* itself as one that never reported.
|
|
605
|
+
*/
|
|
606
|
+
lateMatch(match: RunMatchLike, host?: HostContext): ReplayState;
|
|
583
607
|
/**
|
|
584
608
|
* Run the gate ladder and, if every gate passes, arm a plan.
|
|
585
609
|
*
|
|
@@ -305,18 +305,9 @@ export class ReplayController {
|
|
|
305
305
|
clearTimeout(timer);
|
|
306
306
|
}
|
|
307
307
|
}
|
|
308
|
-
/**
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
* Always returns a state — a declined match still has a ticket to redeem and a
|
|
312
|
-
* baseline sample to contribute, and losing that is how a savings ledger ends
|
|
313
|
-
* up with a denominator nobody measured.
|
|
314
|
-
*
|
|
315
|
-
* `host` is given for a session whose host is not Claude Code, and is kept
|
|
316
|
-
* on the state (see {@link HostContext}).
|
|
317
|
-
*/
|
|
318
|
-
async arm(match, prompt, wrapped, armedBy = "prompt", ctx = {}, host) {
|
|
319
|
-
const state = {
|
|
308
|
+
/** A matched state before any gate has judged it: nothing planned, nothing run. */
|
|
309
|
+
blankState(match, wrapped, armedBy, host) {
|
|
310
|
+
return {
|
|
320
311
|
scenarioId: match.scenarioId,
|
|
321
312
|
ticket: match.executionTicket,
|
|
322
313
|
similarity: match.similarity,
|
|
@@ -345,6 +336,33 @@ export class ReplayController {
|
|
|
345
336
|
verified: match.verified,
|
|
346
337
|
sharedWith: [],
|
|
347
338
|
};
|
|
339
|
+
}
|
|
340
|
+
/**
|
|
341
|
+
* The state of a prompt match that landed too late to be used
|
|
342
|
+
* (`match_late`). No gate is run and nothing is derived — a plan could no
|
|
343
|
+
* longer be announced, and a reading nobody waits for is money spent on
|
|
344
|
+
* nothing. It exists for the one thing the hand-out still owes: its ticket's
|
|
345
|
+
* report, with the reason, so the service does not show a turn the agent did
|
|
346
|
+
* itself as one that never reported.
|
|
347
|
+
*/
|
|
348
|
+
lateMatch(match, host) {
|
|
349
|
+
const state = this.blankState(match, new Set(), "prompt", host);
|
|
350
|
+
state.declined = "match_late";
|
|
351
|
+
state.retired = true;
|
|
352
|
+
return state;
|
|
353
|
+
}
|
|
354
|
+
/**
|
|
355
|
+
* Run the gate ladder and, if every gate passes, arm a plan.
|
|
356
|
+
*
|
|
357
|
+
* Always returns a state — a declined match still has a ticket to redeem and a
|
|
358
|
+
* baseline sample to contribute, and losing that is how a savings ledger ends
|
|
359
|
+
* up with a denominator nobody measured.
|
|
360
|
+
*
|
|
361
|
+
* `host` is given for a session whose host is not Claude Code, and is kept
|
|
362
|
+
* on the state (see {@link HostContext}).
|
|
363
|
+
*/
|
|
364
|
+
async arm(match, prompt, wrapped, armedBy = "prompt", ctx = {}, host) {
|
|
365
|
+
const state = this.blankState(match, wrapped, armedBy, host);
|
|
348
366
|
const isSegment = state.kind === "segment";
|
|
349
367
|
const decline = (why, code) => {
|
|
350
368
|
state.declined = code;
|
package/dist/replay/derive.d.ts
CHANGED
|
@@ -60,6 +60,8 @@ export interface DeriveOptions extends DeriveContext {
|
|
|
60
60
|
prompt: string;
|
|
61
61
|
intent: string;
|
|
62
62
|
paramsObject: ParamsSchema | null;
|
|
63
|
+
/** The day on this machine. Injected by tests; defaults to {@link localDay}. */
|
|
64
|
+
today?: string;
|
|
63
65
|
/** An override that keeps the reading on this machine. Absent → the service. */
|
|
64
66
|
apiKey?: string;
|
|
65
67
|
/** Used when there is no `apiKey`. Absent too → recorded samples. */
|
|
@@ -90,6 +92,16 @@ export interface DeriveResult {
|
|
|
90
92
|
/** Settings filled from their recorded sample (R-PARAM-2). */
|
|
91
93
|
sampled: string[];
|
|
92
94
|
}
|
|
95
|
+
/**
|
|
96
|
+
* The day on this machine, `YYYY-MM-DD`, in its own time zone.
|
|
97
|
+
*
|
|
98
|
+
* Sent to the service with every reading, because a prompt says "tomorrow" and
|
|
99
|
+
* the date is in no prompt: the service works such a value out, and it must
|
|
100
|
+
* count from the user's day. Its own day is the wrong one — a server on UTC is
|
|
101
|
+
* a day ahead of a user in Seattle for a third of the working day. Local on
|
|
102
|
+
* purpose, then, and never `toISOString`, which is UTC.
|
|
103
|
+
*/
|
|
104
|
+
export declare function localDay(now?: Date): string;
|
|
93
105
|
/**
|
|
94
106
|
* First balanced `{…}` in a text blob. Models wrap JSON in prose and fences more
|
|
95
107
|
* often than they emit it bare, and a failed parse here costs a whole replay.
|
package/dist/replay/derive.js
CHANGED
|
@@ -75,6 +75,19 @@ function serviceOk(startedAt) {
|
|
|
75
75
|
const ok = serviceHealth.ok.bind(serviceHealth);
|
|
76
76
|
ok(startedAt);
|
|
77
77
|
}
|
|
78
|
+
/**
|
|
79
|
+
* The day on this machine, `YYYY-MM-DD`, in its own time zone.
|
|
80
|
+
*
|
|
81
|
+
* Sent to the service with every reading, because a prompt says "tomorrow" and
|
|
82
|
+
* the date is in no prompt: the service works such a value out, and it must
|
|
83
|
+
* count from the user's day. Its own day is the wrong one — a server on UTC is
|
|
84
|
+
* a day ahead of a user in Seattle for a third of the working day. Local on
|
|
85
|
+
* purpose, then, and never `toISOString`, which is UTC.
|
|
86
|
+
*/
|
|
87
|
+
export function localDay(now = new Date()) {
|
|
88
|
+
const two = (n) => String(n).padStart(2, "0");
|
|
89
|
+
return `${now.getFullYear()}-${two(now.getMonth() + 1)}-${two(now.getDate())}`;
|
|
90
|
+
}
|
|
78
91
|
/** Every parameter at its recorded sample value. The zero-cost fallback. */
|
|
79
92
|
function sampleValues(schema) {
|
|
80
93
|
const out = {};
|
|
@@ -251,6 +264,15 @@ export async function deriveParameters(opts) {
|
|
|
251
264
|
const missing = [];
|
|
252
265
|
const sampled = [];
|
|
253
266
|
for (const key of keys) {
|
|
267
|
+
// A value worked out from the day ("tomorrow") cannot be read here: this
|
|
268
|
+
// copy of the text does not tell the model the day, and what comes back
|
|
269
|
+
// for such a value is then the recorded date or a guess. The service's
|
|
270
|
+
// reading does work it out; with a key of its own the value is missing.
|
|
271
|
+
if (isTarget(schema, key) && schema[key].from === "today") {
|
|
272
|
+
parsed[key] = null;
|
|
273
|
+
missing.push(key);
|
|
274
|
+
continue;
|
|
275
|
+
}
|
|
254
276
|
if (parsed[key] !== null && parsed[key] !== undefined)
|
|
255
277
|
continue;
|
|
256
278
|
if (isTarget(schema, key)) {
|
|
@@ -318,6 +340,9 @@ async function deriveViaService(opts, service, schema, keys) {
|
|
|
318
340
|
prompt: outgoingPrompt(opts.prompt),
|
|
319
341
|
liveCall: context.liveCall,
|
|
320
342
|
recentResults: context.recentResults,
|
|
343
|
+
// What "tomorrow" is counted from. A service that does not know the
|
|
344
|
+
// field ignores it.
|
|
345
|
+
today: opts.today ?? localDay(),
|
|
321
346
|
}),
|
|
322
347
|
signal: opts.signal ?? timeoutSignal(),
|
|
323
348
|
});
|
package/dist/replay/types.d.ts
CHANGED
|
@@ -138,6 +138,13 @@ export type ParamsSchema = Record<string, {
|
|
|
138
138
|
sampleValue: unknown;
|
|
139
139
|
description: string;
|
|
140
140
|
kind?: ParameterKind;
|
|
141
|
+
/**
|
|
142
|
+
* Where the recorded value came from, when the service calculated the plan
|
|
143
|
+
* beside its request. Only `today` is read here: a value worked out from
|
|
144
|
+
* the day the request is made on ("tomorrow"), which the service works out
|
|
145
|
+
* again from the day this runner sends (replay/derive.ts, `localDay`).
|
|
146
|
+
*/
|
|
147
|
+
from?: string;
|
|
141
148
|
}>;
|
|
142
149
|
/** Whether this schema has anything the runner may not guess (R-PARAM-4). */
|
|
143
150
|
export declare function hasTargets(schema: ParamsSchema | null | undefined): boolean;
|
package/docs/calculatedReplay.md
CHANGED
|
@@ -194,7 +194,7 @@ Every gate that declines logs **one** `replay.decision` line carrying the reason
|
|
|
194
194
|
|
|
195
195
|
| # | Gate | Declines when | Outcome floor |
|
|
196
196
|
|---|---|---|---|
|
|
197
|
-
| 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched) |
|
|
197
|
+
| 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched; unless the create's own answer is a hit after all: `match_late`, below) |
|
|
198
198
|
| 2 | **Replay enabled** | `BIR_REPLAY === "0"`, or, with it unset, the project's stored switch is off (`bir replay off`, or a `bir install` without `--replay`). In a session of another host, also when the run's adapter asked for no plan with its prompt (`host_records_only`, §7.4) | `not_steered` |
|
|
199
199
|
| 3 | **Scenario ready** | `scenario == null`, `state !== "ready"`, or `steps.length === 0` | `not_steered` |
|
|
200
200
|
| 4 | ~~**Similarity**~~ | Retired: a match is the decision to use the scenario (below) | — |
|
|
@@ -203,9 +203,15 @@ Every gate that declines logs **one** `replay.decision` line carrying the reason
|
|
|
203
203
|
Past the table, a plan can still be refused before the model hears of it, each refusal with its own
|
|
204
204
|
`replay.decision` code. The match is not waited on at all while the service is cooling down after a
|
|
205
205
|
failure (`service_unhealthy`), or while prompts are holding off a service that answered matches
|
|
206
|
-
slowly (`match_slow`, the match gate, §10).
|
|
206
|
+
slowly (`match_slow`, the match gate, §10). Nor after the cool-down, while the failure is still on
|
|
207
|
+
record and no answer has cleared it: the recorder answers the prompt "no match" at once, and the
|
|
208
|
+
create goes on without it. A match or a plan that is not ready
|
|
207
209
|
inside the prompt's budget (§10) is dropped (`match_timeout`, `arm_timeout`), and one that lands
|
|
208
|
-
after the prompt was answered is retired unannounced (`match_late`, `arm_unannounced`).
|
|
210
|
+
after the prompt was answered is retired unannounced (`match_late`, `arm_unannounced`). The turn's
|
|
211
|
+
report carries that code as its decline, `match_late` included: nothing is armed or derived for a
|
|
212
|
+
late match, but the ticket of its hand-out is still reported, so the service knows why the turn was
|
|
213
|
+
not steered. That holds for a match the prompt never waited for, too: the turn follows the create's
|
|
214
|
+
own answer (`settledMatch`), not the "no match" the prompt was given. A `direct`
|
|
209
215
|
plan is declined when no running `bir-scenario` has said it serves `run_scenario`
|
|
210
216
|
(`no_delivery_tool`, §6.3). In plan mode (`permission_mode: "plan"`) nothing is announced at all.
|
|
211
217
|
In a session of another host (§7.4) a plan that armed is taken back, before any step of it runs,
|
|
@@ -878,7 +884,7 @@ The hook is on the user's critical path. Every await here is a stall the user wa
|
|
|
878
884
|
|
|
879
885
|
| Moment | Route | Hook timeout | Answered by | What may be awaited | Budget |
|
|
880
886
|
|---|---|---|---|---|---|
|
|
881
|
-
| `UserPromptSubmit` | `/session/prompt` | 10 s |
|
|
887
|
+
| `UserPromptSubmit` | `/session/prompt` | 10 s | 8.5 s | the match; for a scenario with a target, its derivation; a steer plan's argument preview | match `BIR_MATCH_BUDGET_MS` = **3000**; match and arming together ≤ **7.5 s**; derivation ≤ `BIR_HOOK_DERIVE_BUDGET_MS` = **8000**, cut short by the 7.5 s |
|
|
882
888
|
| `PreToolUse` | `/tool/pre` | 15 s | 12 s | `plan.ready()` (derivation); on a divergence, the remaining steps | derivation ≤ min(`BIR_DERIVE_BUDGET_MS`, `BIR_HOOK_DERIVE_BUDGET_MS`) = **8000**; all replay work ≤ `BIR_PRETOOL_REPLAY_BUDGET_MS` = **9000** (held at ≤ 10 000) |
|
|
883
889
|
| `run_scenario` | `/scenario/run` | `bir-scenario`'s own, **290 s** (`BIR_SCENARIO_RUN_TIMEOUT_MS`) | — | `ready()` + N steps | derivation `BIR_DERIVE_BUDGET_MS` = **8000**; plan `BIR_REPLAY_PLAN_BUDGET_MS` = **100000** (`BIR_REPLAY_BUDGET_MS` overrides it where set); per step `BIR_STEP_TIMEOUT_MS` = **60000**, clipped to what the plan has left |
|
|
884
890
|
| `PostToolUse` | `/tool/post` | 5 s | 2 s | nothing | — |
|
|
@@ -915,7 +921,13 @@ minutes, while its steps keep starting. `bir replay` is not held to the ceiling.
|
|
|
915
921
|
`BIR_MATCH_SLOW_MS` (**2000**), or not at all, also makes prompts stop waiting on the match for 5
|
|
916
922
|
minutes, doubling up to 30 while the service stays slow; a fast match ends the hold
|
|
917
923
|
(`replay.match_gate`). A healthy service answers in 300–450 ms, so past 2 s waiting would add
|
|
918
|
-
that much to every prompt. The match is still asked meanwhile, so a hit still stops the recording.
|
|
924
|
+
that much to every prompt. The match is still asked meanwhile, so a hit still stops the recording.
|
|
925
|
+
**A failure on record:** a prompt never waits on a service whose last request failed, until an
|
|
926
|
+
answer clears that failure, and a prompt is never the request that finds out. After hours with
|
|
927
|
+
nothing sent (a machine asleep, a network that dropped) the failure is still on record, so a
|
|
928
|
+
session's start checks it in the background (`GET /health`) before the first prompt arrives. A match
|
|
929
|
+
that lands after a prompt went ahead without it is `match_late`, and its ticket is reported as the
|
|
930
|
+
turn's decline. Derivation budget expired →
|
|
919
931
|
settings take their recorded `sampleValue`s and targets stay `null`, never `{}`; a plan that needs a
|
|
920
932
|
missing target, or whose `paramsLogic` throws, hands over instead of running (in steer mode the
|
|
921
933
|
note replaces the call, in direct mode `run_scenario` answers with it). The derivation request
|
|
@@ -1356,8 +1368,9 @@ fail because of BaseInstRunner.**
|
|
|
1356
1368
|
|
|
1357
1369
|
| Failure | Behaviour |
|
|
1358
1370
|
|---|---|
|
|
1359
|
-
| `getMatch()` slower than the match budget | No plan. Ordinary turn. One `replay.decision` line (`match_timeout`); a match that lands later is `match_late` and never steers. A slow match also makes prompts stop waiting on it for a while (`replay.match_gate`, §10) |
|
|
1371
|
+
| `getMatch()` slower than the match budget | No plan. Ordinary turn. One `replay.decision` line (`match_timeout`); a match that lands later is `match_late` and never steers; while the turn is open it is reported to the service as that turn's decline. A slow match also makes prompts stop waiting on it for a while (`replay.match_gate`, §10) |
|
|
1360
1372
|
| The service is cooling down after a failure | Nothing waits on it: no match wait (`service_unhealthy`), no derivation wait, no intent probe. Ordinary turn |
|
|
1373
|
+
| A failure is still on record after the cool-down, with nothing sent since | No match wait and no intent probe: the prompt is answered "no match" at once and the create goes on. A hit that lands is `match_late`, and is reported as the turn's decline while the turn is open. A session's start checks the service in the background (`GET /health`), so an old failure does not decide its first prompt |
|
|
1361
1374
|
| No derivation key here (`BIR_DERIVE_API_KEY`; the background recorder ignores `ANTHROPIC_API_KEY` unless `BIR_USE_ANTHROPIC_API_KEY=1`) | The service derives instead (segmented.md R-PARAM-5). Signed out as well: settings take their recorded `sampleValue`s, a scenario with a target declines `no_derive_key`, `deriveCostUsd: 0` |
|
|
1362
1375
|
| The service declines, is slow, or cannot be reached | The same, with its reason carried into the `replay.derived` and `replay.decision` lines. Never a throw |
|
|
1363
1376
|
| Derivation call fails or times out | Settings take their samples and targets stay `null`, never `{}`; a plan that needs a missing target hands over instead of running. Plus a `replay.derive_failed` or `replay.derive_timeout` line |
|
|
@@ -638,7 +638,7 @@ Code's pool, as before.
|
|
|
638
638
|
| No `run.matched`, ever | Prompt below the server's `SIMILARITY_THRESHOLD` | Rephrase closer, or lower it server-side. Confirm the original run is in the list — a sub-`RECORDING_MIN_ACTIONS` run is never embedded |
|
|
639
639
|
| `run.matched` but no `plan.armed` | A gate declined | Read the `replay.decision` line's `why=`. Ranked by frequency: replay off — `bir replay off`, `BIR_REPLAY=0`, or never switched on here (a `bir install` without `--replay`; `bir replay status` says which) · scenario not `ready` · similarity below threshold · no step is executable |
|
|
640
640
|
| `replay.decision … code=match_timeout` or `code=service_unhealthy` on many prompts | The service answered the match slower than `BIR_MATCH_BUDGET_MS` (3 s), or is failing and in its cool-down | Nothing breaks: those turns run normally, and after a slow match prompts stop waiting on it for a while by themselves (`replay.match_gate`). `bir doctor` shows the service's state. Raise `BIR_MATCH_BUDGET_MS` only if the service is reliably just over it — every prompt pays it |
|
|
641
|
-
| `replay.decision … code=arm_timeout`, `code=missing_target … derivation did not answer within …ms`, or `code=no_derive_key why="service: …"`, on a prompt that names its values | The reading of the prompt's values (the derivation) was too slow for the hook, or failed on the service. A hook waits `BIR_HOOK_DERIVE_BUDGET_MS` (8 s) for it, and the prompt hook never past
|
|
641
|
+
| `replay.decision … code=arm_timeout`, `code=missing_target … derivation did not answer within …ms`, or `code=no_derive_key why="service: …"`, on a prompt that names its values | The reading of the prompt's values (the derivation) was too slow for the hook, or failed on the service. A hook waits `BIR_HOOK_DERIVE_BUDGET_MS` (8 s) for it, and the prompt hook never past 7.5 s from the prompt, match included; the service gives one reading `RUNNER_DERIVE_TIMEOUT_MS` (6 s) and then answers `service: error` or `service: timeout`, so its answer arrives inside the prompt's wait either way | Nothing to change in the prompt, and nothing to sign in to: that turn ran normally. The `replay.derived` line that follows says when the reading landed and what it cost. If it is slow every time, the service's reading model is the cause; `BIR_DERIVE_API_KEY` where Claude Code starts moves the reading to this machine |
|
|
642
642
|
| `replay.decision … code=no_delivery_tool` | A direct plan, and no running `bir` server has told the recorder it serves `run_scenario`: the server is missing, disabled or failed in `/mcp`, was not added (a subdirectory or worktree of a repository), or the session was started with `BIR_OFF=1` | `/mcp` shows whether `bir` is connected. `bir install --replay` adds it, in the project's own directory; then start a new session |
|
|
643
643
|
| `replay.vetoed … the model's first call was …, not run_scenario` | A direct plan, and the model reached for another tool first — usually because it has no `run_scenario` | As for `no_delivery_tool`. Nothing ran; the model did the task |
|
|
644
644
|
| `replay.decision … code=host_records_only` | A LangChain application (§6.5) whose adapter asked for no plan with its prompt: it only records. `bir investigate` gives the cause: the runner in that application's folder is older than replay for LangChain needs, so the LangChain package only records | In `bir investigate`'s words: nothing to do while replay for LangChain is not released. After that: update the runner. Until then the model does the task, and the run counts as a sample of the usual cost |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@basein/runner",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.12",
|
|
4
4
|
"description": "A recording MCP proxy: sits between any MCP client and its MCP servers, executes each call on the client's behalf, and records the run as a reusable BaseIn scenario.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/production.md
CHANGED
|
@@ -122,7 +122,7 @@ In practice: the runner must never be the reason a Claude Code session is slow,
|
|
|
122
122
|
|---|---|---|---|---|---|
|
|
123
123
|
| SessionStart: command hook `bir-hooks ensure` | the payload on stdin (≤ 2 s); finding or starting the recorder (local only, never the service); relaying the payload (≤ 2.5 s); a read-only stale-copy check (only with ≥ 3 s left) | warm about 0.1 s (Node start-up); cold about 0.7 s on Windows, once per project per boot | 11.5 s (11 s budget plus a 0.5 s hard exit). npx installs pass `--budget-ms 8000`, after npx's own start-up | `/session/start`: 2 s | 15 s (30) |
|
|
124
124
|
| SessionStart: HTTP form (older installs only; `bir install` now always writes the command hook, and `bir setup` replaces the HTTP one) | nothing | milliseconds | 2 s | 2 s | 5 s (10) |
|
|
125
|
-
| UserPromptSubmit | replay on: the similar-prompt match (≤ 3 s; skipped while the breaker is open or the match gate holds). For a matched scenario: arming, derivation (budget 8 s, cut short by this hook's
|
|
125
|
+
| UserPromptSubmit | replay on: the similar-prompt match (≤ 3 s; skipped while the breaker is open or the match gate holds). For a matched scenario: arming, derivation (budget 8 s, cut short by this hook's 7.5 s) and the directive's argument preview (≤ 1.2 s), all inside 7.5 s. Replay off: nothing | about 17 ms with an instant local service; plus one service round trip (about 0.3-0.5 s) with replay on | 7.5 s, only for a matched scenario whose values are still being read; 3 s on the first prompt of a hang | 8.5 s | 10 s (15) |
|
|
126
126
|
| PreToolUse | nothing, unless a replay plan steers this turn. Then: in-hook steps (≤ 9 s, `BIR_PRETOOL_REPLAY_BUDGET_MS`), derivation at the first call (≤ 8 s), the previous pinned step (≤ 1.75 s), and after a hand-over the new fragment run's create (≤ 0.3 s, breaker-gated). Intent arming only when opted in (≤ 0.5 s probe + 1.5 s arming) | 2-5 ms | 12 s, replay only | 12 s | 15 s (30) |
|
|
127
127
|
| PostToolUse, PostToolUseFailure | nothing: the proxy's report is joined in the background | 1-2 ms | 2 s | 2 s | 5 s (30) |
|
|
128
128
|
| SubagentStart, SubagentStop | nothing | not measured (bookkeeping only) | 2 s | 2 s | 3 s (5) |
|
|
@@ -135,7 +135,7 @@ Hard deadlines are in `HOOK_DEADLINE_MS` (`src/control/server.ts`). The timeouts
|
|
|
135
135
|
|
|
136
136
|
- **What setup does.** It signs in, gives the folder an entry of its own in `installed.json` (a fixed port, a token and `hosts: ["langchain"]`), stores replay on (off with `--no-replay`), starts the recorder in the background and prints the two lines of Python to add to the agent.
|
|
137
137
|
- **What it leaves alone.** It writes nothing under `.claude/` and nothing into `~/.claude.json`: no hooks, no wrapped servers, no `bir` MCP server. A folder that is already set up for Claude Code keeps that set-up as it is, and only its entry notes the host.
|
|
138
|
-
- **Deadlines.** The server deadlines above are set per route, so they hold for the adapter's requests too: `/session/prompt`
|
|
138
|
+
- **Deadlines.** The server deadlines above are set per route, so they hold for the adapter's requests too: `/session/prompt` 8.5 s, `/tool/pre` 12 s, `/tool/post` and `/session/stop` 2 s. The timeouts `bir install` writes are Claude Code's own.
|
|
139
139
|
|
|
140
140
|
The fields the adapter adds to each route, and what the recorder does for such a session, are in `docs/BaseInstRunner.md`, Phase 5. The how-to is `langChainGuide.md` in the repository's root; it is not in the npm package. The runner's part is all in this package. The Python package is not on PyPI, and the service's part is built and not deployed yet.
|
|
141
141
|
|
|
@@ -370,7 +370,7 @@ Over 2-minute runs with one turn every 8 s (92 hooks):
|
|
|
370
370
|
|
|
371
371
|
**Deadlines and fail-open answers.**
|
|
372
372
|
|
|
373
|
-
- **Deadlines.** Hard per-route deadlines apply: start 2 s, prompt
|
|
373
|
+
- **Deadlines.** Hard per-route deadlines apply: start 2 s, prompt 8.5 s, pre 12 s, post, agent and stop 2 s, end 1 s. Past a deadline the route answers `{}`, or an honest note mid-replay.
|
|
374
374
|
- **Fail-open answers.** Stale or wrong tokens, unknown routes, non-POST requests, bodies over 64 MB (read and discarded) and non-object bodies all get `200 {}`. They are counted in `/health.hooks`. Non-hook routes keep 401, 404 and 405.
|
|
375
375
|
- **Event name.** Every answer's `hookEventName` matches the event that fired. The final review found a PostToolUseFailure answered as PostToolUse, which Claude Code rejects with a visible hook error that dumps the runner's JSON.
|
|
376
376
|
- **Token.** The project's current token is read from `installed.json`: re-read at most every 3 s, and only when the file changed. A reinstall on the same port needs no recorder restart.
|
|
@@ -379,7 +379,7 @@ Over 2-minute runs with one turn every 8 s (92 hooks):
|
|
|
379
379
|
|
|
380
380
|
- The match waits up to 3 s, and only while the breaker allows it.
|
|
381
381
|
- The match gate: over 2 s, prompts stop waiting for 5 min, doubling to 30 min.
|
|
382
|
-
- Arming a matched scenario, including derivation and the directive's argument preview, stays inside 6 s.
|
|
382
|
+
- Arming a matched scenario, including derivation and the directive's argument preview, stays inside 7.5 s. That is longer than the service may take to read a prompt's values (6 s, its `RUNNER_DERIVE_TIMEOUT_MS`) plus the match and the round trip: at 6 s the hook gave up before the service could answer, and a good plan was thrown away (`arm_timeout`).
|
|
383
383
|
- A plan that is ready only after the prompt was answered is retired unannounced, so it never steers (`replay.decision code=arm_unannounced` or `arm_timeout`).
|
|
384
384
|
- With replay off, the prompt never waits.
|
|
385
385
|
- The prompt after an interrupted turn, and SessionEnd, seal synchronously and drain in the background. They used to wait for the whole backlog to reach the service (0.5 s healthy, over 30 s hung).
|
|
@@ -601,7 +601,7 @@ Truncation is always marked with the original size.
|
|
|
601
601
|
|
|
602
602
|
**Budgets.**
|
|
603
603
|
|
|
604
|
-
- The hook derivation wait is 8 s (the prompt hook stops at its own
|
|
604
|
+
- The hook derivation wait is 8 s (the prompt hook stops at its own 7.5 s). All replay work in one PreToolUse is capped at 9 s (maximum 10 s).
|
|
605
605
|
- The direct plan budget is 100 s (was 120 s). Direct runs are clamped so that derivation, plan and slack stay within 270 s, under `run_scenario`'s 290 s client timeout.
|
|
606
606
|
- The worst case by budget is 120 s: 8 s of derivation, 100 s of plan, and 12 s of slack.
|
|
607
607
|
|
|
@@ -909,7 +909,7 @@ Lines read `[bir] <ISO time> <event> key=value …`, with each field clipped at
|
|
|
909
909
|
|
|
910
910
|
| Topic | Lines and what they mean |
|
|
911
911
|
|---|---|
|
|
912
|
-
| Service health | `service.degraded` (first failure: reason, cool-down), `service.recovered`. `replay.decision code=service_unhealthy` (prompt did not wait: breaker open), `code=match_timeout` (prompt waited its budget), `code=match_late` (the answer landed after the prompt), `code=match_slow` (verbose: gate holding). `replay.match_gate state=holding\|open`. |
|
|
912
|
+
| Service health | `service.degraded` (first failure: reason, cool-down), `service.recovered`. `replay.decision code=service_unhealthy` (prompt did not wait: breaker open), `code=match_timeout` (prompt waited its budget), `code=match_late` (the answer landed after the prompt, including one the prompt never waited for because a failure was still on record), `code=match_slow` (verbose: gate holding). `replay.match_gate state=holding\|open`. |
|
|
913
913
|
| Recording backlog | `recorder.backlog_full` / `recorder.backlog_recovered` (oldest steps dropped). `recorder.run_incomplete` (a run lost steps). `recorder.run_abandoned` (`reason=service_down`: a create given up after a minute of cool-down). `recorder.flush_deferred`, `recorder.flush_incomplete`, `recorder.send_failed` (once per run), `recorder.matched_after_retry`, `run.lossy`. |
|
|
914
914
|
| Sign-in | `auth.ok`, `auth.settling` (plain-http address being checked for https), `auth.refresh_deferred`, `auth.refresh_refused`, `auth.recovered`. `recorder.enabled`, `recorder.signed_out`, `recorder.disabled`, `recorder.paused` / `recorder.resumed`, `recorder.account_changed`, `recorder.report_dropped`, `derive.api_key_ignored`. |
|
|
915
915
|
| Hooks | `control.listening` (port, `recorder=<state>`, `correlation=`). `control.correlation_allow` (auto-approval opted in). `control.hook_deadline`, `control.hook_unauthorized` (a token from another install), `control.hook_unknown`, `control.hook_oversize`, `control.hook_event_mismatch`, `control.route_failed`, `control.drain_abandoned`. |
|