@basein/runner 0.3.11 → 0.3.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/bin/bir-scenario.d.ts +8 -1
- package/dist/bin/bir-scenario.js +10 -2
- package/dist/bin/bir.js +4 -0
- package/dist/bin/investigate.d.ts +10 -0
- package/dist/bin/investigate.js +37 -9
- package/dist/control/client.d.ts +6 -0
- package/dist/control/server.d.ts +42 -1
- package/dist/control/server.js +134 -38
- package/dist/proxy/session.js +1 -0
- package/dist/record/prompt-text.d.ts +32 -0
- package/dist/record/prompt-text.js +65 -0
- package/dist/record/recorder.d.ts +25 -0
- package/dist/record/recorder.js +3 -0
- package/dist/record/remote-recorder.d.ts +30 -5
- package/dist/record/remote-recorder.js +51 -6
- package/dist/record/switchable-recorder.js +2 -0
- package/dist/replay/controller.d.ts +18 -1
- package/dist/replay/controller.js +30 -12
- package/dist/replay/derive.d.ts +12 -0
- package/dist/replay/derive.js +25 -0
- package/dist/replay/executor.d.ts +26 -5
- package/dist/replay/executor.js +40 -5
- package/dist/replay/types.d.ts +7 -0
- package/docs/calculatedReplay.md +31 -10
- package/docs/calculatedReplayGuide.md +8 -1
- package/package.json +1 -1
- package/production.md +3 -1
package/docs/calculatedReplay.md
CHANGED
|
@@ -126,9 +126,12 @@ plan, and keeps the stop.
|
|
|
126
126
|
|
|
127
127
|
Server-side, and worth stating because the threshold in §4 is a judgement about it:
|
|
128
128
|
|
|
129
|
-
- The embedding is of the run's **`input
|
|
130
|
-
|
|
131
|
-
|
|
129
|
+
- The embedding is of the run's **`input`**, computed by `Xenova/all-MiniLM-L6-v2`
|
|
130
|
+
(`EMBEDDING_MODEL`), mean-pooled and normalized (`recordings/embeddings.ts`). The `input` is the
|
|
131
|
+
user's prompt as the runner cleaned it: Claude Code's own blocks in front of it removed and pasted
|
|
132
|
+
text unwrapped (fx121.md). Another host's prompt goes as it was sent. The model reads the first
|
|
133
|
+
512 tokens and drops the rest. The Claude in Chrome guide alone is 1,008, so a runner that sent it
|
|
134
|
+
in front of the prompt had only the guide embedded.
|
|
132
135
|
- Candidates are **the caller's own canonical runs only** (`listUserRunEmbeddings(userId)`);
|
|
133
136
|
duplicates and sub-threshold runs are excluded from the pool at record time.
|
|
134
137
|
- **Inside one host** — on the service as built, which is not deployed yet. A run created with
|
|
@@ -194,7 +197,7 @@ Every gate that declines logs **one** `replay.decision` line carrying the reason
|
|
|
194
197
|
|
|
195
198
|
| # | Gate | Declines when | Outcome floor |
|
|
196
199
|
|---|---|---|---|
|
|
197
|
-
| 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched) |
|
|
200
|
+
| 1 | **A match arrived** | `getMatch()` resolved `null`, or the budget expired (§10) | (no report — nothing matched; unless the create's own answer is a hit after all: `match_late`, below) |
|
|
198
201
|
| 2 | **Replay enabled** | `BIR_REPLAY === "0"`, or, with it unset, the project's stored switch is off (`bir replay off`, or a `bir install` without `--replay`). In a session of another host, also when the run's adapter asked for no plan with its prompt (`host_records_only`, §7.4) | `not_steered` |
|
|
199
202
|
| 3 | **Scenario ready** | `scenario == null`, `state !== "ready"`, or `steps.length === 0` | `not_steered` |
|
|
200
203
|
| 4 | ~~**Similarity**~~ | Retired: a match is the decision to use the scenario (below) | — |
|
|
@@ -203,9 +206,15 @@ Every gate that declines logs **one** `replay.decision` line carrying the reason
|
|
|
203
206
|
Past the table, a plan can still be refused before the model hears of it, each refusal with its own
|
|
204
207
|
`replay.decision` code. The match is not waited on at all while the service is cooling down after a
|
|
205
208
|
failure (`service_unhealthy`), or while prompts are holding off a service that answered matches
|
|
206
|
-
slowly (`match_slow`, the match gate, §10).
|
|
209
|
+
slowly (`match_slow`, the match gate, §10). Nor after the cool-down, while the failure is still on
|
|
210
|
+
record and no answer has cleared it: the recorder answers the prompt "no match" at once, and the
|
|
211
|
+
create goes on without it. A match or a plan that is not ready
|
|
207
212
|
inside the prompt's budget (§10) is dropped (`match_timeout`, `arm_timeout`), and one that lands
|
|
208
|
-
after the prompt was answered is retired unannounced (`match_late`, `arm_unannounced`).
|
|
213
|
+
after the prompt was answered is retired unannounced (`match_late`, `arm_unannounced`). The turn's
|
|
214
|
+
report carries that code as its decline, `match_late` included: nothing is armed or derived for a
|
|
215
|
+
late match, but the ticket of its hand-out is still reported, so the service knows why the turn was
|
|
216
|
+
not steered. That holds for a match the prompt never waited for, too: the turn follows the create's
|
|
217
|
+
own answer (`settledMatch`), not the "no match" the prompt was given. A `direct`
|
|
209
218
|
plan is declined when no running `bir-scenario` has said it serves `run_scenario`
|
|
210
219
|
(`no_delivery_tool`, §6.3). In plan mode (`permission_mode: "plan"`) nothing is announced at all.
|
|
211
220
|
In a session of another host (§7.4) a plan that armed is taken back, before any step of it runs,
|
|
@@ -915,7 +924,13 @@ minutes, while its steps keep starting. `bir replay` is not held to the ceiling.
|
|
|
915
924
|
`BIR_MATCH_SLOW_MS` (**2000**), or not at all, also makes prompts stop waiting on the match for 5
|
|
916
925
|
minutes, doubling up to 30 while the service stays slow; a fast match ends the hold
|
|
917
926
|
(`replay.match_gate`). A healthy service answers in 300–450 ms, so past 2 s waiting would add
|
|
918
|
-
that much to every prompt. The match is still asked meanwhile, so a hit still stops the recording.
|
|
927
|
+
that much to every prompt. The match is still asked meanwhile, so a hit still stops the recording.
|
|
928
|
+
**A failure on record:** a prompt never waits on a service whose last request failed, until an
|
|
929
|
+
answer clears that failure, and a prompt is never the request that finds out. After hours with
|
|
930
|
+
nothing sent (a machine asleep, a network that dropped) the failure is still on record, so a
|
|
931
|
+
session's start checks it in the background (`GET /health`) before the first prompt arrives. A match
|
|
932
|
+
that lands after a prompt went ahead without it is `match_late`, and its ticket is reported as the
|
|
933
|
+
turn's decline. Derivation budget expired →
|
|
919
934
|
settings take their recorded `sampleValue`s and targets stay `null`, never `{}`; a plan that needs a
|
|
920
935
|
missing target, or whose `paramsLogic` throws, hands over instead of running (in steer mode the
|
|
921
936
|
note replaces the call, in direct mode `run_scenario` answers with it). The derivation request
|
|
@@ -1356,8 +1371,9 @@ fail because of BaseInstRunner.**
|
|
|
1356
1371
|
|
|
1357
1372
|
| Failure | Behaviour |
|
|
1358
1373
|
|---|---|
|
|
1359
|
-
| `getMatch()` slower than the match budget | No plan. Ordinary turn. One `replay.decision` line (`match_timeout`); a match that lands later is `match_late` and never steers. A slow match also makes prompts stop waiting on it for a while (`replay.match_gate`, §10) |
|
|
1374
|
+
| `getMatch()` slower than the match budget | No plan. Ordinary turn. One `replay.decision` line (`match_timeout`); a match that lands later is `match_late` and never steers; while the turn is open it is reported to the service as that turn's decline. A slow match also makes prompts stop waiting on it for a while (`replay.match_gate`, §10) |
|
|
1360
1375
|
| The service is cooling down after a failure | Nothing waits on it: no match wait (`service_unhealthy`), no derivation wait, no intent probe. Ordinary turn |
|
|
1376
|
+
| A failure is still on record after the cool-down, with nothing sent since | No match wait and no intent probe: the prompt is answered "no match" at once and the create goes on. A hit that lands is `match_late`, and is reported as the turn's decline while the turn is open. A session's start checks the service in the background (`GET /health`), so an old failure does not decide its first prompt |
|
|
1361
1377
|
| No derivation key here (`BIR_DERIVE_API_KEY`; the background recorder ignores `ANTHROPIC_API_KEY` unless `BIR_USE_ANTHROPIC_API_KEY=1`) | The service derives instead (segmented.md R-PARAM-5). Signed out as well: settings take their recorded `sampleValue`s, a scenario with a target declines `no_derive_key`, `deriveCostUsd: 0` |
|
|
1362
1378
|
| The service declines, is slow, or cannot be reached | The same, with its reason carried into the `replay.derived` and `replay.decision` lines. Never a throw |
|
|
1363
1379
|
| Derivation call fails or times out | Settings take their samples and targets stay `null`, never `{}`; a plan that needs a missing target hands over instead of running. Plus a `replay.derive_failed` or `replay.derive_timeout` line |
|
|
@@ -1370,7 +1386,7 @@ fail because of BaseInstRunner.**
|
|
|
1370
1386
|
| A call arrives while a pinned step is still running | It waits briefly for the pin to thread; still beside it, the plan hands over (`replay.concurrent_call`), `fell_back` or `failed` |
|
|
1371
1387
|
| A pinned step fails (`PostToolUseFailure`, an error report from its proxy) | The plan hands over; nothing after it is pinned or approved |
|
|
1372
1388
|
| A step cannot be sent: a built-in reached by a divergence, no proxy polling for its server, a server off `BIR_REPLAY_ALLOW_SERVERS` | Nothing is sent. The plan stops at that step and hands over (*"did not run … Nothing of it happened"*). With `BIR_REPLAY_RECORDED_OUTPUTS=1` its recorded output stands in instead |
|
|
1373
|
-
| Two sessions are open, and none of the server's proxies is known to be this session's | Nothing is sent: the step is not run, the plan hands over, and the model calls the tool on its own upstream. A lone session uses its unbound proxy as before |
|
|
1389
|
+
| Two sessions are open, and none of the server's proxies is known to be this session's | Nothing is sent: the step is not run, the plan hands over, and the model calls the tool on its own upstream. A lone session uses its unbound proxy as before. A direct plan first binds the proxies started by the same host process as its `bir-scenario`, so for one this is left to a proxy or `bir-scenario` that sends no `ppid` (an older runner), or a server the host started through a wrapper (a shell, `npx`), which gives it another parent |
|
|
1374
1390
|
| A direct step exceeds `BIR_STEP_TIMEOUT_MS` | The proxy cancels the call upstream (`notifications/cancelled`) just before the control server gives up; the plan stops and hands over (*"was sent but did not answer … may or may not have completed"*). No recorded output stands in |
|
|
1375
1391
|
| Whole plan exceeds its budget (`BIR_REPLAY_PLAN_BUDGET_MS`) | Return what completed behind a hand-over note, marked partial; outcome `diverged` (`failed` when no step had run) |
|
|
1376
1392
|
| `run_scenario` cancelled (Esc) or `bir-scenario`'s 290 s timeout | No further step starts. The note saying which steps ran reaches the model on its next prompt or tool call; `fell_back` or `failed`, `measured: false` |
|
|
@@ -1499,7 +1515,12 @@ export class ProxyWorkQueue {
|
|
|
1499
1515
|
> interchangeable: each holds its own upstream (another browser page, another database session). A
|
|
1500
1516
|
> step goes to a proxy bound to the step's session — learned from a correlated call it reported, or
|
|
1501
1517
|
> an earlier step of the session it ran — and otherwise to an unbound proxy **only while no other
|
|
1502
|
-
> session is live**.
|
|
1518
|
+
> session is live**. A direct plan starts with `run_scenario`, the session's first call, so no call
|
|
1519
|
+
> of its own can have bound a proxy yet. But Claude Code starts all of a session's MCP servers as its
|
|
1520
|
+
> own children: every proxy sends its parent's pid (`ppid`) with its poll, `bir-scenario` sends its
|
|
1521
|
+
> own with `/scenario/run`, and before the plan runs the proxies with that parent are bound to the
|
|
1522
|
+
> session (`bindSiblings`). A proxy proven by a correlated call to be another session's is never
|
|
1523
|
+
> rebound. With two sessions open, a session with no bound proxy for the server still gets none:
|
|
1503
1524
|
> the call rejects with `NoSessionProxyError`, nothing is sent, the step is recorded as `skipped`,
|
|
1504
1525
|
> and the plan hands over so the model calls the tool on its own upstream. `bir replay` has no
|
|
1505
1526
|
> session and takes the newest proxy.
|
|
@@ -637,6 +637,7 @@ Code's pool, as before.
|
|
|
637
637
|
|---|---|---|
|
|
638
638
|
| No `run.matched`, ever | Prompt below the server's `SIMILARITY_THRESHOLD` | Rephrase closer, or lower it server-side. Confirm the original run is in the list — a sub-`RECORDING_MIN_ACTIONS` run is never embedded |
|
|
639
639
|
| `run.matched` but no `plan.armed` | A gate declined | Read the `replay.decision` line's `why=`. Ranked by frequency: replay off — `bir replay off`, `BIR_REPLAY=0`, or never switched on here (a `bir install` without `--replay`; `bir replay status` says which) · scenario not `ready` · similarity below threshold · no step is executable |
|
|
640
|
+
| Every prompt in a session with Claude in Chrome connected matches the same recording, at 1.000 | A runner up to 0.3.12 sent the Claude in Chrome guide (1,008 tokens) as part of the prompt, and the service's model reads 512, so only the guide was embedded | Update the runner. Switch off the recordings whose prompt starts with `<browser_instruction>` (`bir investigate list` shows them), and record those tasks again: a recording keeps the embedding it was made with |
|
|
640
641
|
| `replay.decision … code=match_timeout` or `code=service_unhealthy` on many prompts | The service answered the match slower than `BIR_MATCH_BUDGET_MS` (3 s), or is failing and in its cool-down | Nothing breaks: those turns run normally, and after a slow match prompts stop waiting on it for a while by themselves (`replay.match_gate`). `bir doctor` shows the service's state. Raise `BIR_MATCH_BUDGET_MS` only if the service is reliably just over it — every prompt pays it |
|
|
641
642
|
| `replay.decision … code=arm_timeout`, `code=missing_target … derivation did not answer within …ms`, or `code=no_derive_key why="service: …"`, on a prompt that names its values | The reading of the prompt's values (the derivation) was too slow for the hook, or failed on the service. A hook waits `BIR_HOOK_DERIVE_BUDGET_MS` (8 s) for it, and the prompt hook never past 7.5 s from the prompt, match included; the service gives one reading `RUNNER_DERIVE_TIMEOUT_MS` (6 s) and then answers `service: error` or `service: timeout`, so its answer arrives inside the prompt's wait either way | Nothing to change in the prompt, and nothing to sign in to: that turn ran normally. The `replay.derived` line that follows says when the reading landed and what it cost. If it is slow every time, the service's reading model is the cause; `BIR_DERIVE_API_KEY` where Claude Code starts moves the reading to this machine |
|
|
642
643
|
| `replay.decision … code=no_delivery_tool` | A direct plan, and no running `bir` server has told the recorder it serves `run_scenario`: the server is missing, disabled or failed in `/mcp`, was not added (a subdirectory or worktree of a repository), or the session was started with `BIR_OFF=1` | `/mcp` shows whether `bir` is connected. `bir install --replay` adds it, in the project's own directory; then start a new session |
|
|
@@ -652,7 +653,7 @@ Code's pool, as before.
|
|
|
652
653
|
| `run_scenario` answers "Nothing to run for this request; continue normally." | The model called it on a turn with no plan, the runner is off, or no recorder is running for this directory | Harmless, and not an error. It happens when the model remembers the tool from an earlier turn |
|
|
653
654
|
| `run_scenario` answers "The prepared results are not available for this request. Do the task normally with your tools." | The plan could not be delivered — a recorder that does not answer, a timeout, an HTTP error — and the model does the task itself | The `scenario.answered` line on the `bir` server's stderr has the real reason. `BIR_SCENARIO_DEBUG=1` in that server's environment hands the raw reason to the model as an error, for debugging |
|
|
654
655
|
| A hand-over note says a step "was sent but did not answer … may or may not have completed" | The step timed out, or its upstream went away mid-call; nothing after it ran | Check whether its work happened before running it again — the model is told the same. Repeats: look at that upstream, or raise `BIR_STEP_TIMEOUT_MS` |
|
|
655
|
-
| A hand-over note says "a step never runs on another session's upstream" | Two Claude Code sessions are open in the project, and this one has not yet made a call of its own to that server, so bir cannot tell its proxy from the other session's | Expected: the model calls the tool itself. Once the session's own call to the server has gone through, later replays use its proxy |
|
|
656
|
+
| A hand-over note says "a step never runs on another session's upstream" | Two Claude Code sessions are open in the project, and this one has not yet made a call of its own to that server, so bir cannot tell its proxy from the other session's. A direct plan is told apart by the process that started both the `bir` server and the session's proxies, so for one this means an older proxy or `bir` server (restart the session after an update), or a server started through a wrapper such as a shell or `npx` | Expected: the model calls the tool itself. Once the session's own call to the server has gone through, later replays use its proxy |
|
|
656
657
|
| A hand-over note says a step "was not run by the host", or "ran with arguments a person chose, not the scenario's" | A LangChain application (§6.5). A person said no to the step, or answered in the tool's place, or a call limit blocked the call (`host_rejected`); or a person changed the call before it ran (`host_edited`) | Expected, and no fault of the plan: the step is booked `skipped`, the plan stops there and the model takes over. It is told not to run the step again — after a refusal, unless the user asks. §7 has what the report says |
|
|
657
658
|
| `recorder.send_failed … error="HTTP 409 … invalid_ticket"` | The execution report's ticket was already redeemed, or belongs to another scenario. Or the report came more than an hour after its plan was handed out | Harmless if the body says `duplicate: true`. Otherwise a rollover raced `SessionEnd`; the server books once. A report that came too late is not booked; the service as built (not deployed yet) takes a report for 24 hours |
|
|
658
659
|
| `recorder.send_failed … error="HTTP 429 …"` | The account is over one of the service's limits, and the answer is `429` with `Retry-After`. The limits are built on the service and not deployed yet; each is a setting there. By default: 120 runs a minute · 3,000 steps a minute (a tool call is two steps) · 10 title requests a minute · 20 requests in any one hour for Calculate, Prefer and Recalculate together (`bir scenario calc` then prints a line that begins `Could not start calculation (HTTP 429`) | What it costs: a create, a step or a report is sent once more, at most 2 s later, then dropped. A dropped step is a hole in its recording. A run whose create got the 429 runs without its plan. A 429 that the second try did not cure counts as the service being unwell: for 5 to 60 seconds no prompt of that recorder waits for a match. An intent probe over its own limit (1,500 a minute) is answered with no match, never with `429` |
|
|
@@ -702,6 +703,12 @@ fix*. The three questions it answers:
|
|
|
702
703
|
from the transcript, or the runner has no price for the model: update it."*
|
|
703
704
|
The `run.unmeasured` line names what was missing.
|
|
704
705
|
|
|
706
|
+
**A prompt the runner cleaned** has a `stripped` line under it: the blocks
|
|
707
|
+
Claude Code put in front of what you typed (the Claude in Chrome guide, the
|
|
708
|
+
open file, the selection), which the service never got, and the size before
|
|
709
|
+
and after. A background task's notification has no words of yours at all: it
|
|
710
|
+
shows `prompt (none: a task notification, never matched)`.
|
|
711
|
+
|
|
705
712
|
**A turn of another host** (a LangChain application, §6.5) is told apart by
|
|
706
713
|
the journal's `run.start` line, which carries `host=<app>`. For such a turn:
|
|
707
714
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@basein/runner",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.13",
|
|
4
4
|
"description": "A recording MCP proxy: sits between any MCP client and its MCP servers, executes each call on the client's behalf, and records the run as a reusable BaseIn scenario.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/production.md
CHANGED
|
@@ -537,6 +537,8 @@ Over 2-minute runs with one turn every 8 s (92 hooks):
|
|
|
537
537
|
|
|
538
538
|
Truncation is always marked with the original size.
|
|
539
539
|
|
|
540
|
+
**The prompt.** In a Claude Code session the prompt is sent without the blocks Claude Code puts in front of it (the Claude in Chrome guide, the open file, the selection, a task notification), and pasted text is unwrapped (fx121.md). So the open file's path and the selected text stay on the machine. A task notification with no words of the user's is sent with an empty prompt, which the service never matches. Another host's prompt is sent as it came.
|
|
541
|
+
|
|
540
542
|
**What the service does with a report.** The service's new build refuses a report less easily for its size or its age (built, not deployed yet):
|
|
541
543
|
|
|
542
544
|
- The prompt inside a report is cut at 10,000 characters. It used to be refused with 400.
|
|
@@ -909,7 +911,7 @@ Lines read `[bir] <ISO time> <event> key=value …`, with each field clipped at
|
|
|
909
911
|
|
|
910
912
|
| Topic | Lines and what they mean |
|
|
911
913
|
|---|---|
|
|
912
|
-
| Service health | `service.degraded` (first failure: reason, cool-down), `service.recovered`. `replay.decision code=service_unhealthy` (prompt did not wait: breaker open), `code=match_timeout` (prompt waited its budget), `code=match_late` (the answer landed after the prompt), `code=match_slow` (verbose: gate holding). `replay.match_gate state=holding\|open`. |
|
|
914
|
+
| Service health | `service.degraded` (first failure: reason, cool-down), `service.recovered`. `replay.decision code=service_unhealthy` (prompt did not wait: breaker open), `code=match_timeout` (prompt waited its budget), `code=match_late` (the answer landed after the prompt, including one the prompt never waited for because a failure was still on record), `code=match_slow` (verbose: gate holding). `replay.match_gate state=holding\|open`. |
|
|
913
915
|
| Recording backlog | `recorder.backlog_full` / `recorder.backlog_recovered` (oldest steps dropped). `recorder.run_incomplete` (a run lost steps). `recorder.run_abandoned` (`reason=service_down`: a create given up after a minute of cool-down). `recorder.flush_deferred`, `recorder.flush_incomplete`, `recorder.send_failed` (once per run), `recorder.matched_after_retry`, `run.lossy`. |
|
|
914
916
|
| Sign-in | `auth.ok`, `auth.settling` (plain-http address being checked for https), `auth.refresh_deferred`, `auth.refresh_refused`, `auth.recovered`. `recorder.enabled`, `recorder.signed_out`, `recorder.disabled`, `recorder.paused` / `recorder.resumed`, `recorder.account_changed`, `recorder.report_dropped`, `derive.api_key_ignored`. |
|
|
915
917
|
| Hooks | `control.listening` (port, `recorder=<state>`, `correlation=`). `control.correlation_allow` (auto-approval opted in). `control.hook_deadline`, `control.hook_unauthorized` (a token from another install), `control.hook_unknown`, `control.hook_oversize`, `control.hook_event_mismatch`, `control.route_failed`, `control.drain_abandoned`. |
|