pi-crew 0.9.60 → 0.9.62
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/dist/index.mjs +106 -18
- package/package.json +1 -1
- package/src/extension/notification-router.ts +25 -0
- package/src/extension/registration/lifecycle-handlers.ts +47 -6
- package/src/extension/registration/lifecycle.ts +13 -7
- package/src/runtime/live-session/live-session-runtime.ts +12 -1
- package/src/runtime/model/session-model.ts +76 -1
package/CHANGELOG.md
CHANGED
|
@@ -3,10 +3,24 @@
|
|
|
3
3
|
> **Note:** `atomic-write-v2.ts` / `AtomicWriter` mentioned in historical entries below was consolidated into `atomic-write.ts` as of v0.9.42. This changelog is preserved as historical record — the migration was completed (the v2 class was never adopted; v1 won on simplicity + symlink-safety + link+unlink atomicity). See `docs/migration/atomic-write-v2-migration.md` for the decision rationale.
|
|
4
4
|
|
|
5
5
|
|
|
6
|
+
## [0.9.62] — provider-quota attribution per live-session agent + dead-worker alert re-fire fix (2026-08-06)
|
|
7
|
+
|
|
8
|
+
### Bug fixes
|
|
9
|
+
|
|
10
|
+
- **Provider-quota responses were attributed to the wrong provider in the opt-in `live-session` runtime.** pi's `after_provider_response` event carries no provider/sessionId, so quota was keyed off the shared module-scoped `currentSessionModel()`: a provider-B response (e.g. a 429) was recorded under the main session's anchor provider A — mis-deprioritizing A, or overwriting a genuine exhaustion signal so the “deprioritize exhausted providers” contract was not reliably honored. (Only reachable via the opt-in `live-session` runtime; the default `child-process` runtime is unaffected since each worker is its own process with anchor==serving provider.) Fix: carry each in-process live agent's resolved model through `AsyncLocalStorage` and attribute quota from it (`resolveProviderForResponse()`, `src/runtime/model/session-model.ts`); **skip** attribution (rather than guess) when live agents are active but the async context is absent; the default `child-process` path is byte-identical to before. Hardening from review: `unregisterLiveAgentModel` now runs **first** in the live-session `finally` block (a thrown `terminateLiveAgent` on the user-cancel path previously skipped it, permanently blinding quota process-wide via `hasActiveLiveAgents()` stuck `true`); `liveAgentModels` is capped at 5,000 entries matching `MAX_LIVE_AGENTS`. Tests in `test/unit/runtime/model/live-agent-quota-attribution.test.ts`. Investigation/spec in `docs/bugs/model-quota-attribution.md`.
|
|
11
|
+
- **Dead-worker dashboard alert re-fired every 5 min for already-terminal runs.** A stale snapshot cache (not auto-invalidated when a run transitions to terminal) kept task status as "running" with an aged heartbeat, so the health loop re-classified it as dead and re-emitted the alert every 5 min; and there was no dismiss/clear path for health notifications. Fix: (1) the health loop in `renderTick` now re-verifies each run against a FRESH manifest-cache read before emitting; terminal runs are skipped and their stale snapshot is purged; (2) `NotificationDescriptor` gains an optional `clear` flag — when a run goes terminal, previously-emitted `recovery_dead_workers` / `recovery_missing_heartbeat` notifications are cleared and their cooldown entries are removed. Genuinely running runs with stale workers still alert (regression-guarded). Tests in `notification-router.test.ts` and `heartbeat-aggregator.test.ts`. Investigation in `docs/bugs/dead-worker-alert-refire.md`.
|
|
12
|
+
|
|
13
|
+
## [0.9.61] — bundle republish: bug-44 fix shipped in dist (2026-08-05)
|
|
14
|
+
|
|
15
|
+
### Bug fixes
|
|
16
|
+
|
|
17
|
+
- **Bug-44 fix was missing from the v0.9.60 npm bundle (hot-fix republish).** The source-level fix for GitHub #44 (chain run fails fast when `workflow:"chain"` is forwarded to steps) landed in commit `5b43d556`, but the published `dist/index.mjs` was built from a stale source tree *before* the fix, so users installing `pi-crew@0.9.60` got the old buggy bundle (the chain runner still forwarded `workflow` to steps). v0.9.61 rebuilds the bundle with the fix included and verified (`grep` for the reject message + fallback logic now matches in `dist/index.mjs`). No source changes — purely a corrected bundle republish.
|
|
18
|
+
|
|
6
19
|
## [0.9.60] — subagent model routing: live-session model tracking, fallback policy, quota-aware ordering + iterative-audit hardening (2026-08-05)
|
|
7
20
|
|
|
8
21
|
### Bug fixes
|
|
9
22
|
|
|
23
|
+
- **Chain run fails fast (~58 ms) with no error when `workflow:"chain"` is forwarded to steps (GitHub #44).** `chain-dispatch.ts` forwarded `params.workflow` (="chain") into every step's executor overrides; `chain-executor.ts` passed it through to each step's `handleRun`, which then executed the dispatcher-only `chain` workflow via the normal `executeTeamRun` path. Since `chain.workflow.md` is a documentation file (not a runnable workflow — it parses to 10 doc-heading steps with unknown roles), validation failed fast with a confusing empty error. Fix: (1) `chain-dispatch.ts` no longer forwards `workflow` (a chain has no per-step workflow; steps use the team's defaultWorkflow); (2) `chain-executor.ts` defensively drops a `"chain"` workflow override; (3) `run.ts` falls back to the team's default workflow when `workflow="chain"` is requested without a `chain` param; (4) `handleChainRun` rejects `workflow:"chain"` with a clear message. Regression tests added in `chain-executor.test.ts`.
|
|
10
24
|
- **Subagent model "jumped" to whatever a previous session had saved.** `ctx.model` is the session's *saved* model, not the live one. A session that restored stale state could report `anthropic/claude-sonnet-4-5` while actually running `minimax/MiniMax-M3`. Subagents inheriting the parent model (`model: false`, which every builtin agent uses) therefore landed on the wrong model. Fix: track the live model via pi's `model_select` event (`src/runtime/model/session-model.ts`) and use it as the parent model for all spawn paths.
|
|
11
25
|
- **Live-session path silently discarded `ctx.model`.** `resolveParentModelFromRegistry` only accepted strings; `ctx.model` is a pi `Model` object. Every live-session subagent that inherited the parent model fell through to `getAvailable()[0]` instead. Fix: accept both objects and strings via `modelRefToString`.
|
|
12
26
|
- **Background/async runs lost the caller's model context.** A detached background run has no `ExtensionContext`, so it lost the `model=` override, the inherited session model, and the auth-filtered model catalogue — silently routing to whatever `models.json` listed first. Fix: persist `modelContext` (override / parent model / available models) on the manifest at dispatch time and re-hydrate it in `background-runner`.
|
package/dist/index.mjs
CHANGED
|
@@ -22303,6 +22303,32 @@ var init_model_fallback = __esm({
|
|
|
22303
22303
|
});
|
|
22304
22304
|
|
|
22305
22305
|
// src/runtime/model/session-model.ts
|
|
22306
|
+
import { AsyncLocalStorage as AsyncLocalStorage2 } from "node:async_hooks";
|
|
22307
|
+
function registerLiveAgentModel(agentId, model) {
|
|
22308
|
+
if (liveAgentModels.size >= MAX_LIVE_AGENT_MODELS && !liveAgentModels.has(agentId)) {
|
|
22309
|
+
const oldestKey = liveAgentModels.keys().next().value;
|
|
22310
|
+
if (oldestKey !== void 0) {
|
|
22311
|
+
logInternalError(
|
|
22312
|
+
"session-model.liveAgentModels.cap",
|
|
22313
|
+
new Error(`liveAgentModels at cap ${MAX_LIVE_AGENT_MODELS}; evicting oldest ${oldestKey}`)
|
|
22314
|
+
);
|
|
22315
|
+
liveAgentModels.delete(oldestKey);
|
|
22316
|
+
}
|
|
22317
|
+
}
|
|
22318
|
+
liveAgentModels.set(agentId, model);
|
|
22319
|
+
}
|
|
22320
|
+
function unregisterLiveAgentModel(agentId) {
|
|
22321
|
+
liveAgentModels.delete(agentId);
|
|
22322
|
+
}
|
|
22323
|
+
function hasActiveLiveAgents() {
|
|
22324
|
+
return liveAgentModels.size > 0;
|
|
22325
|
+
}
|
|
22326
|
+
function resolveProviderForResponse() {
|
|
22327
|
+
const ctx = liveAgentContext.getStore();
|
|
22328
|
+
if (ctx) return providerOfModelRef(ctx.modelRef);
|
|
22329
|
+
if (hasActiveLiveAgents()) return void 0;
|
|
22330
|
+
return providerOfModelRef(currentSessionModel());
|
|
22331
|
+
}
|
|
22306
22332
|
function noteSessionModel(model, source = "model_select") {
|
|
22307
22333
|
const normalized = modelRefToString(model);
|
|
22308
22334
|
if (!normalized) return;
|
|
@@ -22342,12 +22368,16 @@ function captureRunModelContext(ctx, override) {
|
|
|
22342
22368
|
function sessionModelSnapshot() {
|
|
22343
22369
|
return { ...state };
|
|
22344
22370
|
}
|
|
22345
|
-
var state;
|
|
22371
|
+
var state, liveAgentContext, liveAgentModels, MAX_LIVE_AGENT_MODELS;
|
|
22346
22372
|
var init_session_model = __esm({
|
|
22347
22373
|
"src/runtime/model/session-model.ts"() {
|
|
22348
22374
|
"use strict";
|
|
22375
|
+
init_internal_error();
|
|
22349
22376
|
init_model_fallback();
|
|
22350
22377
|
state = { source: "none" };
|
|
22378
|
+
liveAgentContext = new AsyncLocalStorage2();
|
|
22379
|
+
liveAgentModels = /* @__PURE__ */ new Map();
|
|
22380
|
+
MAX_LIVE_AGENT_MODELS = 5e3;
|
|
22351
22381
|
}
|
|
22352
22382
|
});
|
|
22353
22383
|
|
|
@@ -33237,6 +33267,7 @@ async function runLiveSessionTask(input) {
|
|
|
33237
33267
|
scopeModelsPatterns: await resolveScopeModelsPatterns(input.manifest.cwd)
|
|
33238
33268
|
});
|
|
33239
33269
|
const resolvedModel = modelFromRegistry(input.modelRegistry, modelRouting.candidates[0] ?? modelRouting.requested) ?? input.parentModel;
|
|
33270
|
+
const resolvedModelRef = modelRefToString(resolvedModel) ?? modelRouting.candidates[0];
|
|
33240
33271
|
if (modelRouting.droppedRequested) {
|
|
33241
33272
|
appendEventFireAndForget(input.manifest.eventsPath, {
|
|
33242
33273
|
type: "task.model_dropped",
|
|
@@ -33335,6 +33366,7 @@ async function runLiveSessionTask(input) {
|
|
|
33335
33366
|
appendEvent,
|
|
33336
33367
|
input.manifest.eventsPath
|
|
33337
33368
|
);
|
|
33369
|
+
registerLiveAgentModel(agentId, resolvedModelRef ?? "");
|
|
33338
33370
|
streamOut = createStreamingOutput(input.manifest, input.task.id);
|
|
33339
33371
|
let controlCursor = { offset: 0 };
|
|
33340
33372
|
const seenControlRequestIds = /* @__PURE__ */ new Set();
|
|
@@ -33470,7 +33502,10 @@ ${input.prompt}` : input.prompt;
|
|
|
33470
33502
|
});
|
|
33471
33503
|
const sessionTimeoutMs = DEFAULT_LIVE_SESSION.responseTimeoutMs;
|
|
33472
33504
|
try {
|
|
33473
|
-
await
|
|
33505
|
+
await liveAgentContext.run(
|
|
33506
|
+
{ agentId, modelRef: resolvedModelRef ?? "" },
|
|
33507
|
+
() => promptWithTimeout(session, effectivePrompt, sessionTimeoutMs, "Live-session")
|
|
33508
|
+
);
|
|
33474
33509
|
} catch (promptError) {
|
|
33475
33510
|
const msg = promptError instanceof Error ? promptError.message : String(promptError);
|
|
33476
33511
|
appendEventFireAndForget(input.manifest.eventsPath, {
|
|
@@ -33643,6 +33678,7 @@ ${input.prompt}` : input.prompt;
|
|
|
33643
33678
|
error: message
|
|
33644
33679
|
};
|
|
33645
33680
|
} finally {
|
|
33681
|
+
unregisterLiveAgentModel(agentId);
|
|
33646
33682
|
unsubscribe?.();
|
|
33647
33683
|
unsubscribeControlRealtime?.();
|
|
33648
33684
|
if (onSignalAbort) input.signal?.removeEventListener("abort", onSignalAbort);
|
|
@@ -33681,6 +33717,7 @@ var init_live_session_runtime = __esm({
|
|
|
33681
33717
|
init_model_scope();
|
|
33682
33718
|
init_runtime_resolver();
|
|
33683
33719
|
init_runtime_warmup();
|
|
33720
|
+
init_session_model();
|
|
33684
33721
|
init_sidechain_output();
|
|
33685
33722
|
init_streaming_output();
|
|
33686
33723
|
init_sensitive_paths();
|
|
@@ -49170,7 +49207,7 @@ var init_dispatch = __esm({
|
|
|
49170
49207
|
});
|
|
49171
49208
|
|
|
49172
49209
|
// src/observability/correlation.ts
|
|
49173
|
-
import { AsyncLocalStorage as
|
|
49210
|
+
import { AsyncLocalStorage as AsyncLocalStorage3 } from "node:async_hooks";
|
|
49174
49211
|
import { randomBytes as randomBytes2 } from "node:crypto";
|
|
49175
49212
|
function withCorrelation(ctx, fn) {
|
|
49176
49213
|
return storage.run(ctx, fn);
|
|
@@ -49194,7 +49231,7 @@ var storage;
|
|
|
49194
49231
|
var init_correlation = __esm({
|
|
49195
49232
|
"src/observability/correlation.ts"() {
|
|
49196
49233
|
"use strict";
|
|
49197
|
-
storage = new
|
|
49234
|
+
storage = new AsyncLocalStorage3();
|
|
49198
49235
|
}
|
|
49199
49236
|
});
|
|
49200
49237
|
|
|
@@ -59883,7 +59920,8 @@ var init_chain_executor = __esm({
|
|
|
59883
59920
|
# Current Chain Step
|
|
59884
59921
|
${packet.goal}` : packet.goal;
|
|
59885
59922
|
const stepTeam = context.__chainStepTeam ?? this.overrides.team ?? "default";
|
|
59886
|
-
const
|
|
59923
|
+
const rawWorkflow = context.__chainStepWorkflow ?? this.overrides.workflow;
|
|
59924
|
+
const stepWorkflow = rawWorkflow === "chain" ? void 0 : rawWorkflow;
|
|
59887
59925
|
const stepModel = context.__chainStepModel ?? this.overrides.model;
|
|
59888
59926
|
const runParams = {
|
|
59889
59927
|
action: "run",
|
|
@@ -59928,6 +59966,13 @@ async function handleChainRun(params, ctx, handleRun3) {
|
|
|
59928
59966
|
if (!chainString || chainString.trim().length === 0) {
|
|
59929
59967
|
return result("Chain expression is empty.", { action: "run", status: "error" }, true);
|
|
59930
59968
|
}
|
|
59969
|
+
if (params.workflow === "chain") {
|
|
59970
|
+
return result(
|
|
59971
|
+
"Workflow 'chain' cannot be combined with a chain run: the chain runner owns step execution and has no per-step workflow. Omit `workflow` (steps use the team's defaultWorkflow) or use @team references in the chain expression.",
|
|
59972
|
+
{ action: "run", status: "error" },
|
|
59973
|
+
true
|
|
59974
|
+
);
|
|
59975
|
+
}
|
|
59931
59976
|
const spec = parseChainString(chainString);
|
|
59932
59977
|
if (spec.steps.length === 0) {
|
|
59933
59978
|
return result(
|
|
@@ -59941,8 +59986,8 @@ async function handleChainRun(params, ctx, handleRun3) {
|
|
|
59941
59986
|
ctx,
|
|
59942
59987
|
overrides: {
|
|
59943
59988
|
team: params.team,
|
|
59944
|
-
workflow
|
|
59945
|
-
model: params.model
|
|
59989
|
+
// workflow intentionally omitted — chain runner owns step execution (bug-44)
|
|
59990
|
+
...params.model ? { model: params.model } : {}
|
|
59946
59991
|
}
|
|
59947
59992
|
});
|
|
59948
59993
|
const runner = new ChainRunner(executor, new HandoffManager());
|
|
@@ -61499,7 +61544,7 @@ Commit or stash changes before using worktree mode, or use workspaceMode: 'singl
|
|
|
61499
61544
|
workspaceMode: params.workspaceMode
|
|
61500
61545
|
} : teams.find((item) => item.name === teamName);
|
|
61501
61546
|
if (!team) return result(`Team '${teamName}' not found.`, { action: "run", status: "error" }, true);
|
|
61502
|
-
const workflowName = directAgent ? "direct-agent" : params.workflow ?? team.defaultWorkflow ?? "default";
|
|
61547
|
+
const workflowName = directAgent ? "direct-agent" : params.workflow === "chain" && !params.chain ? team.defaultWorkflow ?? "default" : params.workflow ?? team.defaultWorkflow ?? "default";
|
|
61503
61548
|
const baseWorkflow = directAgent ? {
|
|
61504
61549
|
name: "direct-agent",
|
|
61505
61550
|
description: `Direct task for ${directAgent.name}`,
|
|
@@ -69552,6 +69597,19 @@ var init_notification_router = __esm({
|
|
|
69552
69597
|
...notification,
|
|
69553
69598
|
timestamp: notification.timestamp ?? now
|
|
69554
69599
|
};
|
|
69600
|
+
if (withTime.clear) {
|
|
69601
|
+
const clearKey = notificationKey(withTime);
|
|
69602
|
+
const wasInSeen = this.seen.delete(clearKey);
|
|
69603
|
+
if (wasInSeen) {
|
|
69604
|
+
try {
|
|
69605
|
+
this.opts.sink?.(withTime);
|
|
69606
|
+
} catch (sinkError) {
|
|
69607
|
+
logInternalError("notification-sink", sinkError);
|
|
69608
|
+
}
|
|
69609
|
+
this.deliver(withTime);
|
|
69610
|
+
}
|
|
69611
|
+
return true;
|
|
69612
|
+
}
|
|
69555
69613
|
try {
|
|
69556
69614
|
this.opts.sink?.(withTime);
|
|
69557
69615
|
} catch (sinkError) {
|
|
@@ -70084,11 +70142,15 @@ async function configureNotifications(ctx, state2, deps) {
|
|
|
70084
70142
|
sink: (notification) => state2.notificationSink?.write(notification)
|
|
70085
70143
|
},
|
|
70086
70144
|
(notification) => {
|
|
70087
|
-
|
|
70088
|
-
|
|
70089
|
-
|
|
70090
|
-
|
|
70091
|
-
|
|
70145
|
+
if (notification.clear) {
|
|
70146
|
+
deps.widgetState.notificationCount = Math.max(0, (deps.widgetState.notificationCount ?? 0) - 1);
|
|
70147
|
+
} else {
|
|
70148
|
+
deps.widgetState.notificationCount = (deps.widgetState.notificationCount ?? 0) + 1;
|
|
70149
|
+
sendFollowUp2(
|
|
70150
|
+
deps.pi,
|
|
70151
|
+
[notification.title, notification.body, notification.runId ? `Run: ${notification.runId}` : void 0].filter((line4) => Boolean(line4)).join("\n")
|
|
70152
|
+
);
|
|
70153
|
+
}
|
|
70092
70154
|
const currentCtx = deps.getCurrentCtx();
|
|
70093
70155
|
if (currentCtx) {
|
|
70094
70156
|
const uiConfig = loadConfig(currentCtx.cwd).config.ui;
|
|
@@ -76656,7 +76718,6 @@ function safeStringify(value) {
|
|
|
76656
76718
|
// src/extension/registration/lifecycle-handlers.ts
|
|
76657
76719
|
init_child_pi();
|
|
76658
76720
|
init_live_agent_manager();
|
|
76659
|
-
init_model_fallback();
|
|
76660
76721
|
init_pi_args();
|
|
76661
76722
|
init_provider_quota();
|
|
76662
76723
|
init_session_model();
|
|
@@ -76955,8 +77016,7 @@ function installModelTrackingHandlers(pi) {
|
|
|
76955
77016
|
noteSessionThinking(event.level);
|
|
76956
77017
|
});
|
|
76957
77018
|
pi.on("after_provider_response", (event) => {
|
|
76958
|
-
const
|
|
76959
|
-
const provider = model ? providerOfModelRef(model) : void 0;
|
|
77019
|
+
const provider = resolveProviderForResponse();
|
|
76960
77020
|
if (provider) noteProviderResponse(provider, event.status, event.headers);
|
|
76961
77021
|
});
|
|
76962
77022
|
}
|
|
@@ -77353,12 +77413,40 @@ function setupRenderLoop(pi, ctx, extensionCtx, loadedConfig) {
|
|
|
77353
77413
|
const currentSessionId = ctx.currentCtx?.sessionManager?.getSessionId();
|
|
77354
77414
|
const sessionManifests = filterManifestsForHealthNotifications(manifests, currentSessionId);
|
|
77355
77415
|
const now = Date.now();
|
|
77416
|
+
const clearHealthNotifications = (runId) => {
|
|
77417
|
+
for (const kind of ["recovery_dead_workers", "recovery_missing_heartbeat"]) {
|
|
77418
|
+
const key = `${kind}_${runId}`;
|
|
77419
|
+
ctx.autoRecoveryLast.delete(key);
|
|
77420
|
+
ctx.notifyOperator({
|
|
77421
|
+
id: key,
|
|
77422
|
+
clear: true,
|
|
77423
|
+
severity: "info",
|
|
77424
|
+
source: "health",
|
|
77425
|
+
runId,
|
|
77426
|
+
title: `Cleared ${kind} for ${runId}`
|
|
77427
|
+
});
|
|
77428
|
+
}
|
|
77429
|
+
};
|
|
77356
77430
|
for (const run of sessionManifests) {
|
|
77357
|
-
if (run.status !== "running")
|
|
77431
|
+
if (run.status !== "running") {
|
|
77432
|
+
snapshotCache.invalidate(run.runId);
|
|
77433
|
+
clearHealthNotifications(run.runId);
|
|
77434
|
+
continue;
|
|
77435
|
+
}
|
|
77358
77436
|
try {
|
|
77437
|
+
const freshManifest = ctx.getManifestCache(extensionCtx.cwd).get(run.runId);
|
|
77438
|
+
if (freshManifest?.status !== "running") {
|
|
77439
|
+
snapshotCache.invalidate(run.runId);
|
|
77440
|
+
clearHealthNotifications(run.runId);
|
|
77441
|
+
continue;
|
|
77442
|
+
}
|
|
77359
77443
|
const snapshot = snapshotCache.get(run.runId);
|
|
77360
77444
|
if (!snapshot) continue;
|
|
77361
|
-
if (snapshot.manifest.status !== "running")
|
|
77445
|
+
if (snapshot.manifest.status !== "running") {
|
|
77446
|
+
snapshotCache.invalidate(run.runId);
|
|
77447
|
+
clearHealthNotifications(run.runId);
|
|
77448
|
+
continue;
|
|
77449
|
+
}
|
|
77362
77450
|
const summary = summarizeHeartbeats(snapshot, { now });
|
|
77363
77451
|
const maybeNotifyHealth = (kind, count2, title, body) => {
|
|
77364
77452
|
if (count2 <= 0) return;
|
package/package.json
CHANGED
|
@@ -10,6 +10,8 @@ export interface NotificationDescriptor {
|
|
|
10
10
|
title: string;
|
|
11
11
|
body?: string;
|
|
12
12
|
timestamp?: number;
|
|
13
|
+
/** When true, this is a dismiss/clear request for a previously-emitted notification (same id). Bypasses dedup/severity/quiet-hours; the deliver callback drops the matching notification. */
|
|
14
|
+
clear?: boolean;
|
|
13
15
|
}
|
|
14
16
|
|
|
15
17
|
export interface NotificationRouterOptions {
|
|
@@ -99,6 +101,29 @@ export class NotificationRouter {
|
|
|
99
101
|
...notification,
|
|
100
102
|
timestamp: notification.timestamp ?? now,
|
|
101
103
|
};
|
|
104
|
+
// Clear path: remove the id from `seen` so a future genuine re-occurrence
|
|
105
|
+
// can re-notify. Delivery is guarded on whether the id was actually in
|
|
106
|
+
// `seen` to keep the clear idempotent (bypassing dedup / severity filter /
|
|
107
|
+
// quiet hours when it does deliver — a clear must go through so the
|
|
108
|
+
// dashboard can drop the previously-emitted notification).
|
|
109
|
+
if (withTime.clear) {
|
|
110
|
+
const clearKey = notificationKey(withTime);
|
|
111
|
+
// Idempotent: only deliver the clear if the id was actually in `seen`
|
|
112
|
+
// (i.e., a notification for it was previously emitted and not yet
|
|
113
|
+
// cleared). This prevents multi-fire drift when the render loop calls
|
|
114
|
+
// clearHealthNotifications on consecutive ticks for the same terminal
|
|
115
|
+
// run — each clear after the first is a silent no-op that returns true.
|
|
116
|
+
const wasInSeen = this.seen.delete(clearKey);
|
|
117
|
+
if (wasInSeen) {
|
|
118
|
+
try {
|
|
119
|
+
this.opts.sink?.(withTime);
|
|
120
|
+
} catch (sinkError) {
|
|
121
|
+
logInternalError("notification-sink", sinkError);
|
|
122
|
+
}
|
|
123
|
+
this.deliver(withTime);
|
|
124
|
+
}
|
|
125
|
+
return true;
|
|
126
|
+
}
|
|
102
127
|
try {
|
|
103
128
|
this.opts.sink?.(withTime);
|
|
104
129
|
} catch (sinkError) {
|
|
@@ -25,10 +25,9 @@ import { CrewBroker } from "../../runtime/broker/crew-broker.ts";
|
|
|
25
25
|
import { terminateActiveChildPiProcesses } from "../../runtime/child-pi/child-pi.ts";
|
|
26
26
|
import { listLiveAgents } from "../../runtime/live-session/live-agent-manager.ts";
|
|
27
27
|
import type { createManifestCache } from "../../runtime/manifest-cache.ts";
|
|
28
|
-
import { providerOfModelRef } from "../../runtime/model/model-fallback.ts";
|
|
29
28
|
import { cleanupLegacyOrphanTempDirs, cleanupOrphanTempDirs, currentCrewDepth } from "../../runtime/model/pi-args.ts";
|
|
30
29
|
import { clearProviderQuotaCache, noteProviderResponse } from "../../runtime/model/provider-quota.ts";
|
|
31
|
-
import {
|
|
30
|
+
import { noteSessionModel, noteSessionThinking, resolveProviderForResponse } from "../../runtime/model/session-model.ts";
|
|
32
31
|
import { cleanupOrphanWorkers } from "../../runtime/orphan-worker-registry.ts";
|
|
33
32
|
import { reconcileAllStaleRuns } from "../../runtime/recovery/crash-recovery.ts";
|
|
34
33
|
import { CrewScheduler, type ScheduledJob } from "../../runtime/scheduling/scheduler.ts";
|
|
@@ -93,8 +92,7 @@ function installModelTrackingHandlers(pi: ExtensionAPI): void {
|
|
|
93
92
|
// providers. The event doesn't carry a provider field, so we attribute it
|
|
94
93
|
// to the currently tracked session model's provider.
|
|
95
94
|
pi.on("after_provider_response", (event) => {
|
|
96
|
-
const
|
|
97
|
-
const provider = model ? providerOfModelRef(model) : undefined;
|
|
95
|
+
const provider = resolveProviderForResponse();
|
|
98
96
|
if (provider) noteProviderResponse(provider, event.status, event.headers);
|
|
99
97
|
});
|
|
100
98
|
}
|
|
@@ -666,12 +664,55 @@ function setupRenderLoop(
|
|
|
666
664
|
const currentSessionId = ctx.currentCtx?.sessionManager?.getSessionId();
|
|
667
665
|
const sessionManifests = filterManifestsForHealthNotifications(manifests, currentSessionId);
|
|
668
666
|
const now = Date.now();
|
|
667
|
+
// FIX #2: clear path — when a run is detected terminal, dismiss any
|
|
668
|
+
// previously-emitted health notification for it AND drop its cooldown
|
|
669
|
+
// (autoRecoveryLast) so a future genuine re-occurrence can re-notify.
|
|
670
|
+
// Keeps the dashboard clean and stops the 5-min re-fire cycle.
|
|
671
|
+
const clearHealthNotifications = (runId: string): void => {
|
|
672
|
+
for (const kind of ["recovery_dead_workers", "recovery_missing_heartbeat"]) {
|
|
673
|
+
const key = `${kind}_${runId}`;
|
|
674
|
+
ctx.autoRecoveryLast.delete(key);
|
|
675
|
+
ctx.notifyOperator({
|
|
676
|
+
id: key,
|
|
677
|
+
clear: true,
|
|
678
|
+
severity: "info",
|
|
679
|
+
source: "health",
|
|
680
|
+
runId,
|
|
681
|
+
title: `Cleared ${kind} for ${runId}`,
|
|
682
|
+
});
|
|
683
|
+
}
|
|
684
|
+
};
|
|
669
685
|
for (const run of sessionManifests) {
|
|
670
|
-
if (run.status !== "running")
|
|
686
|
+
if (run.status !== "running") {
|
|
687
|
+
// GATE 1 — preloaded manifest says terminal. Purge any stale snapshot
|
|
688
|
+
// and clear previously-emitted health notifications so the dashboard
|
|
689
|
+
// stays clean (belt-and-suspenders with the FIX #1 fresh-read gate).
|
|
690
|
+
snapshotCache.invalidate(run.runId);
|
|
691
|
+
clearHealthNotifications(run.runId);
|
|
692
|
+
continue;
|
|
693
|
+
}
|
|
671
694
|
try {
|
|
695
|
+
// FIX #1: re-verify against a FRESH manifest read. The preloaded `run`
|
|
696
|
+
// (from lastPreloadedManifests) can lag the on-disk terminal
|
|
697
|
+
// transition; the manifest cache has a 500ms TTL + file watcher so it
|
|
698
|
+
// is the source of truth. A terminal run must NEVER reach
|
|
699
|
+
// maybeNotifyHealth. Also purge the stale snapshot + clear any
|
|
700
|
+
// previously-emitted health notification for this run.
|
|
701
|
+
const freshManifest = ctx.getManifestCache(extensionCtx.cwd).get(run.runId);
|
|
702
|
+
if (freshManifest?.status !== "running") {
|
|
703
|
+
snapshotCache.invalidate(run.runId);
|
|
704
|
+
clearHealthNotifications(run.runId);
|
|
705
|
+
continue;
|
|
706
|
+
}
|
|
672
707
|
const snapshot = snapshotCache.get(run.runId);
|
|
673
708
|
if (!snapshot) continue;
|
|
674
|
-
if (snapshot.manifest.status !== "running")
|
|
709
|
+
if (snapshot.manifest.status !== "running") {
|
|
710
|
+
// GATE 2 — a running snapshot paired with a now-terminal manifest is
|
|
711
|
+
// stale. Purge it so subsequent ticks get a fresh view, and clear.
|
|
712
|
+
snapshotCache.invalidate(run.runId);
|
|
713
|
+
clearHealthNotifications(run.runId);
|
|
714
|
+
continue;
|
|
715
|
+
}
|
|
675
716
|
const summary = summarizeHeartbeats(snapshot, { now });
|
|
676
717
|
const maybeNotifyHealth = (kind: string, count: number, title: string, body: string): void => {
|
|
677
718
|
if (count <= 0) return;
|
|
@@ -148,13 +148,19 @@ export async function configureNotifications(ctx: ExtensionContext, state: Lifec
|
|
|
148
148
|
sink: (notification) => state.notificationSink?.write(notification),
|
|
149
149
|
},
|
|
150
150
|
(notification: NotificationDescriptor) => {
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
151
|
+
if (notification.clear) {
|
|
152
|
+
// Clear/dismiss: decrement the counter (floor 0) and skip the
|
|
153
|
+
// follow-up message — this is a dismissal, not a new alert.
|
|
154
|
+
deps.widgetState.notificationCount = Math.max(0, (deps.widgetState.notificationCount ?? 0) - 1);
|
|
155
|
+
} else {
|
|
156
|
+
deps.widgetState.notificationCount = (deps.widgetState.notificationCount ?? 0) + 1;
|
|
157
|
+
sendFollowUp(
|
|
158
|
+
deps.pi,
|
|
159
|
+
[notification.title, notification.body, notification.runId ? `Run: ${notification.runId}` : undefined]
|
|
160
|
+
.filter((line): line is string => Boolean(line))
|
|
161
|
+
.join("\n"),
|
|
162
|
+
);
|
|
163
|
+
}
|
|
158
164
|
const currentCtx = deps.getCurrentCtx();
|
|
159
165
|
if (currentCtx) {
|
|
160
166
|
const uiConfig = loadConfig(currentCtx.cwd).config.ui;
|
|
@@ -26,6 +26,7 @@ import {
|
|
|
26
26
|
import { readEnabledModelsPatterns } from "../model/model-scope.ts";
|
|
27
27
|
import { isLiveSessionRuntimeAvailable } from "../model/runtime-resolver.ts";
|
|
28
28
|
import { awaitRuntimeWarmup } from "../model/runtime-warmup.ts";
|
|
29
|
+
import { liveAgentContext, registerLiveAgentModel, unregisterLiveAgentModel } from "../model/session-model.ts";
|
|
29
30
|
import { eventToSidechainType, sidechainOutputPath, writeSidechainEntry } from "../output/sidechain-output.ts";
|
|
30
31
|
// NOTE: buildMemoryBlock is intentionally NOT imported here. The agent memory
|
|
31
32
|
// block is injected via renderTaskPrompt().full (the USER prompt), which is
|
|
@@ -705,6 +706,7 @@ export async function runLiveSessionTask(input: LiveSessionSpawnInput): Promise<
|
|
|
705
706
|
});
|
|
706
707
|
const resolvedModel =
|
|
707
708
|
modelFromRegistry(input.modelRegistry, modelRouting.candidates[0] ?? modelRouting.requested) ?? input.parentModel;
|
|
709
|
+
const resolvedModelRef = modelRefToString(resolvedModel) ?? modelRouting.candidates[0];
|
|
708
710
|
// Surface a warning when the caller's requested model was silently replaced.
|
|
709
711
|
if (modelRouting.droppedRequested) {
|
|
710
712
|
appendEventFireAndForget(input.manifest.eventsPath, {
|
|
@@ -839,6 +841,7 @@ export async function runLiveSessionTask(input: LiveSessionSpawnInput): Promise<
|
|
|
839
841
|
appendEvent,
|
|
840
842
|
input.manifest.eventsPath,
|
|
841
843
|
);
|
|
844
|
+
registerLiveAgentModel(agentId, resolvedModelRef ?? "");
|
|
842
845
|
streamOut = createStreamingOutput(input.manifest, input.task.id);
|
|
843
846
|
let controlCursor: LiveAgentControlCursor = { offset: 0 };
|
|
844
847
|
const seenControlRequestIds = new Set<string>();
|
|
@@ -988,7 +991,9 @@ export async function runLiveSessionTask(input: LiveSessionSpawnInput): Promise<
|
|
|
988
991
|
// Phase 3: Wrap session.prompt with timeout for graceful cancellation
|
|
989
992
|
const sessionTimeoutMs = DEFAULT_LIVE_SESSION.responseTimeoutMs;
|
|
990
993
|
try {
|
|
991
|
-
await
|
|
994
|
+
await liveAgentContext.run({ agentId, modelRef: resolvedModelRef ?? "" }, () =>
|
|
995
|
+
promptWithTimeout(session!, effectivePrompt, sessionTimeoutMs, "Live-session"),
|
|
996
|
+
);
|
|
992
997
|
} catch (promptError) {
|
|
993
998
|
const msg = promptError instanceof Error ? promptError.message : String(promptError);
|
|
994
999
|
// P7: fire-and-forget — return value not needed.
|
|
@@ -1181,6 +1186,12 @@ export async function runLiveSessionTask(input: LiveSessionSpawnInput): Promise<
|
|
|
1181
1186
|
error: message,
|
|
1182
1187
|
};
|
|
1183
1188
|
} finally {
|
|
1189
|
+
// Unregister the live-agent model FIRST: a synchronous Map.delete that must run on
|
|
1190
|
+
// every exit path. If skipped (e.g. terminateLiveAgent throws on session.abort()),
|
|
1191
|
+
// hasActiveLiveAgents() stays true for the process lifetime and permanently disables
|
|
1192
|
+
// quota attribution (priority-2 skip) — including the main session's own tracking.
|
|
1193
|
+
// (Review finding H1/M1.)
|
|
1194
|
+
unregisterLiveAgentModel(agentId);
|
|
1184
1195
|
// H6: Unsubscribe listeners FIRST before clearing timer to prevent race
|
|
1185
1196
|
unsubscribe?.();
|
|
1186
1197
|
unsubscribeControlRealtime?.();
|
|
@@ -17,8 +17,10 @@
|
|
|
17
17
|
* the pi events; the spawn paths read it through {@link resolveParentModel}.
|
|
18
18
|
*/
|
|
19
19
|
|
|
20
|
+
import { AsyncLocalStorage } from "node:async_hooks";
|
|
20
21
|
import type { RunModelContext } from "../../state/types.ts";
|
|
21
|
-
import {
|
|
22
|
+
import { logInternalError } from "../../utils/internal-error.ts";
|
|
23
|
+
import { availableModelInfosFromRegistry, modelRefToString, providerOfModelRef } from "./model-fallback.ts";
|
|
22
24
|
|
|
23
25
|
export type SessionModelSource = "model_select" | "session_start" | "none";
|
|
24
26
|
|
|
@@ -31,6 +33,78 @@ interface SessionModelState {
|
|
|
31
33
|
|
|
32
34
|
const state: SessionModelState = { source: "none" };
|
|
33
35
|
|
|
36
|
+
// --- Live-session per-agent quota attribution ---
|
|
37
|
+
//
|
|
38
|
+
// In the opt-in `live-session` runtime, multiple in-process subagents share
|
|
39
|
+
// this ONE module-scoped tracker. The `after_provider_response` event carries
|
|
40
|
+
// no sessionId/model field, so the global `currentSessionModel()` returns the
|
|
41
|
+
// MAIN session's model regardless of which in-process agent actually produced
|
|
42
|
+
// the response. That mis-attributes quota (e.g. a provider-B 429 written under
|
|
43
|
+
// provider-A's key).
|
|
44
|
+
//
|
|
45
|
+
// AsyncLocalStorage propagates each live agent's known model through the
|
|
46
|
+
// async call chain. `resolveProviderForResponse()` checks it first, then falls
|
|
47
|
+
// back to a guard (skip attribution when live agents are active but context
|
|
48
|
+
// is absent — prevents contamination), then the original global tracker (the
|
|
49
|
+
// default child-process path, unchanged).
|
|
50
|
+
|
|
51
|
+
/** Per-agent async context for live-session quota attribution. */
|
|
52
|
+
export const liveAgentContext = new AsyncLocalStorage<{ agentId: string; modelRef: string }>();
|
|
53
|
+
|
|
54
|
+
/** Registered live-session agent models (agentId → "provider/id"). */
|
|
55
|
+
const liveAgentModels = new Map<string, string>();
|
|
56
|
+
|
|
57
|
+
// Cap the tracker to prevent unbounded growth if a caller registers an agent
|
|
58
|
+
// but fails to unregister it (e.g. a crashed/disposed live agent). Matches the
|
|
59
|
+
// precedent in live-agent-manager.ts (MAX_LIVE_AGENTS). When at cap, evict the
|
|
60
|
+
// oldest insertion (Map preserves insertion order); a leaked entry also pins
|
|
61
|
+
// hasActiveLiveAgents()=true, so bounding it matters beyond raw memory.
|
|
62
|
+
const MAX_LIVE_AGENT_MODELS = 5_000;
|
|
63
|
+
|
|
64
|
+
/** Record a live-session agent's resolved model for quota attribution. */
|
|
65
|
+
export function registerLiveAgentModel(agentId: string, model: string): void {
|
|
66
|
+
if (liveAgentModels.size >= MAX_LIVE_AGENT_MODELS && !liveAgentModels.has(agentId)) {
|
|
67
|
+
const oldestKey = liveAgentModels.keys().next().value;
|
|
68
|
+
if (oldestKey !== undefined) {
|
|
69
|
+
logInternalError(
|
|
70
|
+
"session-model.liveAgentModels.cap",
|
|
71
|
+
new Error(`liveAgentModels at cap ${MAX_LIVE_AGENT_MODELS}; evicting oldest ${oldestKey}`),
|
|
72
|
+
);
|
|
73
|
+
liveAgentModels.delete(oldestKey);
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
liveAgentModels.set(agentId, model);
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
/** Remove a live-session agent's model (called in the finally block). */
|
|
80
|
+
export function unregisterLiveAgentModel(agentId: string): void {
|
|
81
|
+
liveAgentModels.delete(agentId);
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
/** Whether any live-session agents are currently registered. */
|
|
85
|
+
export function hasActiveLiveAgents(): boolean {
|
|
86
|
+
return liveAgentModels.size > 0;
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
/**
|
|
90
|
+
* Resolve the provider for an `after_provider_response` event.
|
|
91
|
+
*
|
|
92
|
+
* Priority:
|
|
93
|
+
* 1. Async context from the live-session agent that issued the request
|
|
94
|
+
* (correct per-agent attribution — pi-crew knows each agent's model).
|
|
95
|
+
* 2. Live agents are active but the context didn't propagate → skip
|
|
96
|
+
* attribution entirely (return undefined) to PREVENT cross-agent
|
|
97
|
+
* contamination.
|
|
98
|
+
* 3. No live agents (default child-process runtime) → original behavior:
|
|
99
|
+
* attribute to the global session model's provider.
|
|
100
|
+
*/
|
|
101
|
+
export function resolveProviderForResponse(): string | undefined {
|
|
102
|
+
const ctx = liveAgentContext.getStore();
|
|
103
|
+
if (ctx) return providerOfModelRef(ctx.modelRef);
|
|
104
|
+
if (hasActiveLiveAgents()) return undefined;
|
|
105
|
+
return providerOfModelRef(currentSessionModel());
|
|
106
|
+
}
|
|
107
|
+
|
|
34
108
|
/**
|
|
35
109
|
* Record the model the main session is running. Accepts pi's `Model` object
|
|
36
110
|
* (`{ provider, id }`) or a `"provider/id"` string; anything unrecognized is
|
|
@@ -132,4 +206,5 @@ export function __test_resetSessionModel(): void {
|
|
|
132
206
|
state.thinking = undefined;
|
|
133
207
|
state.source = "none";
|
|
134
208
|
state.updatedAt = undefined;
|
|
209
|
+
liveAgentModels.clear();
|
|
135
210
|
}
|