@danypops/jittor 0.9.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -6
- package/extension/src/benchmark-tui.ts +4 -0
- package/extension/src/capabilities/codex-recovery.ts +127 -0
- package/extension/src/capabilities/http-headers.ts +5 -0
- package/extension/src/capabilities/local-run-telemetry.ts +104 -0
- package/extension/src/capabilities/provider-response-telemetry.ts +96 -0
- package/extension/src/footer.ts +10 -53
- package/extension/src/index.ts +100 -276
- package/extension/src/session-identity.ts +20 -0
- package/extension/src/tui.ts +20 -11
- package/package.json +1 -1
- package/src/adapters/sqlite-metric-store.ts +11 -2
- package/src/adapters/sqlite-session-identity-store.ts +45 -0
- package/src/cli-commands/benchmarks.ts +140 -0
- package/src/cli-commands/compaction.ts +17 -0
- package/src/cli-commands/context.ts +49 -0
- package/src/cli-commands/metrics.ts +296 -0
- package/src/cli-commands/op.ts +40 -0
- package/src/cli-commands/route-args.ts +15 -0
- package/src/cli-commands/router.ts +207 -0
- package/src/cli-commands/service-daemon.ts +72 -0
- package/src/cli-commands/session.ts +42 -0
- package/src/cli-commands/support.ts +33 -0
- package/src/cli.ts +42 -766
- package/src/constants.ts +7 -0
- package/src/daemon.ts +13 -3
- package/src/db.ts +15 -1
- package/src/domain/task-cost.ts +44 -9
- package/src/operations/benchmark-operations.ts +12 -0
- package/src/operations/context-operations.ts +30 -0
- package/src/operations/metrics-operations.ts +77 -0
- package/src/operations/model-ranking-operations.ts +16 -0
- package/src/operations/router-operations.ts +19 -0
- package/src/operations/session-identity-operations.ts +15 -0
- package/src/operations/session-scope.ts +31 -0
- package/src/operations/types.ts +3 -0
- package/src/ports/metric-store.ts +2 -0
- package/src/ports/router-controller.ts +9 -9
- package/src/ports/session-identity-store.ts +5 -0
- package/src/providers/telemetry-sources.ts +2 -1
- package/src/router.ts +124 -67
- package/src/service.ts +60 -118
- package/src/session-identity-service.ts +55 -0
package/README.md
CHANGED
|
@@ -39,7 +39,7 @@ Provider adapters currently include official OpenRouter key/usage/model telemetr
|
|
|
39
39
|
|
|
40
40
|
The third-party `anthropic-vertex` provider (Anthropic Claude models served through Google Vertex, e.g. via `@twogiants/pi-anthropic-vertex`) is tracked separately from both of the above: it reuses Pi's own Anthropic Messages stream implementation with Anthropic's official `@anthropic-ai/vertex-sdk` client, so its wire shape is Anthropic's, but its quota accounting is Google's. Jittor applies Google Vertex's failure classification to it (real-world reports confirm its 429s still carry GCP's own quota-exceeded shape even through Anthropic's own SDK) and, best-effort, also checks for genuine Anthropic rate-limit response headers on it, since it is unverified whether this specific passthrough ever forwards them. Either way, every metric is tagged `anthropic-vertex`, never blended into direct Anthropic's `anthropic` source or Pi's unrelated native `google-vertex` provider, since each represents a different account/quota pool. Its footer budget (labeled `vtok`/`vreq` when headers are observed) stays `null` (may still resolve) rather than `undefined` (provably impossible) until it's confirmed one way or the other.
|
|
41
41
|
|
|
42
|
-
The native Pi extension preflights input and every provider turn, applies model/thinking decisions, records response headers and finalized usage through the daemon, and blocks requests when required telemetry is unsafe. It follows Pi's current authenticated model/provider and synchronizes Pi's available models before every decision, so unavailable catalog routes are never selected. Its responsive integrated footer groups repository and model identity with cumulative usage, a color-coded context-window bar, and current-provider budget telemetry. Codex shows the active model's bounded quota as a draining remaining-budget bar with reset and freshness information. OpenRouter uses the same drain semantics when its official key telemetry exposes a configured limit and remaining balance; keys without a limit remain honest text-only spend and never receive a fabricated denominator. Anthropic shows the same drain semantics from its most-restrictive-in-effect token bucket, falling back to the request bucket when no token telemetry has been observed yet. During Pi compaction, the context bar drains against a learned median duration estimated from the last few completed compactions (bounded to the most recent 20 samples, requiring at least 3 before trusting it)
|
|
42
|
+
The native Pi extension preflights input and every provider turn, applies model/thinking decisions, records response headers and finalized usage through the daemon, and blocks requests when required telemetry is unsafe. It follows Pi's current authenticated model/provider and synchronizes Pi's available models before every decision, so unavailable catalog routes are never selected. Mutable route state is scoped by Pi session, so concurrent sessions cannot replace each other's active provider or footer budget selection. Each session registers an opaque secret with the daemon at `session_start` (best-effort; a registration failure leaves that session unarmored rather than blocked) and presents it on every router-mutating call for the rest of its lifetime; an unregistered `session_id` continues to mutate exactly as before, so this is additive armor, not a breaking change for other callers of the same API. A configured required budget source still fails closed; a provider with no enforceable budget window continues explicitly monitor-only. Its responsive integrated footer groups repository and model identity with cumulative usage, a color-coded context-window bar, and current-provider budget telemetry. Codex shows the active model's bounded quota as a draining remaining-budget bar with reset and freshness information. OpenRouter uses the same drain semantics when its official key telemetry exposes a configured limit and remaining balance; keys without a limit remain honest text-only spend and never receive a fabricated denominator. Anthropic shows the same drain semantics from its most-restrictive-in-effect token bucket, falling back to the request bucket when no token telemetry has been observed yet. During Pi compaction, the context bar drains from its captured fill against a learned median duration estimated from the last few completed compactions (bounded to the most recent 20 samples, requiring at least 3 before trusting it). It never renders a timer. Until enough evidence exists, the bar does not drain; it blinks in place at its captured fill. Run `jittor compaction estimate [--json]` to inspect the current estimate and its confidence directly. Unknown and stale telemetry are marked explicitly. Run `/jittor` for the consolidated Settings TUI (its default action), or `/jittor status` for detailed burn pressure, freshness, route state, and confirmed emergency-halt/override controls.
|
|
43
43
|
|
|
44
44
|
Jittor currently registers no model-callable native tools, so Pi's native model `content` versus renderer `details` contract is explicitly not applicable. Daemon JSON, CLI `--json`, human CLI output, command notifications, panels, and the footer remain separate bounded channels. See [`docs/OUTPUT_CHANNELS.md`](docs/OUTPUT_CHANNELS.md) for the conformance matrix and the requirements that apply if a native tool is introduced later.
|
|
45
45
|
|
|
@@ -72,7 +72,7 @@ The graph fetches metrics per distinct provider/model scope (`jittor metrics dis
|
|
|
72
72
|
|
|
73
73
|
### Cost per Papyrus task
|
|
74
74
|
|
|
75
|
-
Jittor observes Papyrus's task-focus lifecycle in real time over a shared Pi extension event bus (`papyrus.task-focus.v1`) -- Papyrus never depends on Jittor, it only broadcasts which task is currently focused. Every token/cost metric Jittor already records on a finalized Pi assistant message is tagged with the currently focused task's id the moment it is recorded (no time-window estimation, no new instrumentation). A paused or cleared focus stops tagging; spend recorded with nothing focused is reported separately as unattributed, never dropped or folded into an invented task. Run `jittor metrics cost-by-task --since <ms> --until <ms> [--json]` for a bounded per-task breakdown of cost and input/output/cache tokens.
|
|
75
|
+
Jittor observes Papyrus's task-focus lifecycle in real time over a shared Pi extension event bus (`papyrus.task-focus.v1`) -- Papyrus never depends on Jittor, it only broadcasts which task is currently focused. Every token/cost metric Jittor already records on a finalized Pi assistant message is tagged with the currently focused task's id, and the provider/model/thinking level active at that moment, the instant it is recorded (no time-window estimation, no new instrumentation). A paused or cleared focus stops tagging; spend recorded with nothing focused is reported separately as unattributed, never dropped or folded into an invented task. Run `jittor metrics cost-by-task --since <ms> --until <ms> [--json]` for a bounded per-task breakdown of cost and input/output/cache tokens, broken down further by which provider/model/thinking combination each task actually spent on.
|
|
76
76
|
|
|
77
77
|
Series are colored with a categorical palette chosen to avoid this UI's own status colors ("success"/"warning"/"error" already mean something specific elsewhere in this panel, so reusing them for arbitrary model identity would make a model's bar segment look like a warning or a failure) and instead reuses the theme's syntax-highlighting roles, which are already tuned by theme authors to stay mutually distinguishable on screen — the same design problem as a categorical data palette. Once more series are active than there are hues, a series reuses a hue in bold rather than repeating an indistinguishable color. Multiple models active within the same cumulative time frame are rendered as one bar stacked by color, not separate bars.
|
|
78
78
|
|
|
@@ -112,6 +112,7 @@ Every daemon operation is reachable from the CLI through the authenticated typed
|
|
|
112
112
|
|
|
113
113
|
```text
|
|
114
114
|
jittor metrics record --source <s> --scope <s> --metric <s> --value <number|null> --unit <unit> [--observed-at <ms>] [--attributes <json>] [--json]
|
|
115
|
+
jittor metrics record-batch --observations <json-array, max 100> [--json]
|
|
115
116
|
jittor metrics query [--source <s>] [--scope <s>] [--metric <s>] [--since <ms>] [--until <ms>] [--limit <n>] [--order asc|desc] [--json]
|
|
116
117
|
jittor metrics prune --before <ms> [--json]
|
|
117
118
|
jittor metrics distinct-scopes --source <s> --since <ms> --until <ms> [--limit 1..40] [--json]
|
|
@@ -120,10 +121,12 @@ jittor metrics prune --before <ms> [--force] [--json] # force required if befor
|
|
|
120
121
|
jittor service checkpoint [--json]
|
|
121
122
|
jittor telemetry poll [--json]
|
|
122
123
|
jittor compaction estimate [--json]
|
|
123
|
-
jittor
|
|
124
|
-
jittor
|
|
125
|
-
jittor router
|
|
126
|
-
jittor router
|
|
124
|
+
jittor session register --session-id <id> [--json]
|
|
125
|
+
jittor session release --session-id <id> [--session-secret <secret>] [--json]
|
|
126
|
+
jittor router status|decide|pause|resume|clear-override [--session-id <id>] [--session-secret <secret>] [--json]
|
|
127
|
+
jittor router override --route <provider/model@thinking> [--expires-at <ms>] [--session-id <id>] [--session-secret <secret>] [--json]
|
|
128
|
+
jittor router current-route --route <provider/model@thinking> [--session-id <id>] [--session-secret <secret>] [--json]
|
|
129
|
+
jittor router available-routes [--route <provider/model@thinking> ...] [--session-id <id>] [--session-secret <secret>] [--json]
|
|
127
130
|
jittor op <operation> [--input <json>]
|
|
128
131
|
```
|
|
129
132
|
|
|
@@ -11,6 +11,7 @@ import {
|
|
|
11
11
|
} from "../../src/constants.ts";
|
|
12
12
|
import type { ModelTaskDomain, ModelTaskType } from "../../src/domain/model-observation.ts";
|
|
13
13
|
import type { ModelCandidate, ModelRankingResult, RankedModel, UtilityComponentName } from "../../src/domain/model-ranking.ts";
|
|
14
|
+
import { sessionSecretField } from "./session-identity.ts";
|
|
14
15
|
|
|
15
16
|
export interface BenchmarkPanelClient {
|
|
16
17
|
call(operation: string, input: unknown): Promise<any>;
|
|
@@ -72,8 +73,11 @@ export async function showBenchmarkPanel(
|
|
|
72
73
|
type: ModelTaskType,
|
|
73
74
|
): Promise<void> {
|
|
74
75
|
for (;;) {
|
|
76
|
+
const session_id = ctx.sessionManager.getSessionId();
|
|
75
77
|
const result = await client.call("models.rank", {
|
|
76
78
|
candidates,
|
|
79
|
+
session_id,
|
|
80
|
+
...sessionSecretField(session_id),
|
|
77
81
|
scopeAuthority: "available-models",
|
|
78
82
|
domain,
|
|
79
83
|
type,
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
|
|
2
|
+
import {
|
|
3
|
+
CODEX_RECOVERY_ATTEMPT_WINDOW_MS,
|
|
4
|
+
CODEX_RECOVERY_BASE_DELAY_MS,
|
|
5
|
+
CODEX_RECOVERY_JITTER_RATIO,
|
|
6
|
+
CODEX_RECOVERY_MAX_ATTEMPTS,
|
|
7
|
+
CODEX_RECOVERY_MAX_DELAY_MS,
|
|
8
|
+
MILLISECONDS_PER_MINUTE,
|
|
9
|
+
MILLISECONDS_PER_SECOND,
|
|
10
|
+
} from "../../../src/constants.ts";
|
|
11
|
+
import { CodexRecoveryPolicy, classifyCodexFailure, type CodexFailureKind, type CodexFailureMetadata } from "../../../src/domain/codex-recovery.ts";
|
|
12
|
+
import type { CodexRecoveryControl } from "../settings.ts";
|
|
13
|
+
import { headerValue } from "./http-headers.ts";
|
|
14
|
+
|
|
15
|
+
export interface CodexRecoveryRuntime {
|
|
16
|
+
now(): number;
|
|
17
|
+
random(): number;
|
|
18
|
+
setTimeout(callback: () => void | Promise<void>, delayMs: number): unknown;
|
|
19
|
+
clearTimeout(handle: unknown): void;
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
export const SYSTEM_RECOVERY_RUNTIME: CodexRecoveryRuntime = {
|
|
23
|
+
now: Date.now,
|
|
24
|
+
random: Math.random,
|
|
25
|
+
setTimeout(callback, delayMs) { return setTimeout(() => { void callback(); }, delayMs); },
|
|
26
|
+
clearTimeout(handle) { clearTimeout(handle as ReturnType<typeof setTimeout>); },
|
|
27
|
+
};
|
|
28
|
+
|
|
29
|
+
/**
|
|
30
|
+
* Codex's settled-turn hidden-retry recovery, as its own capability: tracks the most recent
|
|
31
|
+
* Codex response (status/retry-after) across the current turn, classifies a finalized failure,
|
|
32
|
+
* and schedules at most one bounded, jittered follow-up once Pi's own turn has genuinely
|
|
33
|
+
* settled. Fully self-contained -- the only external dependencies are Pi's own message-send API,
|
|
34
|
+
* the persisted on/off control, and a runtime the tests can fake (timers, randomness, clock).
|
|
35
|
+
*/
|
|
36
|
+
export class CodexRecoveryCapability {
|
|
37
|
+
private readonly policy: CodexRecoveryPolicy;
|
|
38
|
+
private lastResponse: CodexFailureMetadata = {};
|
|
39
|
+
private timer: unknown;
|
|
40
|
+
private cooldown: { until: number; attempt: number; failureKind: CodexFailureKind } | undefined;
|
|
41
|
+
|
|
42
|
+
constructor(
|
|
43
|
+
private readonly pi: ExtensionAPI,
|
|
44
|
+
private readonly control: CodexRecoveryControl,
|
|
45
|
+
private readonly runtime: CodexRecoveryRuntime,
|
|
46
|
+
) {
|
|
47
|
+
this.policy = new CodexRecoveryPolicy({
|
|
48
|
+
baseDelayMs: CODEX_RECOVERY_BASE_DELAY_MS,
|
|
49
|
+
maxDelayMs: CODEX_RECOVERY_MAX_DELAY_MS,
|
|
50
|
+
maxAttempts: CODEX_RECOVERY_MAX_ATTEMPTS,
|
|
51
|
+
attemptWindowMs: CODEX_RECOVERY_ATTEMPT_WINDOW_MS,
|
|
52
|
+
jitterRatio: CODEX_RECOVERY_JITTER_RATIO,
|
|
53
|
+
}, runtime.random);
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
/** Clears the tracked response at the start of every new turn, before any Codex response for it has arrived. */
|
|
57
|
+
resetTurn(): void {
|
|
58
|
+
this.lastResponse = {};
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
notifyResponse(status: number, headers: Record<string, string>): void {
|
|
62
|
+
this.lastResponse = { status, ...(headerValue(headers, "retry-after") ? { retryAfter: headerValue(headers, "retry-after") } : {}) };
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
notifyMessageEnd(stopReason: string, errorMessage: string | undefined): void {
|
|
66
|
+
if (stopReason === "error") {
|
|
67
|
+
const failure = classifyCodexFailure(errorMessage, this.lastResponse);
|
|
68
|
+
if (this.control.isCodexRecoveryEnabled() && failure.transient) this.policy.observeFailure(failure, this.runtime.now());
|
|
69
|
+
else this.cancel(true);
|
|
70
|
+
} else if (stopReason !== "aborted") {
|
|
71
|
+
this.cancel(true);
|
|
72
|
+
}
|
|
73
|
+
this.lastResponse = {};
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
cancel(resetPolicy: boolean): void {
|
|
77
|
+
if (this.timer !== undefined) this.runtime.clearTimeout(this.timer);
|
|
78
|
+
this.timer = undefined;
|
|
79
|
+
this.cooldown = undefined;
|
|
80
|
+
if (resetPolicy) this.policy.cancel();
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
statusText(): string {
|
|
84
|
+
const now = this.runtime.now();
|
|
85
|
+
const state = this.policy.state(now);
|
|
86
|
+
const enabled = this.control.isCodexRecoveryEnabled();
|
|
87
|
+
const attempt = this.cooldown?.attempt ?? (state.pending ? state.attempts + 1 : state.attempts);
|
|
88
|
+
const phase = this.cooldown
|
|
89
|
+
? `cooldown ${Math.ceil(Math.max(0, this.cooldown.until - now) / MILLISECONDS_PER_SECOND)}s`
|
|
90
|
+
: state.pending ? "pending"
|
|
91
|
+
: state.attempts >= CODEX_RECOVERY_MAX_ATTEMPTS ? "exhausted"
|
|
92
|
+
: state.attempts > 0 ? "waiting" : "idle";
|
|
93
|
+
const failureKind = this.cooldown?.failureKind ?? state.lastFailureKind;
|
|
94
|
+
return [
|
|
95
|
+
`Codex recovery: ${enabled ? "on" : "off"}`,
|
|
96
|
+
phase,
|
|
97
|
+
`attempt ${attempt}/${CODEX_RECOVERY_MAX_ATTEMPTS}`,
|
|
98
|
+
`window ${CODEX_RECOVERY_ATTEMPT_WINDOW_MS / MILLISECONDS_PER_MINUTE}m`,
|
|
99
|
+
...(failureKind ? [failureKind] : []),
|
|
100
|
+
].join(" · ");
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
scheduleIfIdle(ctx: ExtensionContext): void {
|
|
104
|
+
if (!this.control.isCodexRecoveryEnabled() || this.timer !== undefined || !ctx.isIdle() || ctx.hasPendingMessages()) return;
|
|
105
|
+
const plan = this.policy.plan(this.runtime.now());
|
|
106
|
+
if (plan.action === "exhausted") {
|
|
107
|
+
this.policy.abandonFailure();
|
|
108
|
+
if (ctx.hasUI) ctx.ui.notify(`Jittor Codex recovery stopped: ${plan.reason}.`, "warning");
|
|
109
|
+
return;
|
|
110
|
+
}
|
|
111
|
+
if (plan.action !== "schedule") return;
|
|
112
|
+
this.cooldown = { until: this.runtime.now() + plan.delayMs, attempt: plan.attempt, failureKind: plan.failureKind };
|
|
113
|
+
this.timer = this.runtime.setTimeout(async () => {
|
|
114
|
+
this.timer = undefined;
|
|
115
|
+
this.cooldown = undefined;
|
|
116
|
+
if (!ctx.isIdle() || ctx.hasPendingMessages()) return;
|
|
117
|
+
const attempt = this.policy.recordAttempt(this.runtime.now());
|
|
118
|
+
if (!attempt) return;
|
|
119
|
+
this.pi.sendMessage({
|
|
120
|
+
customType: "jittor-codex-recovery",
|
|
121
|
+
content: `Retry the previous Codex request after a transient ${attempt.failureKind} failure. Automatic recovery attempt ${attempt.attempt} of ${CODEX_RECOVERY_MAX_ATTEMPTS}.`,
|
|
122
|
+
display: false,
|
|
123
|
+
details: { attempt: attempt.attempt, failureKind: attempt.failureKind },
|
|
124
|
+
}, { triggerTurn: true, deliverAs: "followUp" });
|
|
125
|
+
}, plan.delayMs);
|
|
126
|
+
}
|
|
127
|
+
}
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
/** Case-insensitive header lookup -- Pi's provider-response event headers are a plain Record, not a Headers instance. */
|
|
2
|
+
export function headerValue(headers: Record<string, string>, name: string): string | undefined {
|
|
3
|
+
const expected = name.toLowerCase();
|
|
4
|
+
return Object.entries(headers).find(([key]) => key.toLowerCase() === expected)?.[1];
|
|
5
|
+
}
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
import { classifyTaskFromTools, modelRunMetrics, type ModelRunObservation } from "../../../src/domain/model-observation.ts";
|
|
2
|
+
import type { MetricObservation } from "../../../src/domain/metric.ts";
|
|
3
|
+
|
|
4
|
+
export interface ActiveLocalModelRun {
|
|
5
|
+
runId: string;
|
|
6
|
+
startedAt: number;
|
|
7
|
+
firstTokenAt: number | null;
|
|
8
|
+
providerResponses: number;
|
|
9
|
+
toolNames: string[];
|
|
10
|
+
toolCalls: number;
|
|
11
|
+
toolFailures: number;
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
/**
|
|
15
|
+
* Content-free local model observations derived only from Pi's public lifecycle: TTFT, wall
|
|
16
|
+
* latency, tool-loop counts and failures, and a bounded tool-name list used only to derive
|
|
17
|
+
* domain/type classification -- never prompts, responses, tool arguments/results, or project
|
|
18
|
+
* paths. Owns the one in-flight run plus the most recently completed one (for `/jittor outcome`).
|
|
19
|
+
*/
|
|
20
|
+
export class LocalRunTelemetry {
|
|
21
|
+
private active: ActiveLocalModelRun | undefined;
|
|
22
|
+
private lastCompleted: ModelRunObservation | undefined;
|
|
23
|
+
private sequence = 0;
|
|
24
|
+
|
|
25
|
+
beginTurn(timestamp: number): void {
|
|
26
|
+
this.active = {
|
|
27
|
+
runId: `local-${timestamp}-${++this.sequence}`,
|
|
28
|
+
startedAt: timestamp,
|
|
29
|
+
firstTokenAt: null,
|
|
30
|
+
providerResponses: 0,
|
|
31
|
+
toolNames: [],
|
|
32
|
+
toolCalls: 0,
|
|
33
|
+
toolFailures: 0,
|
|
34
|
+
};
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
discardTurn(): void {
|
|
38
|
+
this.active = undefined;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
onMessageUpdate(assistantMessageEventType: string): void {
|
|
42
|
+
if (!this.active || this.active.firstTokenAt !== null) return;
|
|
43
|
+
if (["text_delta", "thinking_delta", "toolcall_delta"].includes(assistantMessageEventType)) this.active.firstTokenAt = Date.now();
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
onToolExecutionEnd(toolName: string, isError: boolean): void {
|
|
47
|
+
if (!this.active) return;
|
|
48
|
+
this.active.toolCalls += 1;
|
|
49
|
+
if (isError) this.active.toolFailures += 1;
|
|
50
|
+
if (this.active.toolNames.length < 100) this.active.toolNames.push(toolName);
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
onProviderResponse(): void {
|
|
54
|
+
if (this.active) this.active.providerResponses += 1;
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
/** Finalizes the active run against a completed assistant turn_end message, returning metrics to record (empty if the message shape doesn't match a completed assistant turn). */
|
|
58
|
+
completeTurn(message: unknown, thinkingLevel: string): MetricObservation[] {
|
|
59
|
+
const active = this.active;
|
|
60
|
+
this.active = undefined;
|
|
61
|
+
if (!active || typeof message !== "object" || message === null || Array.isArray(message)) return [];
|
|
62
|
+
const value = message as Record<string, unknown>;
|
|
63
|
+
if (value["role"] !== "assistant" || typeof value["provider"] !== "string" || typeof value["model"] !== "string") return [];
|
|
64
|
+
const usage = typeof value["usage"] === "object" && value["usage"] !== null ? value["usage"] as Record<string, unknown> : {};
|
|
65
|
+
const amount = (name: string): number => typeof usage[name] === "number" && Number.isFinite(usage[name]) ? usage[name] as number : 0;
|
|
66
|
+
const cost = typeof usage["cost"] === "object" && usage["cost"] !== null && typeof (usage["cost"] as Record<string, unknown>)["total"] === "number"
|
|
67
|
+
? (usage["cost"] as Record<string, number>)["total"] ?? 0 : 0;
|
|
68
|
+
const stopReason = ["stop", "length", "toolUse", "error", "aborted"].includes(String(value["stopReason"]))
|
|
69
|
+
? value["stopReason"] as ModelRunObservation["stopReason"] : "unknown";
|
|
70
|
+
const completedAt = Math.max(Date.now(), active.firstTokenAt ?? active.startedAt, active.startedAt);
|
|
71
|
+
this.lastCompleted = {
|
|
72
|
+
runId: active.runId,
|
|
73
|
+
provider: value["provider"],
|
|
74
|
+
model: value["model"],
|
|
75
|
+
thinking: thinkingLevel,
|
|
76
|
+
...classifyTaskFromTools(active.toolNames),
|
|
77
|
+
startedAt: active.startedAt,
|
|
78
|
+
firstTokenAt: active.firstTokenAt,
|
|
79
|
+
completedAt,
|
|
80
|
+
inputTokens: amount("input"),
|
|
81
|
+
outputTokens: amount("output"),
|
|
82
|
+
cacheReadTokens: amount("cacheRead"),
|
|
83
|
+
cacheWriteTokens: amount("cacheWrite"),
|
|
84
|
+
costUsd: Number.isFinite(cost) && cost >= 0 ? cost : 0,
|
|
85
|
+
providerResponses: Math.max(1, active.providerResponses),
|
|
86
|
+
toolCalls: active.toolCalls,
|
|
87
|
+
toolFailures: active.toolFailures,
|
|
88
|
+
stopReason,
|
|
89
|
+
explicitOutcome: "unknown",
|
|
90
|
+
};
|
|
91
|
+
return modelRunMetrics(this.lastCompleted);
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
/** Builds the single outcome-accepted/outcome-accepted=0 metric for the most recently completed run, or null if none exists yet. */
|
|
95
|
+
explicitOutcomeMetric(explicitOutcome: "accepted" | "rejected"): MetricObservation | null {
|
|
96
|
+
if (!this.lastCompleted) return null;
|
|
97
|
+
return modelRunMetrics({ ...this.lastCompleted, explicitOutcome }).find((metric) => metric.metric === "outcome-accepted") ?? null;
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
reset(): void {
|
|
101
|
+
this.active = undefined;
|
|
102
|
+
this.lastCompleted = undefined;
|
|
103
|
+
}
|
|
104
|
+
}
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
import type { MetricObservation } from "../../../src/domain/metric.ts";
|
|
2
|
+
import { hasAnthropicRateLimitHeaders, parseAnthropicRateLimitHeaders } from "../../../src/providers/anthropic-contracts.ts";
|
|
3
|
+
import { parseCodexRateLimitHeaders } from "../../../src/providers/codex.ts";
|
|
4
|
+
import { classifyGoogleVertexFailure, googleVertexFailureMetrics, type GoogleVertexFailureMetadata } from "../../../src/providers/google-vertex-contracts.ts";
|
|
5
|
+
import { headerValue } from "./http-headers.ts";
|
|
6
|
+
|
|
7
|
+
export interface ProviderTelemetryClient {
|
|
8
|
+
call(operation: string, input: unknown): Promise<any>;
|
|
9
|
+
}
|
|
10
|
+
|
|
11
|
+
async function recordMetrics(client: ProviderTelemetryClient, metrics: MetricObservation[]): Promise<void> {
|
|
12
|
+
if (metrics.length === 0) return;
|
|
13
|
+
await client.call("metrics.record_batch", { observations: metrics });
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
/**
|
|
17
|
+
* Bounded telemetry derived directly from provider HTTP responses and finalized messages, for
|
|
18
|
+
* every provider except Codex (whose response tracking is settled-turn-recovery's own concern --
|
|
19
|
+
* see codex-recovery.ts). Anthropic and anthropic-vertex official rate-limit headers become
|
|
20
|
+
* budget metrics; Google Vertex and anthropic-vertex failures become bounded, content-free
|
|
21
|
+
* failure-count metrics classified from GCP's own `google.rpc.Status` shape. anthropic-vertex is
|
|
22
|
+
* tracked distinctly from both google-vertex (different code path, different quota pool) and
|
|
23
|
+
* direct anthropic (different transport), never conflated with either.
|
|
24
|
+
*/
|
|
25
|
+
export class ProviderResponseTelemetry {
|
|
26
|
+
private lastGoogleVertexResponse: GoogleVertexFailureMetadata = {};
|
|
27
|
+
private lastAnthropicVertexResponse: GoogleVertexFailureMetadata = {};
|
|
28
|
+
|
|
29
|
+
resetTurn(): void {
|
|
30
|
+
this.lastGoogleVertexResponse = {};
|
|
31
|
+
this.lastAnthropicVertexResponse = {};
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
async handleProviderResponse(
|
|
35
|
+
client: ProviderTelemetryClient,
|
|
36
|
+
provider: string | undefined,
|
|
37
|
+
status: number,
|
|
38
|
+
headers: Record<string, string>,
|
|
39
|
+
notifySchemaDrift: (message: string) => void,
|
|
40
|
+
): Promise<void> {
|
|
41
|
+
if (provider === "anthropic") {
|
|
42
|
+
const parsedHeaders = new Headers(headers);
|
|
43
|
+
if (hasAnthropicRateLimitHeaders(parsedHeaders)) {
|
|
44
|
+
try {
|
|
45
|
+
await recordMetrics(client, parseAnthropicRateLimitHeaders(parsedHeaders, Date.now()).metrics);
|
|
46
|
+
} catch {
|
|
47
|
+
notifySchemaDrift("Anthropic telemetry schema drift");
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
if (provider === "anthropic-vertex") {
|
|
52
|
+
// Best-effort only: unverified whether this passthrough ever forwards Anthropic's own
|
|
53
|
+
// rate-limit headers. If it doesn't, hasAnthropicRateLimitHeaders is false and nothing is
|
|
54
|
+
// recorded -- the same honest default as every other unconfirmed signal in this module.
|
|
55
|
+
const parsedHeaders = new Headers(headers);
|
|
56
|
+
if (hasAnthropicRateLimitHeaders(parsedHeaders)) {
|
|
57
|
+
try {
|
|
58
|
+
await recordMetrics(client, parseAnthropicRateLimitHeaders(parsedHeaders, Date.now(), "anthropic-vertex").metrics);
|
|
59
|
+
} catch {
|
|
60
|
+
notifySchemaDrift("Anthropic-on-Vertex telemetry schema drift");
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
// Well-evidenced regardless of headers: GCP's own quota system fronts this transport, so the
|
|
64
|
+
// same failure classification as google-vertex applies below.
|
|
65
|
+
this.lastAnthropicVertexResponse = { status, ...(headerValue(headers, "retry-after") ? { retryAfter: headerValue(headers, "retry-after") } : {}) };
|
|
66
|
+
}
|
|
67
|
+
if (provider === "google-vertex") {
|
|
68
|
+
this.lastGoogleVertexResponse = { status, ...(headerValue(headers, "retry-after") ? { retryAfter: headerValue(headers, "retry-after") } : {}) };
|
|
69
|
+
}
|
|
70
|
+
if (Object.keys(headers).some((name) => name.toLowerCase().startsWith("x-codex-"))) {
|
|
71
|
+
try {
|
|
72
|
+
const updates = parseCodexRateLimitHeaders(new Headers(headers), Date.now());
|
|
73
|
+
await recordMetrics(client, updates.flatMap((update) => update.metrics));
|
|
74
|
+
} catch {
|
|
75
|
+
notifySchemaDrift("Codex telemetry schema drift");
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
async handleMessageEnd(client: ProviderTelemetryClient, provider: string | undefined, stopReason: string | undefined, errorMessage: string | undefined): Promise<void> {
|
|
81
|
+
if (provider === "google-vertex") {
|
|
82
|
+
if (stopReason === "error") {
|
|
83
|
+
const failure = classifyGoogleVertexFailure(errorMessage, this.lastGoogleVertexResponse);
|
|
84
|
+
await recordMetrics(client, googleVertexFailureMetrics(failure, Date.now())).catch(() => undefined);
|
|
85
|
+
}
|
|
86
|
+
this.lastGoogleVertexResponse = {};
|
|
87
|
+
}
|
|
88
|
+
if (provider === "anthropic-vertex") {
|
|
89
|
+
if (stopReason === "error") {
|
|
90
|
+
const failure = classifyGoogleVertexFailure(errorMessage, this.lastAnthropicVertexResponse);
|
|
91
|
+
await recordMetrics(client, googleVertexFailureMetrics(failure, Date.now(), "anthropic-vertex")).catch(() => undefined);
|
|
92
|
+
}
|
|
93
|
+
this.lastAnthropicVertexResponse = {};
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
}
|
package/extension/src/footer.ts
CHANGED
|
@@ -12,7 +12,6 @@ import {
|
|
|
12
12
|
MILLISECONDS_PER_DAY,
|
|
13
13
|
MILLISECONDS_PER_HOUR,
|
|
14
14
|
MILLISECONDS_PER_MINUTE,
|
|
15
|
-
MILLISECONDS_PER_SECOND,
|
|
16
15
|
TELEMETRY_STALE_AFTER_MS,
|
|
17
16
|
} from "../../src/constants.ts";
|
|
18
17
|
|
|
@@ -102,7 +101,7 @@ function sanitize(value: string): string {
|
|
|
102
101
|
}
|
|
103
102
|
|
|
104
103
|
function usageTotals(context: FooterContext): UsageTotals {
|
|
105
|
-
let input = 0, output = 0, cacheRead = 0, cacheWrite = 0, cost = 0
|
|
104
|
+
let input = 0, output = 0, cacheRead = 0, cacheWrite = 0, cost = 0;
|
|
106
105
|
for (const entry of context.sessionManager.getEntries()) {
|
|
107
106
|
if (entry.type !== "message" || entry.message?.role !== "assistant") continue;
|
|
108
107
|
const usage = entry.message.usage;
|
|
@@ -111,10 +110,9 @@ function usageTotals(context: FooterContext): UsageTotals {
|
|
|
111
110
|
cacheRead += usage?.cacheRead ?? 0;
|
|
112
111
|
cacheWrite += usage?.cacheWrite ?? 0;
|
|
113
112
|
cost += usage?.cost?.total ?? 0;
|
|
114
|
-
const prompt = (usage?.input ?? 0) + (usage?.cacheRead ?? 0) + (usage?.cacheWrite ?? 0);
|
|
115
|
-
if (prompt > 0) cacheHit = (usage.cacheRead ?? 0) / prompt * 100;
|
|
116
113
|
}
|
|
117
|
-
|
|
114
|
+
const prompt = input + cacheRead + cacheWrite;
|
|
115
|
+
return { input, output, cacheRead, cacheWrite, cost, ...(prompt > 0 ? { cacheHit: cacheRead / prompt * 100 } : {}) };
|
|
118
116
|
}
|
|
119
117
|
|
|
120
118
|
function barWidth(width: number): number {
|
|
@@ -136,58 +134,21 @@ function fillColor(fraction: number | null): FooterColor {
|
|
|
136
134
|
return "dim";
|
|
137
135
|
}
|
|
138
136
|
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
*
|
|
144
|
-
* the estimate fetch has not resolved yet — there is no real duration to drain against, so the
|
|
145
|
-
* fill holds steady at the fraction observed when compaction started; the blink alone (see
|
|
146
|
-
* compactionBarGlyph) communicates liveness without fabricating a rate.
|
|
147
|
-
*/
|
|
148
|
-
function compactionFraction(progress: CompactionProgress, width: number, now: number): number {
|
|
149
|
-
if (progress.confidence === "learned" && typeof progress.estimatedMs === "number" && progress.estimatedMs > 0) {
|
|
150
|
-
const elapsed = Math.max(0, now - progress.startedAt);
|
|
151
|
-
return Math.max(0, Math.min(1, 1 - (elapsed / progress.estimatedMs)));
|
|
152
|
-
}
|
|
153
|
-
return Math.min(1, Math.max(0, progress.initialFraction));
|
|
137
|
+
function compactionFraction(progress: CompactionProgress, now: number): number {
|
|
138
|
+
const initial = Math.min(1, Math.max(0, progress.initialFraction));
|
|
139
|
+
if (progress.confidence !== "learned" || typeof progress.estimatedMs !== "number" || progress.estimatedMs <= 0) return initial;
|
|
140
|
+
const elapsedFraction = Math.max(0, Math.min(1, (now - progress.startedAt) / progress.estimatedMs));
|
|
141
|
+
return initial * (1 - elapsedFraction);
|
|
154
142
|
}
|
|
155
143
|
|
|
156
|
-
/**
|
|
157
|
-
* Liveness blink independent of whether the drain bar reflects a learned estimate or the
|
|
158
|
-
* fixed-rate cold-start fallback: it does not claim to know how long compaction will take, only
|
|
159
|
-
* that it has not stalled. It toggles once per render tick so a single owned interval (installed
|
|
160
|
-
* in beginCompactionUi) drives both the drain and the blink — no extra timer is created here.
|
|
161
|
-
*/
|
|
162
144
|
export function compactionBlinkOn(startedAt: number, now: number, halfPeriodMs = FOOTER_COMPACTION_BLINK_HALF_PERIOD_MS): boolean {
|
|
163
145
|
const elapsed = Math.max(0, now - startedAt);
|
|
164
146
|
return Math.floor(elapsed / halfPeriodMs) % 2 === 0;
|
|
165
147
|
}
|
|
166
148
|
|
|
167
|
-
/**
|
|
168
|
-
* The compaction signal lives in the bar itself: it blinks between its normal draining fill and a
|
|
169
|
-
* blank track of the same width, rather than a separate indicator glyph next to it. Off-phase
|
|
170
|
-
* intentionally renders identically to the "no data" empty track (dim, all "░") so the bar reads
|
|
171
|
-
* as a single blinking element, not a bar plus a decoration.
|
|
172
|
-
*/
|
|
173
149
|
function compactionBarGlyph(progress: CompactionProgress, theme: FooterTheme, width: number, now: number): string {
|
|
174
150
|
if (!compactionBlinkOn(progress.startedAt, now)) return theme.fg("dim", "░".repeat(width));
|
|
175
|
-
|
|
176
|
-
return theme.fg("accent", progressBar(fraction, width));
|
|
177
|
-
}
|
|
178
|
-
|
|
179
|
-
/**
|
|
180
|
-
* A countdown, never a count-up: once a learned estimate exists it reports seconds remaining,
|
|
181
|
-
* ticking down toward zero in step with the draining bar. Before that (cold start, no estimate
|
|
182
|
-
* yet) there is nothing true to count down from, so this reports nothing at all rather than a
|
|
183
|
-
* fabricated elapsed count or a guessed total — the blinking, non-draining bar is the only signal.
|
|
184
|
-
*/
|
|
185
|
-
function compactionStatusText(progress: CompactionProgress, now: number): string | undefined {
|
|
186
|
-
if (progress.confidence === "learned" && typeof progress.estimatedMs === "number" && progress.estimatedMs > 0) {
|
|
187
|
-
const remainingSeconds = Math.max(0, Math.ceil((progress.estimatedMs - (now - progress.startedAt)) / MILLISECONDS_PER_SECOND));
|
|
188
|
-
return `compact ~${remainingSeconds}s left`;
|
|
189
|
-
}
|
|
190
|
-
return undefined;
|
|
151
|
+
return theme.fg("accent", progressBar(compactionFraction(progress, now), width));
|
|
191
152
|
}
|
|
192
153
|
|
|
193
154
|
function contextSegment(
|
|
@@ -199,11 +160,7 @@ function contextSegment(
|
|
|
199
160
|
compaction?: CompactionProgress,
|
|
200
161
|
): string {
|
|
201
162
|
const w = barWidth(width);
|
|
202
|
-
if (compaction) {
|
|
203
|
-
const bar = compactionBarGlyph(compaction, theme, w, now);
|
|
204
|
-
const statusText = compactionStatusText(compaction, now);
|
|
205
|
-
return statusText === undefined ? `ctx ${bar}` : `ctx ${bar} ${statusText}`;
|
|
206
|
-
}
|
|
163
|
+
if (compaction) return `ctx ${compactionBarGlyph(compaction, theme, w, now)}`;
|
|
207
164
|
const usage = context.getContextUsage();
|
|
208
165
|
const window = usage?.contextWindow ?? context.model?.contextWindow ?? 0;
|
|
209
166
|
const fraction = usage?.percent === null || usage?.percent === undefined ? null : usage.percent / 100;
|