@danypops/jittor 0.1.1 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -3
- package/docs/USAGE_PRIOR_ART.md +2 -2
- package/extension/src/footer.ts +81 -12
- package/extension/src/index.ts +143 -16
- package/extension/src/settings.ts +18 -3
- package/extension/src/tui.ts +49 -11
- package/package.json +1 -1
- package/src/constants.ts +13 -1
- package/src/domain/codex-recovery.ts +195 -0
- package/src/providers/openrouter-contracts.ts +7 -1
- package/src/service.ts +2 -1
- package/src/version.ts +16 -0
package/README.md
CHANGED
|
@@ -35,11 +35,23 @@ Operations currently include `metrics.record`, `metrics.query`, `metrics.prune`,
|
|
|
35
35
|
|
|
36
36
|
Provider adapters currently include official OpenRouter key/usage/model telemetry and an explicitly experimental Codex subscription adapter. The Codex adapter follows the pinned open-source CLI `/wham/usage` payload and `x-codex-*` response-header contracts, accepts additional metered limits, and fails closed on malformed windows or impossible percentages. File credentials must be explicitly configured and private (`0600`); Jittor reads only the access token and account ID, never refreshes credentials, and never logs or persists OAuth secrets.
|
|
37
37
|
|
|
38
|
-
The native Pi extension preflights input and every provider turn, applies model/thinking decisions, records response headers and finalized usage through the daemon, and blocks requests when required telemetry is unsafe. It follows Pi's current authenticated model/provider and synchronizes Pi's available models before every decision, so unavailable catalog routes are never selected. Its responsive
|
|
38
|
+
The native Pi extension preflights input and every provider turn, applies model/thinking decisions, records response headers and finalized usage through the daemon, and blocks requests when required telemetry is unsafe. It follows Pi's current authenticated model/provider and synchronizes Pi's available models before every decision, so unavailable catalog routes are never selected. Its responsive integrated footer groups repository and model identity with cumulative usage, a color-coded context-window bar, and current-provider budget telemetry. Codex shows the active model's bounded quota as a draining remaining-budget bar with reset and freshness information. OpenRouter uses the same drain semantics when its official key telemetry exposes a configured limit and remaining balance; keys without a limit remain honest text-only spend and never receive a fabricated denominator. During Pi compaction, the context bar drains as an animation and reports elapsed time. Unknown and stale telemetry are marked explicitly. Run `/jittor` for detailed burn pressure, freshness, route state, and confirmed emergency-halt/override controls.
|
|
39
39
|
|
|
40
|
-
Blocking always has a daemon-independent escape hatch. `/jittor off`
|
|
40
|
+
Blocking always has a daemon-independent escape hatch. `/jittor off` immediately enters persisted monitor-only mode and never blocks provider requests. The informational footer is independently controlled with `/jittor footer on` and `/jittor footer off`, so showing status never enables enforcement. `/jittor on` only enables enforcement after telemetry polling and available-route synchronization succeed. Every fail-closed error includes these recovery commands plus the daemon restart command.
|
|
41
41
|
|
|
42
|
-
|
|
42
|
+
### Opt-in Codex settled-turn recovery
|
|
43
|
+
|
|
44
|
+
Transient Codex recovery is securely off by default. Opt in explicitly through `$XDG_CONFIG_HOME/jittor/extension.json` (or `~/.config/jittor/extension.json`):
|
|
45
|
+
|
|
46
|
+
```json
|
|
47
|
+
{
|
|
48
|
+
"codexRecoveryEnabled": true
|
|
49
|
+
}
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Jittor observes finalized Codex assistant errors through Pi's public message lifecycle, classifies only bounded error metadata, and waits for `agent_settled` before acting. That boundary guarantees Pi's built-in retry, compaction retry, and queued follow-up work has finished. A transient concurrency, rate-limit, overload, or transport failure then schedules one hidden follow-up with Retry-After-aware capped jitter. Recovery is limited to three attempts per ten-minute window, never overlaps pending Pi messages, resets after success, and is canceled by human input or session shutdown. Quota, authentication, invalid-request, unknown, and aborted failures remain terminal. Raw provider payloads are never retained or injected.
|
|
53
|
+
|
|
54
|
+
Run `/jittor usage` for a colored Unicode token histogram with X/Y axes, provider/model series, input/output/cache totals, refresh, and `24h`, `7d`, `30d`, or `90d` ranges. Left/Right changes range and `r` refreshes. Usage is persisted by the daemon from finalized Pi assistant messages.
|
|
43
55
|
|
|
44
56
|
See [`docs/CALIBRATION.md`](docs/CALIBRATION.md) for thresholds and rollback, and [`docs/USAGE_PRIOR_ART.md`](docs/USAGE_PRIOR_ART.md) for the chart design research.
|
|
45
57
|
|
package/docs/USAGE_PRIOR_ART.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Token-usage TUI prior art
|
|
2
2
|
|
|
3
|
-
Research performed before implementing Jittor's `/usage` frontend.
|
|
3
|
+
Research performed before implementing Jittor's `/jittor usage` frontend.
|
|
4
4
|
|
|
5
5
|
## Local agent implementations
|
|
6
6
|
|
|
@@ -60,5 +60,5 @@ Jittor combines the best applicable patterns:
|
|
|
60
60
|
3. Provider/model-preserving series and input/output/cache totals.
|
|
61
61
|
4. Vertically scaled, colored, stacked Unicode bars with fractional top blocks.
|
|
62
62
|
5. Width-safe X/Y axes and provider/model legend.
|
|
63
|
-
6. Native `/usage` panel with Left/Right range switching and refresh.
|
|
63
|
+
6. Native `/jittor usage` panel with Left/Right range switching and refresh.
|
|
64
64
|
7. Data access only through authenticated daemon `metrics.query`; the extension never opens SQLite or reads provider credentials.
|
package/extension/src/footer.ts
CHANGED
|
@@ -6,8 +6,13 @@ import {
|
|
|
6
6
|
FOOTER_BAR_MIN_WIDTH,
|
|
7
7
|
FOOTER_CONTEXT_ACCENT_FRACTION,
|
|
8
8
|
FOOTER_CONTEXT_ERROR_FRACTION,
|
|
9
|
+
FOOTER_COMPACTION_DRAIN_STEP_MS,
|
|
9
10
|
FOOTER_CONTEXT_WARNING_FRACTION,
|
|
10
11
|
FOOTER_WIDE_TERMINAL_WIDTH,
|
|
12
|
+
MILLISECONDS_PER_DAY,
|
|
13
|
+
MILLISECONDS_PER_HOUR,
|
|
14
|
+
MILLISECONDS_PER_MINUTE,
|
|
15
|
+
MILLISECONDS_PER_SECOND,
|
|
11
16
|
TELEMETRY_STALE_AFTER_MS,
|
|
12
17
|
} from "../../src/constants.ts";
|
|
13
18
|
|
|
@@ -42,12 +47,24 @@ interface FooterContext {
|
|
|
42
47
|
};
|
|
43
48
|
}
|
|
44
49
|
|
|
45
|
-
/** A
|
|
46
|
-
export
|
|
50
|
+
/** A bounded quota is explicitly remaining; unbounded values never receive a fabricated bar. */
|
|
51
|
+
export type ProviderBudget = {
|
|
52
|
+
kind: "bounded";
|
|
53
|
+
label: string;
|
|
54
|
+
remainingFraction: number;
|
|
55
|
+
observedAt?: number;
|
|
56
|
+
resetsAt?: number;
|
|
57
|
+
resetText?: string;
|
|
58
|
+
} | {
|
|
59
|
+
kind: "unbounded";
|
|
47
60
|
label: string;
|
|
48
|
-
fraction: number | null;
|
|
49
61
|
valueText: string;
|
|
50
62
|
observedAt?: number;
|
|
63
|
+
};
|
|
64
|
+
|
|
65
|
+
export interface CompactionProgress {
|
|
66
|
+
startedAt: number;
|
|
67
|
+
initialFraction: number;
|
|
51
68
|
}
|
|
52
69
|
|
|
53
70
|
interface UsageTotals {
|
|
@@ -116,24 +133,72 @@ function fillColor(fraction: number | null): FooterColor {
|
|
|
116
133
|
return "dim";
|
|
117
134
|
}
|
|
118
135
|
|
|
119
|
-
function
|
|
136
|
+
function compactionFraction(progress: CompactionProgress, width: number, now: number): number {
|
|
137
|
+
const initialFilled = Math.round(Math.min(1, Math.max(0, progress.initialFraction)) * width);
|
|
138
|
+
const drained = Math.floor(Math.max(0, now - progress.startedAt) / FOOTER_COMPACTION_DRAIN_STEP_MS);
|
|
139
|
+
return Math.max(0, initialFilled - drained) / width;
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
function contextSegment(
|
|
143
|
+
context: FooterContext,
|
|
144
|
+
theme: FooterTheme,
|
|
145
|
+
width: number,
|
|
146
|
+
compact: boolean,
|
|
147
|
+
now: number,
|
|
148
|
+
compaction?: CompactionProgress,
|
|
149
|
+
): string {
|
|
150
|
+
const w = barWidth(width);
|
|
151
|
+
if (compaction) {
|
|
152
|
+
const fraction = compactionFraction(compaction, w, now);
|
|
153
|
+
const elapsedSeconds = Math.floor(Math.max(0, now - compaction.startedAt) / MILLISECONDS_PER_SECOND);
|
|
154
|
+
return `ctx ${theme.fg("accent", progressBar(fraction, w))} compact ${elapsedSeconds}s`;
|
|
155
|
+
}
|
|
120
156
|
const usage = context.getContextUsage();
|
|
121
157
|
const window = usage?.contextWindow ?? context.model?.contextWindow ?? 0;
|
|
122
158
|
const fraction = usage?.percent === null || usage?.percent === undefined ? null : usage.percent / 100;
|
|
123
|
-
const bar = theme.fg(fillColor(fraction), progressBar(fraction,
|
|
159
|
+
const bar = theme.fg(fillColor(fraction), progressBar(fraction, w));
|
|
124
160
|
if (usage?.tokens === null || usage?.tokens === undefined) return `ctx ${bar} ?/${formatTokens(window)}`;
|
|
125
161
|
const value = compact ? `${Math.round((fraction ?? 0) * 100)}%` : `${formatTokens(usage.tokens)}/${formatTokens(window)}`;
|
|
126
162
|
return `ctx ${bar} ${value}`;
|
|
127
163
|
}
|
|
128
164
|
|
|
165
|
+
function minimalContextSegment(
|
|
166
|
+
context: FooterContext,
|
|
167
|
+
theme: FooterTheme,
|
|
168
|
+
width: number,
|
|
169
|
+
now: number,
|
|
170
|
+
compaction?: CompactionProgress,
|
|
171
|
+
): string {
|
|
172
|
+
const w = barWidth(width);
|
|
173
|
+
if (compaction) {
|
|
174
|
+
const fraction = compactionFraction(compaction, w, now);
|
|
175
|
+
return `ctx ${theme.fg("accent", progressBar(fraction, w))}`;
|
|
176
|
+
}
|
|
177
|
+
const percent = context.getContextUsage()?.percent;
|
|
178
|
+
const fraction = percent === null || percent === undefined ? null : percent / 100;
|
|
179
|
+
return `ctx ${theme.fg(fillColor(fraction), progressBar(fraction, w))}`;
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
function resetLabel(resetsAt: number | undefined, now: number): string | undefined {
|
|
183
|
+
if (resetsAt === undefined) return undefined;
|
|
184
|
+
const remaining = resetsAt - now;
|
|
185
|
+
if (remaining <= 0) return "reset due";
|
|
186
|
+
if (remaining >= MILLISECONDS_PER_DAY) return `resets in ${Math.floor(remaining / MILLISECONDS_PER_DAY)}d`;
|
|
187
|
+
if (remaining >= MILLISECONDS_PER_HOUR) return `resets in ${Math.floor(remaining / MILLISECONDS_PER_HOUR)}h`;
|
|
188
|
+
return `resets in ${Math.max(1, Math.ceil(remaining / MILLISECONDS_PER_MINUTE))}m`;
|
|
189
|
+
}
|
|
190
|
+
|
|
129
191
|
function budgetSegment(budget: ProviderBudget | null, theme: FooterTheme, width: number, compact: boolean, now: number): string {
|
|
130
192
|
const w = barWidth(width);
|
|
131
193
|
if (!budget) return `budget ${theme.fg("dim", progressBar(null, w))} ?`;
|
|
132
|
-
if (budget.fraction === null) return `${budget.label} ${budget.valueText}`;
|
|
133
|
-
const bar = theme.fg(fillColor(budget.fraction), progressBar(budget.fraction, w));
|
|
134
|
-
const value = compact ? `${Math.round(budget.fraction * 100)}%` : budget.valueText;
|
|
135
194
|
const stale = budget.observedAt !== undefined && now - budget.observedAt > TELEMETRY_STALE_AFTER_MS;
|
|
136
|
-
|
|
195
|
+
const staleText = stale ? ` ${theme.fg("warning", "stale")}` : "";
|
|
196
|
+
if (budget.kind === "unbounded") return `${budget.label} ${budget.valueText}${staleText}`;
|
|
197
|
+
const remaining = Math.min(1, Math.max(0, budget.remainingFraction));
|
|
198
|
+
const bar = theme.fg(fillColor(1 - remaining), progressBar(remaining, w));
|
|
199
|
+
const value = `${(compact ? Math.round(remaining * 100) : (remaining * 100).toFixed(1))}% left`;
|
|
200
|
+
const reset = compact ? undefined : resetLabel(budget.resetsAt, now) ?? budget.resetText;
|
|
201
|
+
return `${budget.label} ${bar} ${value}${reset ? ` · ${reset}` : ""}${staleText}`;
|
|
137
202
|
}
|
|
138
203
|
|
|
139
204
|
function usageSegment(context: FooterContext): string {
|
|
@@ -187,14 +252,16 @@ export function renderFooterLines(
|
|
|
187
252
|
thinkingLevel: string,
|
|
188
253
|
width: number,
|
|
189
254
|
now = Date.now(),
|
|
255
|
+
compaction?: CompactionProgress,
|
|
190
256
|
): string[] {
|
|
191
257
|
const safeWidth = Math.max(1, width);
|
|
192
258
|
const repository = repositorySegment(context, footerData, theme);
|
|
193
259
|
const model = modelSegments(context, footerData, theme, thinkingLevel);
|
|
194
260
|
const usage = usageSegment(context);
|
|
195
261
|
const compactUsage = compactUsageSegment(context);
|
|
196
|
-
const fullContext = contextSegment(context, theme, safeWidth, false);
|
|
197
|
-
const compactContext = contextSegment(context, theme, safeWidth, true);
|
|
262
|
+
const fullContext = contextSegment(context, theme, safeWidth, false, now, compaction);
|
|
263
|
+
const compactContext = contextSegment(context, theme, safeWidth, true, now, compaction);
|
|
264
|
+
const minimalContext = minimalContextSegment(context, theme, safeWidth, now, compaction);
|
|
198
265
|
const fullBudget = budgetSegment(providerBudget, theme, safeWidth, false, now);
|
|
199
266
|
const compactBudget = budgetSegment(providerBudget, theme, safeWidth, true, now);
|
|
200
267
|
const statuses = [...footerData.getExtensionStatuses().entries()]
|
|
@@ -211,6 +278,7 @@ export function renderFooterLines(
|
|
|
211
278
|
joinSegments([model.full, compactUsage, compactContext, compactBudget]),
|
|
212
279
|
joinSegments([model.compact, compactUsage, compactContext, compactBudget]),
|
|
213
280
|
joinSegments([model.compact, compactContext, compactBudget]),
|
|
281
|
+
joinSegments([model.compact, minimalContext, compactBudget]),
|
|
214
282
|
];
|
|
215
283
|
const line = candidates.find((candidate) => visibleWidth(candidate) <= safeWidth) ?? candidates.at(-1) ?? "";
|
|
216
284
|
return [truncateToWidth(line, safeWidth, "")];
|
|
@@ -218,6 +286,7 @@ export function renderFooterLines(
|
|
|
218
286
|
|
|
219
287
|
export interface IntegratedFooterState {
|
|
220
288
|
providerBudget: ProviderBudget | null;
|
|
289
|
+
compaction?: CompactionProgress;
|
|
221
290
|
requestRender?: () => void;
|
|
222
291
|
}
|
|
223
292
|
|
|
@@ -229,7 +298,7 @@ export function installIntegratedFooter(ctx: ExtensionContext, state: Integrated
|
|
|
229
298
|
return {
|
|
230
299
|
invalidate() {},
|
|
231
300
|
render(width: number): string[] {
|
|
232
|
-
return renderFooterLines(ctx as unknown as FooterContext, footerData, theme, state.providerBudget, getThinkingLevel(), width);
|
|
301
|
+
return renderFooterLines(ctx as unknown as FooterContext, footerData, theme, state.providerBudget, getThinkingLevel(), width, Date.now(), state.compaction);
|
|
233
302
|
},
|
|
234
303
|
dispose() {
|
|
235
304
|
unsubscribe?.();
|
package/extension/src/index.ts
CHANGED
|
@@ -1,12 +1,21 @@
|
|
|
1
1
|
import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
|
|
2
|
-
import {
|
|
2
|
+
import {
|
|
3
|
+
CODEX_RECOVERY_ATTEMPT_WINDOW_MS,
|
|
4
|
+
CODEX_RECOVERY_BASE_DELAY_MS,
|
|
5
|
+
CODEX_RECOVERY_JITTER_RATIO,
|
|
6
|
+
CODEX_RECOVERY_MAX_ATTEMPTS,
|
|
7
|
+
CODEX_RECOVERY_MAX_DELAY_MS,
|
|
8
|
+
FOOTER_COMPACTION_RENDER_INTERVAL_MS,
|
|
9
|
+
MAX_DYNAMIC_ROUTES,
|
|
10
|
+
} from "../../src/constants.ts";
|
|
11
|
+
import { CodexRecoveryPolicy, classifyCodexFailure, type CodexFailureMetadata } from "../../src/domain/codex-recovery.ts";
|
|
3
12
|
import type { MetricObservation, StoredMetricObservation } from "../../src/domain/metric.ts";
|
|
4
13
|
import type { PolicyDecision, Route } from "../../src/policy.ts";
|
|
5
14
|
import type { RouterStatus } from "../../src/ports/router-controller.ts";
|
|
6
15
|
import { parseCodexRateLimitHeaders } from "../../src/providers/codex.ts";
|
|
7
16
|
import { installIntegratedFooter, type IntegratedFooterState } from "./footer.ts";
|
|
8
17
|
import { callJittor } from "./service-client.ts";
|
|
9
|
-
import { persistentEnforcementControl, type EnforcementControl } from "./settings.ts";
|
|
18
|
+
import { persistentEnforcementControl, type CodexRecoveryControl, type EnforcementControl } from "./settings.ts";
|
|
10
19
|
import { buildFooterBudget, formatFooterStatus, showJittorPanel } from "./tui.ts";
|
|
11
20
|
import { showUsagePanel } from "./usage.ts";
|
|
12
21
|
|
|
@@ -23,6 +32,32 @@ const daemonClient: JittorExtensionClient = {
|
|
|
23
32
|
call: (operation, input) => callJittor(operation as Parameters<typeof callJittor>[0], input as never),
|
|
24
33
|
};
|
|
25
34
|
|
|
35
|
+
export interface CodexRecoveryRuntime {
|
|
36
|
+
now(): number;
|
|
37
|
+
random(): number;
|
|
38
|
+
setTimeout(callback: () => void | Promise<void>, delayMs: number): unknown;
|
|
39
|
+
clearTimeout(handle: unknown): void;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
const SYSTEM_RECOVERY_RUNTIME: CodexRecoveryRuntime = {
|
|
43
|
+
now: Date.now,
|
|
44
|
+
random: Math.random,
|
|
45
|
+
setTimeout(callback, delayMs) { return setTimeout(() => { void callback(); }, delayMs); },
|
|
46
|
+
clearTimeout(handle) { clearTimeout(handle as ReturnType<typeof setTimeout>); },
|
|
47
|
+
};
|
|
48
|
+
|
|
49
|
+
function recoveryControl(enforcement: EnforcementControl): CodexRecoveryControl {
|
|
50
|
+
const candidate = enforcement as EnforcementControl & Partial<CodexRecoveryControl>;
|
|
51
|
+
return typeof candidate.isCodexRecoveryEnabled === "function"
|
|
52
|
+
? { isCodexRecoveryEnabled: () => candidate.isCodexRecoveryEnabled!() }
|
|
53
|
+
: { isCodexRecoveryEnabled: () => false };
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
function header(headers: Record<string, string>, name: string): string | undefined {
|
|
57
|
+
const expected = name.toLowerCase();
|
|
58
|
+
return Object.entries(headers).find(([key]) => key.toLowerCase() === expected)?.[1];
|
|
59
|
+
}
|
|
60
|
+
|
|
26
61
|
async function recordMetrics(client: JittorExtensionClient, metrics: MetricObservation[]): Promise<void> {
|
|
27
62
|
for (const metric of metrics) await client.call("metrics.record", metric);
|
|
28
63
|
}
|
|
@@ -32,7 +67,7 @@ async function refreshFooter(client: JittorExtensionClient, state: IntegratedFoo
|
|
|
32
67
|
const provider = status.currentRoute?.provider;
|
|
33
68
|
const query = provider === "openai-codex"
|
|
34
69
|
? { source: "codex-subscription", metric: "used-fraction", limit: 100, order: "desc" }
|
|
35
|
-
: provider === "openrouter" ? { source: "openrouter",
|
|
70
|
+
: provider === "openrouter" ? { source: "openrouter", limit: 20, order: "desc" } : null;
|
|
36
71
|
const metrics = query ? await client.call("metrics.query", query) as StoredMetricObservation[] : [];
|
|
37
72
|
state.providerBudget = buildFooterBudget(status, metrics);
|
|
38
73
|
state.requestRender?.();
|
|
@@ -169,8 +204,65 @@ export function registerJittorExtension(
|
|
|
169
204
|
pi: ExtensionAPI,
|
|
170
205
|
client: JittorExtensionClient = daemonClient,
|
|
171
206
|
enforcement: EnforcementControl = persistentEnforcementControl(),
|
|
207
|
+
codexRecovery: CodexRecoveryControl = recoveryControl(enforcement),
|
|
208
|
+
recoveryRuntime: CodexRecoveryRuntime = SYSTEM_RECOVERY_RUNTIME,
|
|
172
209
|
): void {
|
|
173
210
|
const footerState: IntegratedFooterState = { providerBudget: null };
|
|
211
|
+
const recoveryPolicy = new CodexRecoveryPolicy({
|
|
212
|
+
baseDelayMs: CODEX_RECOVERY_BASE_DELAY_MS,
|
|
213
|
+
maxDelayMs: CODEX_RECOVERY_MAX_DELAY_MS,
|
|
214
|
+
maxAttempts: CODEX_RECOVERY_MAX_ATTEMPTS,
|
|
215
|
+
attemptWindowMs: CODEX_RECOVERY_ATTEMPT_WINDOW_MS,
|
|
216
|
+
jitterRatio: CODEX_RECOVERY_JITTER_RATIO,
|
|
217
|
+
}, recoveryRuntime.random);
|
|
218
|
+
let recoveryTimer: unknown;
|
|
219
|
+
let lastCodexResponse: CodexFailureMetadata = {};
|
|
220
|
+
const cancelRecovery = (resetPolicy: boolean): void => {
|
|
221
|
+
if (recoveryTimer !== undefined) recoveryRuntime.clearTimeout(recoveryTimer);
|
|
222
|
+
recoveryTimer = undefined;
|
|
223
|
+
if (resetPolicy) recoveryPolicy.cancel();
|
|
224
|
+
};
|
|
225
|
+
const scheduleCodexRecovery = (ctx: ExtensionContext): void => {
|
|
226
|
+
if (!codexRecovery.isCodexRecoveryEnabled() || recoveryTimer !== undefined || !ctx.isIdle() || ctx.hasPendingMessages()) return;
|
|
227
|
+
const plan = recoveryPolicy.plan(recoveryRuntime.now());
|
|
228
|
+
if (plan.action === "exhausted") {
|
|
229
|
+
recoveryPolicy.abandonFailure();
|
|
230
|
+
if (ctx.hasUI) ctx.ui.notify(`Jittor Codex recovery stopped: ${plan.reason}.`, "warning");
|
|
231
|
+
return;
|
|
232
|
+
}
|
|
233
|
+
if (plan.action !== "schedule") return;
|
|
234
|
+
recoveryTimer = recoveryRuntime.setTimeout(async () => {
|
|
235
|
+
recoveryTimer = undefined;
|
|
236
|
+
if (!ctx.isIdle() || ctx.hasPendingMessages()) return;
|
|
237
|
+
const attempt = recoveryPolicy.recordAttempt(recoveryRuntime.now());
|
|
238
|
+
if (!attempt) return;
|
|
239
|
+
pi.sendMessage({
|
|
240
|
+
customType: "jittor-codex-recovery",
|
|
241
|
+
content: `Retry the previous Codex request after a transient ${attempt.failureKind} failure. Automatic recovery attempt ${attempt.attempt} of ${CODEX_RECOVERY_MAX_ATTEMPTS}.`,
|
|
242
|
+
display: false,
|
|
243
|
+
details: { attempt: attempt.attempt, failureKind: attempt.failureKind },
|
|
244
|
+
}, { triggerTurn: true, deliverAs: "followUp" });
|
|
245
|
+
}, plan.delayMs);
|
|
246
|
+
};
|
|
247
|
+
let compactionTimer: ReturnType<typeof setInterval> | undefined;
|
|
248
|
+
const finishCompaction = (): void => {
|
|
249
|
+
if (compactionTimer) clearInterval(compactionTimer);
|
|
250
|
+
compactionTimer = undefined;
|
|
251
|
+
footerState.compaction = undefined;
|
|
252
|
+
footerState.requestRender?.();
|
|
253
|
+
};
|
|
254
|
+
const beginCompaction = (ctx: ExtensionContext, signal: AbortSignal): void => {
|
|
255
|
+
finishCompaction();
|
|
256
|
+
const usage = ctx.getContextUsage();
|
|
257
|
+
footerState.compaction = {
|
|
258
|
+
startedAt: Date.now(),
|
|
259
|
+
initialFraction: usage?.percent === null || usage?.percent === undefined ? 1 : usage.percent / 100,
|
|
260
|
+
};
|
|
261
|
+
compactionTimer = setInterval(() => footerState.requestRender?.(), FOOTER_COMPACTION_RENDER_INTERVAL_MS);
|
|
262
|
+
signal.addEventListener("abort", finishCompaction, { once: true });
|
|
263
|
+
if (signal.aborted) finishCompaction();
|
|
264
|
+
else footerState.requestRender?.();
|
|
265
|
+
};
|
|
174
266
|
const showFooter = (ctx: ExtensionContext): void => {
|
|
175
267
|
if (enforcement.isFooterEnabled()) installIntegratedFooter(ctx, footerState, () => pi.getThinkingLevel());
|
|
176
268
|
else ctx.ui.setFooter(undefined);
|
|
@@ -201,7 +293,7 @@ export function registerJittorExtension(
|
|
|
201
293
|
};
|
|
202
294
|
|
|
203
295
|
pi.registerCommand("jittor", {
|
|
204
|
-
description: "Inspect
|
|
296
|
+
description: "Inspect or control Jittor routing, budgets, and usage",
|
|
205
297
|
handler: async (args, ctx) => {
|
|
206
298
|
const action = args.trim().toLowerCase();
|
|
207
299
|
if (action === "off" || action === "disable") { disable(ctx); return; }
|
|
@@ -219,6 +311,10 @@ export function registerJittorExtension(
|
|
|
219
311
|
ctx.ui.notify("Jittor informational footer enabled; routing enforcement is unchanged.", "info");
|
|
220
312
|
return;
|
|
221
313
|
}
|
|
314
|
+
if (action === "usage") {
|
|
315
|
+
await showUsagePanel(ctx, client);
|
|
316
|
+
return;
|
|
317
|
+
}
|
|
222
318
|
if (!enforcement.isEnabled()) {
|
|
223
319
|
ctx.ui.notify("Jittor is monitor-only. Run /jittor on to re-enable blocking.", "info");
|
|
224
320
|
return;
|
|
@@ -226,20 +322,11 @@ export function registerJittorExtension(
|
|
|
226
322
|
await showJittorPanel(ctx, client);
|
|
227
323
|
},
|
|
228
324
|
});
|
|
229
|
-
pi.registerCommand("jittor-off", {
|
|
230
|
-
description: "Emergency local bypass: disable Jittor blocking without daemon access",
|
|
231
|
-
handler: async (_args, ctx) => { disable(ctx); },
|
|
232
|
-
});
|
|
233
|
-
pi.registerCommand("jittor-on", {
|
|
234
|
-
description: "Enable Jittor only after telemetry and routes pass readiness",
|
|
235
|
-
handler: async (_args, ctx) => { await enable(ctx); },
|
|
236
|
-
});
|
|
237
|
-
pi.registerCommand("usage", {
|
|
238
|
-
description: "Show Jittor token usage over time",
|
|
239
|
-
handler: async (_args, ctx) => { await showUsagePanel(ctx, client); },
|
|
240
|
-
});
|
|
241
325
|
|
|
242
326
|
pi.on("session_start", async (_event, ctx) => {
|
|
327
|
+
finishCompaction();
|
|
328
|
+
cancelRecovery(true);
|
|
329
|
+
lastCodexResponse = {};
|
|
243
330
|
ctx.ui.setStatus("jittor", undefined);
|
|
244
331
|
showFooter(ctx);
|
|
245
332
|
try {
|
|
@@ -253,7 +340,30 @@ export function registerJittorExtension(
|
|
|
253
340
|
}
|
|
254
341
|
});
|
|
255
342
|
|
|
343
|
+
pi.on("session_before_compact", (event, ctx) => {
|
|
344
|
+
beginCompaction(ctx, event.signal);
|
|
345
|
+
});
|
|
346
|
+
|
|
347
|
+
pi.on("session_compact", () => {
|
|
348
|
+
finishCompaction();
|
|
349
|
+
});
|
|
350
|
+
|
|
351
|
+
pi.on("agent_settled", async (_event, ctx) => {
|
|
352
|
+
if (footerState.compaction) finishCompaction();
|
|
353
|
+
scheduleCodexRecovery(ctx);
|
|
354
|
+
if (!enforcement.isFooterEnabled()) return;
|
|
355
|
+
try {
|
|
356
|
+
await syncCurrentRoute(pi, client, ctx);
|
|
357
|
+
await syncAvailableRoutes(pi, client, ctx);
|
|
358
|
+
await refreshFooter(client, footerState);
|
|
359
|
+
} catch {
|
|
360
|
+
footerState.providerBudget = null;
|
|
361
|
+
footerState.requestRender?.();
|
|
362
|
+
}
|
|
363
|
+
});
|
|
364
|
+
|
|
256
365
|
pi.on("input", async (event, ctx) => {
|
|
366
|
+
if (event.source !== "extension") cancelRecovery(true);
|
|
257
367
|
if (event.source === "extension" || !enforcement.isEnabled()) return { action: "continue" as const };
|
|
258
368
|
try {
|
|
259
369
|
const next = await client.call("router.decide", {}) as PolicyDecision;
|
|
@@ -278,6 +388,7 @@ export function registerJittorExtension(
|
|
|
278
388
|
});
|
|
279
389
|
|
|
280
390
|
pi.on("turn_start", async (_event, ctx) => {
|
|
391
|
+
lastCodexResponse = {};
|
|
281
392
|
if (!enforcement.isEnabled()) return;
|
|
282
393
|
try {
|
|
283
394
|
await syncCurrentRoute(pi, client, ctx);
|
|
@@ -290,6 +401,9 @@ export function registerJittorExtension(
|
|
|
290
401
|
});
|
|
291
402
|
|
|
292
403
|
pi.on("after_provider_response", async (event, ctx) => {
|
|
404
|
+
if (ctx.model?.provider === "openai-codex") {
|
|
405
|
+
lastCodexResponse = { status: event.status, ...(header(event.headers, "retry-after") ? { retryAfter: header(event.headers, "retry-after") } : {}) };
|
|
406
|
+
}
|
|
293
407
|
if (!Object.keys(event.headers).some((name) => name.toLowerCase().startsWith("x-codex-"))) return;
|
|
294
408
|
try {
|
|
295
409
|
const updates = parseCodexRateLimitHeaders(new Headers(event.headers), Date.now());
|
|
@@ -301,12 +415,25 @@ export function registerJittorExtension(
|
|
|
301
415
|
});
|
|
302
416
|
|
|
303
417
|
pi.on("message_end", async (event, _ctx) => {
|
|
418
|
+
if (event.message.role === "assistant" && event.message.provider === "openai-codex") {
|
|
419
|
+
if (event.message.stopReason === "error") {
|
|
420
|
+
const failure = classifyCodexFailure(event.message.errorMessage, lastCodexResponse);
|
|
421
|
+
if (codexRecovery.isCodexRecoveryEnabled() && failure.transient) recoveryPolicy.observeFailure(failure, recoveryRuntime.now());
|
|
422
|
+
else cancelRecovery(true);
|
|
423
|
+
} else if (event.message.stopReason !== "aborted") {
|
|
424
|
+
cancelRecovery(true);
|
|
425
|
+
}
|
|
426
|
+
lastCodexResponse = {};
|
|
427
|
+
}
|
|
304
428
|
const metrics = assistantUsageMetrics(event.message, Date.now());
|
|
305
429
|
if (metrics.length > 0) await recordMetrics(client, metrics).catch(() => undefined);
|
|
306
430
|
if (enforcement.isFooterEnabled()) await refreshFooter(client, footerState).catch(() => undefined);
|
|
307
431
|
});
|
|
308
432
|
|
|
309
433
|
pi.on("session_shutdown", async (_event, ctx) => {
|
|
434
|
+
finishCompaction();
|
|
435
|
+
cancelRecovery(true);
|
|
436
|
+
lastCodexResponse = {};
|
|
310
437
|
ctx.ui.setStatus("jittor", undefined);
|
|
311
438
|
ctx.ui.setFooter(undefined);
|
|
312
439
|
});
|
|
@@ -9,9 +9,18 @@ export interface EnforcementControl {
|
|
|
9
9
|
setFooterEnabled(enabled: boolean): void;
|
|
10
10
|
}
|
|
11
11
|
|
|
12
|
+
export interface CodexRecoveryControl {
|
|
13
|
+
isCodexRecoveryEnabled(): boolean;
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
export interface PersistentExtensionControl extends EnforcementControl, CodexRecoveryControl {
|
|
17
|
+
setCodexRecoveryEnabled(enabled: boolean): void;
|
|
18
|
+
}
|
|
19
|
+
|
|
12
20
|
interface ExtensionSettings {
|
|
13
21
|
enforcementEnabled: boolean;
|
|
14
22
|
footerEnabled: boolean;
|
|
23
|
+
codexRecoveryEnabled: boolean;
|
|
15
24
|
}
|
|
16
25
|
|
|
17
26
|
function settingsPath(env: Record<string, string | undefined> = process.env): string {
|
|
@@ -22,14 +31,15 @@ function settingsPath(env: Record<string, string | undefined> = process.env): st
|
|
|
22
31
|
function loadSettings(path: string): ExtensionSettings {
|
|
23
32
|
try {
|
|
24
33
|
const value = JSON.parse(readFileSync(path, "utf8")) as unknown;
|
|
25
|
-
if (typeof value !== "object" || value === null || Array.isArray(value)) return { enforcementEnabled: true, footerEnabled: true };
|
|
34
|
+
if (typeof value !== "object" || value === null || Array.isArray(value)) return { enforcementEnabled: true, footerEnabled: true, codexRecoveryEnabled: false };
|
|
26
35
|
const record = value as Record<string, unknown>;
|
|
27
36
|
return {
|
|
28
37
|
enforcementEnabled: record["enforcementEnabled"] !== false,
|
|
29
38
|
footerEnabled: record["footerEnabled"] !== false,
|
|
39
|
+
codexRecoveryEnabled: record["codexRecoveryEnabled"] === true,
|
|
30
40
|
};
|
|
31
41
|
} catch {
|
|
32
|
-
return { enforcementEnabled: true, footerEnabled: true };
|
|
42
|
+
return { enforcementEnabled: true, footerEnabled: true, codexRecoveryEnabled: false };
|
|
33
43
|
}
|
|
34
44
|
}
|
|
35
45
|
|
|
@@ -41,7 +51,7 @@ function persistSettings(path: string, settings: ExtensionSettings): void {
|
|
|
41
51
|
renameSync(temporary, path);
|
|
42
52
|
}
|
|
43
53
|
|
|
44
|
-
export function persistentEnforcementControl(env: Record<string, string | undefined> = process.env):
|
|
54
|
+
export function persistentEnforcementControl(env: Record<string, string | undefined> = process.env): PersistentExtensionControl {
|
|
45
55
|
const path = settingsPath(env);
|
|
46
56
|
const settings = loadSettings(path);
|
|
47
57
|
return {
|
|
@@ -55,5 +65,10 @@ export function persistentEnforcementControl(env: Record<string, string | undefi
|
|
|
55
65
|
settings.footerEnabled = value;
|
|
56
66
|
persistSettings(path, settings);
|
|
57
67
|
},
|
|
68
|
+
isCodexRecoveryEnabled: () => settings.codexRecoveryEnabled,
|
|
69
|
+
setCodexRecoveryEnabled(value: boolean): void {
|
|
70
|
+
settings.codexRecoveryEnabled = value;
|
|
71
|
+
persistSettings(path, settings);
|
|
72
|
+
},
|
|
58
73
|
};
|
|
59
74
|
}
|
package/extension/src/tui.ts
CHANGED
|
@@ -15,10 +15,35 @@ function latest(rows: StoredMetricObservation[], predicate: (row: StoredMetricOb
|
|
|
15
15
|
return rows.filter(predicate).sort((left, right) => right.observedAt - left.observedAt || right.id - left.id)[0];
|
|
16
16
|
}
|
|
17
17
|
|
|
18
|
-
function
|
|
19
|
-
return
|
|
20
|
-
|
|
21
|
-
|
|
18
|
+
function sanitizedText(value: string): string {
|
|
19
|
+
return value.replace(/[\r\n\t]/g, " ").replace(/ +/g, " ").trim();
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
function normalizedIdentity(value: unknown): string {
|
|
23
|
+
return typeof value === "string" ? value.toLowerCase().replace(/[^a-z0-9]+/g, "") : "";
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
function longestWindow(rows: StoredMetricObservation[]): StoredMetricObservation | undefined {
|
|
27
|
+
return [...rows].sort((left, right) =>
|
|
28
|
+
Number(right.attributes["windowSeconds"] ?? 0) - Number(left.attributes["windowSeconds"] ?? 0)
|
|
29
|
+
|| right.observedAt - left.observedAt
|
|
30
|
+
|| right.id - left.id,
|
|
31
|
+
)[0];
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
function codexWindowForModel(rows: StoredMetricObservation[], model: string): StoredMetricObservation | undefined {
|
|
35
|
+
const codexRows = rows.filter((row) => row.source === "codex-subscription" && row.metric === "used-fraction" && typeof row.value === "number");
|
|
36
|
+
const modelIdentity = normalizedIdentity(model);
|
|
37
|
+
const matchingAdditional = codexRows.filter((row) => {
|
|
38
|
+
const limitId = normalizedIdentity(row.attributes["limitId"]);
|
|
39
|
+
const limitName = normalizedIdentity(row.attributes["limitName"]);
|
|
40
|
+
return limitId !== "codex" && limitName.length > 0 && limitName === modelIdentity;
|
|
41
|
+
});
|
|
42
|
+
if (matchingAdditional.length > 0) return longestWindow(matchingAdditional);
|
|
43
|
+
return longestWindow(codexRows.filter((row) => {
|
|
44
|
+
const limitId = normalizedIdentity(row.attributes["limitId"]);
|
|
45
|
+
return limitId === "codex" || (limitId.length === 0 && row.scope.startsWith("codex:"));
|
|
46
|
+
}));
|
|
22
47
|
}
|
|
23
48
|
|
|
24
49
|
function compactWindowName(seconds: number): string {
|
|
@@ -36,19 +61,32 @@ function windowName(seconds: number): string {
|
|
|
36
61
|
export function buildFooterBudget(status: RouterStatus, metrics: StoredMetricObservation[]): ProviderBudget | null {
|
|
37
62
|
if (!status.ready || !status.currentRoute) return null;
|
|
38
63
|
if (status.currentRoute.provider === "openai-codex") {
|
|
39
|
-
const codex =
|
|
64
|
+
const codex = codexWindowForModel(metrics, status.currentRoute.model);
|
|
40
65
|
if (!codex || typeof codex.value !== "number") return null;
|
|
66
|
+
const resetsAtSeconds = Number(codex.attributes["resetsAt"]);
|
|
41
67
|
return {
|
|
68
|
+
kind: "bounded",
|
|
42
69
|
label: compactWindowName(Number(codex.attributes["windowSeconds"] ?? 0)),
|
|
43
|
-
|
|
44
|
-
valueText: `${(codex.value * 100).toFixed(1)}% used`,
|
|
70
|
+
remainingFraction: 1 - codex.value,
|
|
45
71
|
observedAt: codex.observedAt,
|
|
72
|
+
...(Number.isFinite(resetsAtSeconds) && resetsAtSeconds > 0 ? { resetsAt: resetsAtSeconds * 1_000 } : {}),
|
|
46
73
|
};
|
|
47
74
|
}
|
|
48
75
|
if (status.currentRoute.provider === "openrouter") {
|
|
49
76
|
const openRouter = latest(metrics, (row) => row.source === "openrouter" && row.metric === "usage" && typeof row.value === "number");
|
|
77
|
+
const remaining = latest(metrics, (row) => row.source === "openrouter" && row.metric === "remaining-fraction" && typeof row.value === "number");
|
|
78
|
+
if (remaining && typeof remaining.value === "number" && (!openRouter || remaining.observedAt >= openRouter.observedAt)) {
|
|
79
|
+
const reset = typeof remaining.attributes["reset"] === "string" ? sanitizedText(remaining.attributes["reset"]) : undefined;
|
|
80
|
+
return {
|
|
81
|
+
kind: "bounded",
|
|
82
|
+
label: "OR",
|
|
83
|
+
remainingFraction: remaining.value,
|
|
84
|
+
observedAt: remaining.observedAt,
|
|
85
|
+
...(reset ? { resetText: `${reset} reset` } : {}),
|
|
86
|
+
};
|
|
87
|
+
}
|
|
50
88
|
if (!openRouter || typeof openRouter.value !== "number") return null;
|
|
51
|
-
return {
|
|
89
|
+
return { kind: "unbounded", label: "spend", valueText: `$${openRouter.value.toFixed(3)}`, observedAt: openRouter.observedAt };
|
|
52
90
|
}
|
|
53
91
|
return null;
|
|
54
92
|
}
|
|
@@ -56,7 +94,7 @@ export function buildFooterBudget(status: RouterStatus, metrics: StoredMetricObs
|
|
|
56
94
|
export function formatFooterStatus(status: RouterStatus, metrics: StoredMetricObservation[]): string {
|
|
57
95
|
const budget = buildFooterBudget(status, metrics);
|
|
58
96
|
if (!budget) return "";
|
|
59
|
-
return budget.
|
|
97
|
+
return budget.kind === "unbounded" ? budget.valueText : `${budget.label} ${(budget.remainingFraction * 100).toFixed(1)}% left`;
|
|
60
98
|
}
|
|
61
99
|
|
|
62
100
|
function nextAction(action: PolicyAction | undefined): string {
|
|
@@ -87,10 +125,10 @@ function burnLine(rows: StoredMetricObservation[], current: StoredMetricObservat
|
|
|
87
125
|
|
|
88
126
|
export function buildStatusView(status: RouterStatus, metrics: StoredMetricObservation[], now = Date.now()): string[] {
|
|
89
127
|
const lines = [status.ready ? "Ready" : "Not ready"];
|
|
90
|
-
const codex = status.currentRoute?.provider === "openai-codex" ?
|
|
128
|
+
const codex = status.currentRoute?.provider === "openai-codex" ? codexWindowForModel(metrics, status.currentRoute.model) : undefined;
|
|
91
129
|
if (codex && typeof codex.value === "number") {
|
|
92
130
|
const seconds = Number(codex.attributes["windowSeconds"] ?? 0);
|
|
93
|
-
lines.push(`Codex ${windowName(seconds)}: ${(codex.value * 100).toFixed(1)}
|
|
131
|
+
lines.push(`Codex ${windowName(seconds)}: ${((1 - codex.value) * 100).toFixed(1)}% left`);
|
|
94
132
|
lines.push(burnLine(metrics, codex, now));
|
|
95
133
|
}
|
|
96
134
|
const openRouter = status.currentRoute?.provider === "openrouter"
|
package/package.json
CHANGED
package/src/constants.ts
CHANGED
|
@@ -1,4 +1,3 @@
|
|
|
1
|
-
export const VERSION = "0.1.1";
|
|
2
1
|
export const SQLITE_SCHEMA_VERSION = 1;
|
|
3
2
|
export const SQLITE_BUSY_TIMEOUT_MS = 5_000;
|
|
4
3
|
export const DEFAULT_QUERY_LIMIT = 1_000;
|
|
@@ -13,6 +12,12 @@ export const FOOTER_CONTEXT_ERROR_FRACTION = 0.9;
|
|
|
13
12
|
export const FOOTER_BAR_MIN_WIDTH = 4;
|
|
14
13
|
export const FOOTER_BAR_MAX_WIDTH = 8;
|
|
15
14
|
export const FOOTER_WIDE_TERMINAL_WIDTH = 100;
|
|
15
|
+
export const FOOTER_COMPACTION_RENDER_INTERVAL_MS = 1_000;
|
|
16
|
+
export const FOOTER_COMPACTION_DRAIN_STEP_MS = 3_000;
|
|
17
|
+
export const MILLISECONDS_PER_SECOND = 1_000;
|
|
18
|
+
export const MILLISECONDS_PER_MINUTE = 60 * MILLISECONDS_PER_SECOND;
|
|
19
|
+
export const MILLISECONDS_PER_HOUR = 60 * MILLISECONDS_PER_MINUTE;
|
|
20
|
+
export const MILLISECONDS_PER_DAY = 24 * MILLISECONDS_PER_HOUR;
|
|
16
21
|
export const LOOPBACK_HOST = "127.0.0.1";
|
|
17
22
|
export const JITTOR_STATE_DIRECTORY = "jittor";
|
|
18
23
|
export const JITTOR_EXTENSION_SETTINGS_FILENAME = "extension.json";
|
|
@@ -24,3 +29,10 @@ export const USAGE_CHART_HEIGHT = 8;
|
|
|
24
29
|
export const USAGE_Y_AXIS_WIDTH = 7;
|
|
25
30
|
export const USAGE_TOKEN_QUERY_LIMIT = 10_000;
|
|
26
31
|
export const MAX_DYNAMIC_ROUTES = 100;
|
|
32
|
+
export const CODEX_ERROR_MESSAGE_LIMIT = 160;
|
|
33
|
+
export const CODEX_RETRY_AFTER_MAX_MS = 5 * MILLISECONDS_PER_MINUTE;
|
|
34
|
+
export const CODEX_RECOVERY_BASE_DELAY_MS = 2 * MILLISECONDS_PER_SECOND;
|
|
35
|
+
export const CODEX_RECOVERY_MAX_DELAY_MS = CODEX_RETRY_AFTER_MAX_MS;
|
|
36
|
+
export const CODEX_RECOVERY_MAX_ATTEMPTS = 3;
|
|
37
|
+
export const CODEX_RECOVERY_ATTEMPT_WINDOW_MS = 10 * MILLISECONDS_PER_MINUTE;
|
|
38
|
+
export const CODEX_RECOVERY_JITTER_RATIO = 0.2;
|
|
@@ -0,0 +1,195 @@
|
|
|
1
|
+
import {
|
|
2
|
+
CODEX_ERROR_MESSAGE_LIMIT,
|
|
3
|
+
CODEX_RETRY_AFTER_MAX_MS,
|
|
4
|
+
MILLISECONDS_PER_SECOND,
|
|
5
|
+
} from "../constants.ts";
|
|
6
|
+
|
|
7
|
+
export type CodexFailureKind =
|
|
8
|
+
| "concurrency"
|
|
9
|
+
| "rate-limit"
|
|
10
|
+
| "overload"
|
|
11
|
+
| "transport"
|
|
12
|
+
| "quota"
|
|
13
|
+
| "authentication"
|
|
14
|
+
| "invalid-request"
|
|
15
|
+
| "unknown";
|
|
16
|
+
|
|
17
|
+
export interface CodexFailure {
|
|
18
|
+
kind: CodexFailureKind;
|
|
19
|
+
transient: boolean;
|
|
20
|
+
code?: string;
|
|
21
|
+
source?: string;
|
|
22
|
+
message?: string;
|
|
23
|
+
retryAfterMs?: number;
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
export interface CodexFailureMetadata {
|
|
27
|
+
status?: number;
|
|
28
|
+
retryAfter?: string;
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
export interface CodexRecoveryOptions {
|
|
32
|
+
baseDelayMs: number;
|
|
33
|
+
maxDelayMs: number;
|
|
34
|
+
maxAttempts: number;
|
|
35
|
+
attemptWindowMs: number;
|
|
36
|
+
jitterRatio: number;
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
export type CodexRecoveryPlan =
|
|
40
|
+
| { action: "schedule"; attempt: number; delayMs: number; failureKind: CodexFailureKind }
|
|
41
|
+
| { action: "wait"; reason: string }
|
|
42
|
+
| { action: "exhausted"; reason: string };
|
|
43
|
+
|
|
44
|
+
export interface CodexRecoveryAttempt {
|
|
45
|
+
attempt: number;
|
|
46
|
+
failureKind: CodexFailureKind;
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
export class CodexRecoveryPolicy {
|
|
50
|
+
private pendingFailure: CodexFailure | undefined;
|
|
51
|
+
private attempts = 0;
|
|
52
|
+
private windowStartedAt: number | undefined;
|
|
53
|
+
|
|
54
|
+
constructor(
|
|
55
|
+
private readonly options: CodexRecoveryOptions,
|
|
56
|
+
private readonly random: () => number = Math.random,
|
|
57
|
+
) {
|
|
58
|
+
if (!Number.isFinite(options.baseDelayMs) || options.baseDelayMs < 0) throw new Error("baseDelayMs must be non-negative");
|
|
59
|
+
if (!Number.isFinite(options.maxDelayMs) || options.maxDelayMs < options.baseDelayMs) throw new Error("maxDelayMs must be at least baseDelayMs");
|
|
60
|
+
if (!Number.isInteger(options.maxAttempts) || options.maxAttempts < 1) throw new Error("maxAttempts must be a positive integer");
|
|
61
|
+
if (!Number.isFinite(options.attemptWindowMs) || options.attemptWindowMs <= 0) throw new Error("attemptWindowMs must be positive");
|
|
62
|
+
if (!Number.isFinite(options.jitterRatio) || options.jitterRatio < 0 || options.jitterRatio > 1) throw new Error("jitterRatio must be between 0 and 1");
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
observeFailure(failure: CodexFailure, now: number): void {
|
|
66
|
+
this.normalizeWindow(now);
|
|
67
|
+
this.pendingFailure = failure.transient ? failure : undefined;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
observeSuccess(): void {
|
|
71
|
+
this.cancel();
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
cancel(): void {
|
|
75
|
+
this.pendingFailure = undefined;
|
|
76
|
+
this.attempts = 0;
|
|
77
|
+
this.windowStartedAt = undefined;
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
abandonFailure(): void {
|
|
81
|
+
this.pendingFailure = undefined;
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
plan(now: number): CodexRecoveryPlan {
|
|
85
|
+
this.normalizeWindow(now);
|
|
86
|
+
if (!this.pendingFailure) return { action: "wait", reason: "no transient Codex failure is pending" };
|
|
87
|
+
if (this.attempts >= this.options.maxAttempts) {
|
|
88
|
+
return { action: "exhausted", reason: `${this.options.maxAttempts} recovery attempts reached within ${this.options.attemptWindowMs}ms` };
|
|
89
|
+
}
|
|
90
|
+
const base = this.pendingFailure.retryAfterMs
|
|
91
|
+
?? this.options.baseDelayMs * (2 ** this.attempts);
|
|
92
|
+
const sample = this.random();
|
|
93
|
+
const unit = Number.isFinite(sample) ? Math.min(1, Math.max(0, sample)) : 0;
|
|
94
|
+
const multiplier = 1 + ((unit * 2) - 1) * this.options.jitterRatio;
|
|
95
|
+
const jittered = Math.max(0, Math.round(base * multiplier));
|
|
96
|
+
const delayMs = this.pendingFailure.retryAfterMs === undefined ? jittered : Math.max(base, jittered);
|
|
97
|
+
return {
|
|
98
|
+
action: "schedule",
|
|
99
|
+
attempt: this.attempts + 1,
|
|
100
|
+
delayMs: Math.min(this.options.maxDelayMs, delayMs),
|
|
101
|
+
failureKind: this.pendingFailure.kind,
|
|
102
|
+
};
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
recordAttempt(now: number): CodexRecoveryAttempt | undefined {
|
|
106
|
+
this.normalizeWindow(now);
|
|
107
|
+
if (!this.pendingFailure || this.attempts >= this.options.maxAttempts) return undefined;
|
|
108
|
+
if (this.windowStartedAt === undefined) this.windowStartedAt = now;
|
|
109
|
+
this.attempts += 1;
|
|
110
|
+
const attempt = { attempt: this.attempts, failureKind: this.pendingFailure.kind };
|
|
111
|
+
this.pendingFailure = undefined;
|
|
112
|
+
return attempt;
|
|
113
|
+
}
|
|
114
|
+
|
|
115
|
+
state(): { attempts: number; pending: boolean } {
|
|
116
|
+
return { attempts: this.attempts, pending: this.pendingFailure !== undefined };
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
private normalizeWindow(now: number): void {
|
|
120
|
+
if (this.windowStartedAt !== undefined && now - this.windowStartedAt >= this.options.attemptWindowMs) {
|
|
121
|
+
this.attempts = 0;
|
|
122
|
+
this.windowStartedAt = undefined;
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
function asRecord(value: unknown): Record<string, unknown> | undefined {
|
|
128
|
+
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
129
|
+
? value as Record<string, unknown>
|
|
130
|
+
: undefined;
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
function firstString(...values: unknown[]): string | undefined {
|
|
134
|
+
for (const value of values) {
|
|
135
|
+
if (typeof value === "string" && value.length > 0) return value;
|
|
136
|
+
}
|
|
137
|
+
return undefined;
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
function retryAfterMs(value: string | undefined): number | undefined {
|
|
141
|
+
if (!value) return undefined;
|
|
142
|
+
const seconds = Number(value.trim());
|
|
143
|
+
if (!Number.isFinite(seconds) || seconds < 0) return undefined;
|
|
144
|
+
return Math.min(CODEX_RETRY_AFTER_MAX_MS, Math.round(seconds * MILLISECONDS_PER_SECOND));
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
function matches(value: string, patterns: readonly string[]): boolean {
|
|
148
|
+
return patterns.some((pattern) => value.includes(pattern));
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
export function classifyCodexFailure(value: unknown, metadata: CodexFailureMetadata = {}): CodexFailure {
|
|
152
|
+
const root = asRecord(value);
|
|
153
|
+
const detail = asRecord(root?.["detail"]);
|
|
154
|
+
const nestedError = asRecord(root?.["error"]);
|
|
155
|
+
const code = firstString(detail?.["code"], detail?.["error_code"], nestedError?.["code"], root?.["code"]);
|
|
156
|
+
const source = firstString(detail?.["source"], nestedError?.["source"], root?.["source"]);
|
|
157
|
+
const rawMessage = firstString(
|
|
158
|
+
detail?.["message"],
|
|
159
|
+
typeof root?.["error"] === "string" ? root["error"] : undefined,
|
|
160
|
+
nestedError?.["message"],
|
|
161
|
+
root?.["message"],
|
|
162
|
+
typeof value === "string" ? value : undefined,
|
|
163
|
+
);
|
|
164
|
+
const message = rawMessage?.slice(0, CODEX_ERROR_MESSAGE_LIMIT);
|
|
165
|
+
const evidence = [code, source, message].filter(Boolean).join(" ").toLowerCase();
|
|
166
|
+
const base = {
|
|
167
|
+
...(code ? { code } : {}),
|
|
168
|
+
...(source ? { source } : {}),
|
|
169
|
+
...(message ? { message } : {}),
|
|
170
|
+
...(retryAfterMs(metadata.retryAfter) !== undefined ? { retryAfterMs: retryAfterMs(metadata.retryAfter) } : {}),
|
|
171
|
+
};
|
|
172
|
+
|
|
173
|
+
if (matches(evidence, ["insufficient_quota", "quota exceeded", "out of credits", "billing"])) {
|
|
174
|
+
return { kind: "quota", transient: false, ...base };
|
|
175
|
+
}
|
|
176
|
+
if (metadata.status === 401 || metadata.status === 403 || matches(evidence, ["invalid_api_key", "authentication", "unauthorized", "permission_denied"])) {
|
|
177
|
+
return { kind: "authentication", transient: false, ...base };
|
|
178
|
+
}
|
|
179
|
+
if (matches(evidence, ["invalid_prompt", "invalid_request", "context_length_exceeded"]) || metadata.status === 400 || metadata.status === 422) {
|
|
180
|
+
return { kind: "invalid-request", transient: false, ...base };
|
|
181
|
+
}
|
|
182
|
+
if (matches(evidence, ["concurrency_limit", "too many concurrent requests", "throttled"])) {
|
|
183
|
+
return { kind: "concurrency", transient: true, ...base };
|
|
184
|
+
}
|
|
185
|
+
if (matches(evidence, ["server_is_overloaded", "slow_down", "service unavailable", "overloaded"]) || (metadata.status !== undefined && metadata.status >= 500 && metadata.status <= 599)) {
|
|
186
|
+
return { kind: "overload", transient: true, ...base };
|
|
187
|
+
}
|
|
188
|
+
if (metadata.status === 429 || matches(evidence, ["rate_limit_exceeded", "rate limit", "too many requests"])) {
|
|
189
|
+
return { kind: "rate-limit", transient: true, ...base };
|
|
190
|
+
}
|
|
191
|
+
if (matches(evidence, ["timeout", "timed out", "network", "connection", "websocket", "fetch failed"])) {
|
|
192
|
+
return { kind: "transport", transient: true, ...base };
|
|
193
|
+
}
|
|
194
|
+
return { kind: "unknown", transient: false, ...base };
|
|
195
|
+
}
|
|
@@ -164,7 +164,13 @@ export function parseOpenRouterKey(rootValue: unknown, observedAt: number): Open
|
|
|
164
164
|
add("usage-monthly", snapshot.usageMonthly);
|
|
165
165
|
add("limit", limit);
|
|
166
166
|
add("limit-remaining", remaining);
|
|
167
|
-
if (limit !== null && limit > 0
|
|
167
|
+
if (limit !== null && limit > 0 && remaining !== null) {
|
|
168
|
+
const remainingFraction = remaining / limit;
|
|
169
|
+
if (remainingFraction < 0 || remainingFraction > 1) throw new Error("OpenRouter key remaining fraction is outside its configured limit");
|
|
170
|
+
const attributes = { limit, remaining, reset: snapshot.reset };
|
|
171
|
+
snapshot.metrics.push(metric(scope, "remaining-fraction", remainingFraction, "ratio", observedAt, attributes));
|
|
172
|
+
snapshot.metrics.push(metric(scope, "used-fraction", 1 - remainingFraction, "ratio", observedAt, attributes));
|
|
173
|
+
}
|
|
168
174
|
return snapshot;
|
|
169
175
|
}
|
|
170
176
|
|
package/src/service.ts
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
import { SERVICE_MAX_BODY_BYTES
|
|
1
|
+
import { SERVICE_MAX_BODY_BYTES } from "./constants.ts";
|
|
2
|
+
import { VERSION } from "./version.ts";
|
|
2
3
|
import { validateMetricObservation, type MetricObservation, type MetricQuery, type StoredMetricObservation } from "./domain/metric.ts";
|
|
3
4
|
import type { MetricStore } from "./ports/metric-store.ts";
|
|
4
5
|
import type { RouteOverride, RouterController, RouterStatus, TelemetryPollResult } from "./ports/router-controller.ts";
|
package/src/version.ts
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
import { readFileSync } from "node:fs";
|
|
2
|
+
|
|
3
|
+
function packageVersion(): string {
|
|
4
|
+
const manifest = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8")) as unknown;
|
|
5
|
+
if (typeof manifest !== "object" || manifest === null || Array.isArray(manifest)) {
|
|
6
|
+
throw new Error("Jittor package manifest must be an object");
|
|
7
|
+
}
|
|
8
|
+
const version = (manifest as Record<string, unknown>)["version"];
|
|
9
|
+
if (typeof version !== "string" || !/^\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?$/.test(version)) {
|
|
10
|
+
throw new Error("Jittor package manifest has an invalid version");
|
|
11
|
+
}
|
|
12
|
+
return version;
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
/** Runtime package version; package.json is the single release source of truth. */
|
|
16
|
+
export const VERSION = packageVersion();
|