llm-relay 0.12.0 → 0.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -295,6 +295,25 @@ Claude Code subagent frontmatter (`model:`), which accepts a full model id but n
295
295
  list. A pool is the indirection that gives those callers ranking and failover. `pool` is a
296
296
  reserved provider name; configuring a provider called `pool` fails at load.
297
297
 
298
+ **What failover actually does** (both `/v1/messages` and `/v1/chat/completions`):
299
+
300
+ - **429 / 5xx / 400 / 404** → the candidate is recorded as a breaker failure and the next one is
301
+ tried. A `Retry-After` sets that candidate's cooldown for exactly as long as the provider asked.
302
+ - **401 / 403** → the next candidate is tried, but the fault is recorded on its own axis rather
303
+ than as ill health, so `llm-relay candidates` shows it as `AUTH 401` instead of hiding it. It
304
+ expires after 5 minutes, so a rotated key recovers with no restart.
305
+ - **A genuine client 4xx** (413, 422, …) → returned as-is. Every other candidate would reject it
306
+ identically.
307
+ - **Every candidate failed** → the last real upstream error, not a synthesized one.
308
+
309
+ Responses carry **`x-llm-relay-served-by`**: the deployment that served, or on an error every
310
+ deployment that was tried, in order.
311
+
312
+ ⚠ **A pool routes to fewer members than it lists** when some declare an `authEnv` that is unset —
313
+ those are dropped before ranking, so a 14-member pool can resolve to 7 and the config's *tenth*
314
+ entry can legitimately be the one that answers. `llm-relay candidates` reports the count.
315
+ Background: [docs/pool-failover.md](docs/pool-failover.md).
316
+
298
317
  ### Quieting the onboarding nudge (`leave_me_alone`)
299
318
 
300
319
  `llm-relay onboard` walks every known provider and prompts for the keys you are missing. For a
@@ -518,7 +537,7 @@ silently at request time.
518
537
 
519
538
  ### Discovery endpoint (`GET /registry`) — for a dispatcher
520
539
 
521
- For a caller that does its own selection (e.g. audit-tools dispatch, which weighs
540
+ For a caller that does its own selection (an external dispatcher weighing
522
541
  quota / rate limits / token budget), `GET http://127.0.0.1:8791/registry` returns one
523
542
  coherent JSON view:
524
543
 
@@ -629,7 +648,7 @@ headroom's backend-agnostic memory/learn layer. So stack it for context-fit, not
629
648
 
630
649
  ## Design
631
650
 
632
- Consumers (audit-tools dispatch, plain `claude` CLI) point `ANTHROPIC_BASE_URL` at this proxy; it validates one backend per request. Target *selection* / token-prediction is a separate concern (the router/auditor), deliberately not here. For architecture, invariants, and the script inventory, see [CLAUDE.md](CLAUDE.md).
651
+ Consumers (an external dispatcher, plain `claude` CLI) point `ANTHROPIC_BASE_URL` at this proxy; it validates one backend per request. Target *selection* / token-prediction is a separate concern (the router/auditor), deliberately not here. For architecture, invariants, and the script inventory, see [CLAUDE.md](CLAUDE.md).
633
652
 
634
653
  ## Dev
635
654
 
package/dist/backend.d.ts CHANGED
@@ -13,6 +13,25 @@ export declare const ERROR_ORIGIN_HEADER = "x-llm-relay-error-origin";
13
13
  export type ErrorOrigin = "upstream" | "local";
14
14
  /** Read the origin marker off a Response, when it carries one. */
15
15
  export declare function errorOrigin(res: Response): ErrorOrigin | null;
16
+ /**
17
+ * Response header naming the deployment that actually answered — the (provider, model) left
18
+ * standing after pool expansion, benchmark ranking, breaker demotion and failover.
19
+ *
20
+ * A pool request's most basic debugging question is "who served this?", and until now the only
21
+ * way to answer it was to correlate timestamps against the proxy's own log. When every candidate
22
+ * fails it carries the list that was tried instead, so an exhausted pool is self-describing.
23
+ */
24
+ export declare const SERVED_BY_HEADER = "x-llm-relay-served-by";
25
+ /**
26
+ * The provider's `Retry-After` in milliseconds, or null.
27
+ *
28
+ * Accepts both RFC 9110 forms — delta-seconds and an HTTP-date — because providers use both
29
+ * (groq sends seconds, some CDNs in front of a provider send a date). A date in the past, a
30
+ * negative delta or an unparseable value yields null rather than 0: "the provider said nothing
31
+ * usable" and "the provider said retry immediately" call for different cooldowns, and treating
32
+ * garbage as 0 would silently disable the backoff this exists to honour.
33
+ */
34
+ export declare function parseRetryAfterMs(value: string | null | undefined, now?: number): number | null;
16
35
  /**
17
36
  * Fetch the resolved provider target and return an ANTHROPIC-shaped `Response`,
18
37
  * regardless of the backend's native wire format. For kind="anthropic" this is a
@@ -32,13 +51,34 @@ export declare function fetchBackend(target: ResolvedTarget, args: {
32
51
  }, fetchFn?: typeof fetch): Promise<Response>;
33
52
  /** Map a non-streaming OpenAI chat completion into an Anthropic message. */
34
53
  export declare function openAiResponseToAnthropic(j: Record<string, unknown>, model: string): object;
54
+ /**
55
+ * Coerce an upstream error body into the OpenAI error envelope — WITHOUT rewriting one that
56
+ * already conforms.
57
+ *
58
+ * The OpenAI front promises "OpenAI in, OpenAI out", but on the error path it returned whatever
59
+ * shape the provider chose. Gemini wraps its error in a JSON ARRAY (`[{"error":{…}}]`); a client
60
+ * reading `response.choices[0]` gets `undefined` from that and reports a malformed completion,
61
+ * so a plain 429 surfaces as "the model returned garbage" — which is exactly how a rate limit
62
+ * cost two days of debugging on the caller's side.
63
+ *
64
+ * Rules, in order:
65
+ * - already `{error:{…}}` → returned BYTE-EXACT. A conforming provider's message, code and
66
+ * type are its own to state, and rewriting them would lose detail.
67
+ * - `[{error:{…}}, …]` → unwrapped to the element. Same fields, now at the top level.
68
+ * - anything else (HTML, text,
69
+ * a bare string, empty) → wrapped, with the original preserved as the message.
70
+ *
71
+ * Returns null when the body is already conforming, so the caller can stream the original bytes
72
+ * rather than re-serialize them.
73
+ */
74
+ export declare function normalizeOpenAiErrorBody(body: string, status: number): string | null;
35
75
  /**
36
76
  * OpenAI-compatible FRONT: an OpenAI `/chat/completions` request comes in, its `model`
37
77
  * has already been resolved to a provider target by namespace/tier routing. For an
38
78
  * openai-kind target this is a routing reverse-proxy — rewrite `model` to the backend
39
79
  * id, inject the backend key, and stream the upstream OpenAI response straight back
40
80
  * (OpenAI in, OpenAI out — no translation). This is the transport a dispatcher (e.g.
41
- * audit-tools) consumes to reach many backends behind one endpoint.
81
+ * an external dispatcher) consumes to reach many backends behind one endpoint.
42
82
  *
43
83
  * anthropic-kind targets are not served on the OpenAI front (they need OpenAI↔Anthropic
44
84
  * translation and are not the dispatcher use case) — a clean 400, never a mistranslation.
package/dist/backend.js CHANGED
@@ -15,6 +15,41 @@ export function errorOrigin(res) {
15
15
  const v = res.headers.get(ERROR_ORIGIN_HEADER);
16
16
  return v === "upstream" || v === "local" ? v : null;
17
17
  }
18
+ /**
19
+ * Response header naming the deployment that actually answered — the (provider, model) left
20
+ * standing after pool expansion, benchmark ranking, breaker demotion and failover.
21
+ *
22
+ * A pool request's most basic debugging question is "who served this?", and until now the only
23
+ * way to answer it was to correlate timestamps against the proxy's own log. When every candidate
24
+ * fails it carries the list that was tried instead, so an exhausted pool is self-describing.
25
+ */
26
+ export const SERVED_BY_HEADER = "x-llm-relay-served-by";
27
+ /**
28
+ * The provider's `Retry-After` in milliseconds, or null.
29
+ *
30
+ * Accepts both RFC 9110 forms — delta-seconds and an HTTP-date — because providers use both
31
+ * (groq sends seconds, some CDNs in front of a provider send a date). A date in the past, a
32
+ * negative delta or an unparseable value yields null rather than 0: "the provider said nothing
33
+ * usable" and "the provider said retry immediately" call for different cooldowns, and treating
34
+ * garbage as 0 would silently disable the backoff this exists to honour.
35
+ */
36
+ export function parseRetryAfterMs(value, now = Date.now()) {
37
+ if (typeof value !== "string")
38
+ return null;
39
+ const raw = value.trim();
40
+ if (raw.length === 0)
41
+ return null;
42
+ // delta-seconds — integer per the RFC, but providers do emit fractions ("20.45").
43
+ if (/^\d+(\.\d+)?$/.test(raw)) {
44
+ const ms = Math.round(Number(raw) * 1000);
45
+ return Number.isFinite(ms) && ms >= 0 ? ms : null;
46
+ }
47
+ const at = Date.parse(raw);
48
+ if (Number.isNaN(at))
49
+ return null;
50
+ const delta = at - now;
51
+ return delta > 0 ? delta : null;
52
+ }
18
53
  /**
19
54
  * Fetch the resolved provider target and return an ANTHROPIC-shaped `Response`,
20
55
  * regardless of the backend's native wire format. For kind="anthropic" this is a
@@ -88,7 +123,12 @@ export async function fetchBackend(target, args, fetchFn = fetch) {
88
123
  ? ` — model "${target.model}" is not served by provider "${target.provider}" (a model can be listed in /models and still 404 here)`
89
124
  : "";
90
125
  // The provider really answered with this status — the body is reworded, the origin is not.
91
- return anthropicError(res.status, `openai backend HTTP ${res.status}${hint}: ${body.slice(0, 300)}`, "upstream");
126
+ // `Retry-After` is carried onto the synthesized error: this response is a NEW Response, so
127
+ // without this the one header stating when the provider will serve again was destroyed here,
128
+ // and neither the breaker's cooldown nor the client's backoff could ever honour it.
129
+ return anthropicError(res.status, `openai backend HTTP ${res.status}${hint}: ${body.slice(0, 300)}`, "upstream", {
130
+ ...retryAfterHeader(res.headers),
131
+ });
92
132
  }
93
133
  if (args.wantsStream && res.body) {
94
134
  const anthStream = handleUniversalStreamRequest(res.body, "openai", "anthropic");
@@ -138,10 +178,15 @@ export function openAiResponseToAnthropic(j, model) {
138
178
  usage: { input_tokens: usage.prompt_tokens ?? 0, output_tokens: usage.completion_tokens ?? 0 },
139
179
  };
140
180
  }
141
- function anthropicError(status, message, origin) {
181
+ /** The upstream's `Retry-After`, as a header object to spread, or `{}` when it sent none. */
182
+ function retryAfterHeader(hh) {
183
+ const v = hh.get("retry-after");
184
+ return v ? { "retry-after": v } : {};
185
+ }
186
+ function anthropicError(status, message, origin, extra = {}) {
142
187
  return new Response(JSON.stringify({ type: "error", error: { type: "api_error", message } }), {
143
188
  status,
144
- headers: { "content-type": "application/json", [ERROR_ORIGIN_HEADER]: origin },
189
+ headers: { "content-type": "application/json", [ERROR_ORIGIN_HEADER]: origin, ...extra },
145
190
  });
146
191
  }
147
192
  function openaiError(status, message, origin) {
@@ -150,13 +195,58 @@ function openaiError(status, message, origin) {
150
195
  headers: { "content-type": "application/json", [ERROR_ORIGIN_HEADER]: origin },
151
196
  });
152
197
  }
198
+ /**
199
+ * Coerce an upstream error body into the OpenAI error envelope — WITHOUT rewriting one that
200
+ * already conforms.
201
+ *
202
+ * The OpenAI front promises "OpenAI in, OpenAI out", but on the error path it returned whatever
203
+ * shape the provider chose. Gemini wraps its error in a JSON ARRAY (`[{"error":{…}}]`); a client
204
+ * reading `response.choices[0]` gets `undefined` from that and reports a malformed completion,
205
+ * so a plain 429 surfaces as "the model returned garbage" — which is exactly how a rate limit
206
+ * cost two days of debugging on the caller's side.
207
+ *
208
+ * Rules, in order:
209
+ * - already `{error:{…}}` → returned BYTE-EXACT. A conforming provider's message, code and
210
+ * type are its own to state, and rewriting them would lose detail.
211
+ * - `[{error:{…}}, …]` → unwrapped to the element. Same fields, now at the top level.
212
+ * - anything else (HTML, text,
213
+ * a bare string, empty) → wrapped, with the original preserved as the message.
214
+ *
215
+ * Returns null when the body is already conforming, so the caller can stream the original bytes
216
+ * rather than re-serialize them.
217
+ */
218
+ export function normalizeOpenAiErrorBody(body, status) {
219
+ let parsed;
220
+ try {
221
+ parsed = JSON.parse(body);
222
+ }
223
+ catch {
224
+ const message = body.trim().slice(0, 2000) || `upstream returned HTTP ${status} with an empty body`;
225
+ return JSON.stringify({ error: { message, type: "upstream_error", code: status } });
226
+ }
227
+ const hasError = (v) => typeof v === "object" && v !== null && typeof v.error === "object" && v.error !== null;
228
+ if (hasError(parsed))
229
+ return null; // conforms — do not touch it
230
+ if (Array.isArray(parsed)) {
231
+ const wrapped = parsed.find(hasError);
232
+ if (wrapped)
233
+ return JSON.stringify(wrapped);
234
+ }
235
+ return JSON.stringify({
236
+ error: {
237
+ message: body.trim().slice(0, 2000) || `upstream returned HTTP ${status} with an empty body`,
238
+ type: "upstream_error",
239
+ code: status,
240
+ },
241
+ });
242
+ }
153
243
  /**
154
244
  * OpenAI-compatible FRONT: an OpenAI `/chat/completions` request comes in, its `model`
155
245
  * has already been resolved to a provider target by namespace/tier routing. For an
156
246
  * openai-kind target this is a routing reverse-proxy — rewrite `model` to the backend
157
247
  * id, inject the backend key, and stream the upstream OpenAI response straight back
158
248
  * (OpenAI in, OpenAI out — no translation). This is the transport a dispatcher (e.g.
159
- * audit-tools) consumes to reach many backends behind one endpoint.
249
+ * an external dispatcher) consumes to reach many backends behind one endpoint.
160
250
  *
161
251
  * anthropic-kind targets are not served on the OpenAI front (they need OpenAI↔Anthropic
162
252
  * translation and are not the dispatcher use case) — a clean 400, never a mistranslation.
@@ -1 +1 @@
1
- {"version":3,"file":"backend.js","sourceRoot":"","sources":["../src/backend.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,yBAAyB,EAAE,4BAA4B,EAAE,MAAM,YAAY,CAAC;AAErF,OAAO,EAAE,aAAa,EAAE,kBAAkB,EAAE,MAAM,gBAAgB,CAAC;AAEnE;;;;;;;;GAQG;AACH,MAAM,CAAC,MAAM,mBAAmB,GAAG,0BAA0B,CAAC;AAK9D,kEAAkE;AAClE,MAAM,UAAU,WAAW,CAAC,GAAa;IACvC,MAAM,CAAC,GAAG,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,CAAC,CAAC;IAC/C,OAAO,CAAC,KAAK,UAAU,IAAI,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;AACtD,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,MAAsB,EACtB,IAQC,EACD,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;QAChC,MAAM,IAAI,GAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,OAAO,EAAE,IAAI,CAAC,gBAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC;QACvG,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM;YAAE,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC;QAChD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,oBAAoB;IACpB,sFAAsF;IACtF,sFAAsF;IACtF,IAAI,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;IAC3B,IAAI,CAAC;QACH,OAAO,GAAG,MAAM,kBAAkB,CAAC,OAAO,CAAC,CAAC;IAC9C,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,sFAAsF;QACtF,sFAAsF;QACtF,IAAI,CAAC,YAAY,aAAa;YAAE,OAAO,cAAc,CAAC,GAAG,EAAE,cAAc,CAAC,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;QAC/F,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC7F,CAAC;IAED,IAAI,UAAmC,CAAC;IACxC,IAAI,CAAC;QACH,UAAU,GAAG,yBAAyB,CAAC,WAAW,EAAE,QAAQ,EAAE,CAAC,OAAO,IAAI,EAAE,CAAU,CAA4B,CAAC;IACrH,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC7F,CAAC;IACD,UAAU,CAAC,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC;IAChC,UAAU,CAAC,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC;IACrC,sFAAsF;IACtF,qFAAqF;IACrF,mFAAmF;IACnF,IAAI,IAAI,CAAC,WAAW;QAAE,UAAU,CAAC,cAAc,GAAG,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC;IAE1E,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,IAA6B,EAAE,EAAE,CAC7C,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QACzC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;QAC1B,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;IAEL,IAAI,GAAG,GAAG,MAAM,IAAI,CAAC,UAAU,CAAC,CAAC;IAEjC,oFAAoF;IACpF,6EAA6E;IAC7E,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,UAAU,CAAC,cAAc,IAAI,CAAC,GAAG,CAAC,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,CAAC,EAAE,CAAC;QACvF,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,GAAG,YAAY,EAAE,GAAG,UAAU,CAAC;QAC9D,GAAG,GAAG,MAAM,IAAI,CAAC,YAAY,CAAC,CAAC;IACjC,CAAC;IAED,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;QACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC;QAC9B,6EAA6E;QAC7E,sEAAsE;QACtE,2DAA2D;QAC3D,MAAM,IAAI,GACR,GAAG,CAAC,MAAM,KAAK,GAAG;YAChB,CAAC,CAAC,aAAa,MAAM,CAAC,KAAK,gCAAgC,MAAM,CAAC,QAAQ,yDAAyD;YACnI,CAAC,CAAC,EAAE,CAAC;QACT,2FAA2F;QAC3F,OAAO,cAAc,CAAC,GAAG,CAAC,MAAM,EAAE,uBAAuB,GAAG,CAAC,MAAM,GAAG,IAAI,KAAK,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,EAAE,UAAU,CAAC,CAAC;IACnH,CAAC;IAED,IAAI,IAAI,CAAC,WAAW,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;QACjC,MAAM,UAAU,GAAG,4BAA4B,CAAC,GAAG,CAAC,IAAI,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC;QACjF,OAAO,IAAI,QAAQ,CAAC,UAAU,EAAE,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,mBAAmB,EAAE,EAAE,CAAC,CAAC;IAC5G,CAAC;IAED,IAAI,aAAqB,CAAC;IAC1B,IAAI,CAAC;QACH,aAAa,GAAG,yBAAyB,CAAC,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAA4B,EAAE,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC;IAC/G,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,kFAAkF;QAClF,kFAAkF;QAClF,OAAO,cAAc,CAAC,GAAG,EAAE,gCAAiC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC9F,CAAC;IACD,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,EAAE,EAAE,MAAM,EAAE,GAAG,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,EAAE,CAAC,CAAC;AACvH,CAAC;AAED,4EAA4E;AAC5E,MAAM,UAAU,yBAAyB,CAAC,CAA0B,EAAE,KAAa;IACjF,MAAM,MAAM,GAAI,CAAC,CAAC,OAAsD,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;IACpF,MAAM,GAAG,GAAI,MAAM,CAAC,OAA+C,IAAI,EAAE,CAAC;IAC1E,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,IAAI,OAAO,GAAG,CAAC,OAAO,KAAK,QAAQ,IAAI,GAAG,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;IACjH,MAAM,SAAS,GAAI,GAAG,CAAC,UAAyD,IAAI,EAAE,CAAC;IACvF,KAAK,MAAM,EAAE,IAAI,SAAS,EAAE,CAAC;QAC3B,MAAM,EAAE,GAAI,EAAE,CAAC,QAAgD,IAAI,EAAE,CAAC;QACtE,IAAI,KAAc,CAAC;QACnB,IAAI,CAAC;YAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAE,EAAE,CAAC,SAAoB,IAAI,IAAI,CAAC,CAAC;QAAC,CAAC;QAAC,MAAM,CAAC;YAAC,KAAK,GAAG,EAAE,CAAC,SAAS,IAAI,EAAE,CAAC;QAAC,CAAC;QACnG,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,UAAU,EAAE,EAAE,EAAG,EAAE,CAAC,EAAa,IAAI,IAAI,EAAE,IAAI,EAAG,EAAE,CAAC,IAAe,IAAI,EAAE,EAAE,KAAK,EAAE,CAAC,CAAC;IAC5G,CAAC;IACD,MAAM,MAAM,GAAG,MAAM,CAAC,aAAmC,CAAC;IAC1D,MAAM,UAAU,GACd,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,KAAK,QAAQ,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,IAAI,UAAU,CAAC;IACjI,MAAM,KAAK,GAAI,CAAC,CAAC,KAA4C,IAAI,EAAE,CAAC;IACpE,OAAO;QACL,EAAE,EAAG,CAAC,CAAC,EAAa,IAAI,gBAAgB;QACxC,IAAI,EAAE,SAAS;QACf,IAAI,EAAE,WAAW;QACjB,KAAK,EAAE,KAAK,IAAI,CAAE,CAAC,CAAC,KAAgB,IAAI,EAAE,CAAC;QAC3C,OAAO;QACP,WAAW,EAAE,UAAU;QACvB,aAAa,EAAE,IAAI;QACnB,KAAK,EAAE,EAAE,YAAY,EAAE,KAAK,CAAC,aAAa,IAAI,CAAC,EAAE,aAAa,EAAE,KAAK,CAAC,iBAAiB,IAAI,CAAC,EAAE;KAC/F,CAAC;AACJ,CAAC;AAED,SAAS,cAAc,CAAC,MAAc,EAAE,OAAe,EAAE,MAAmB;IAC1E,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE,IAAI,EAAE,WAAW,EAAE,OAAO,EAAE,EAAE,CAAC,EAAE;QAC5F,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,mBAAmB,CAAC,EAAE,MAAM,EAAE;KAC/E,CAAC,CAAC;AACL,CAAC;AAED,SAAS,WAAW,CAAC,MAAc,EAAE,OAAe,EAAE,MAAmB;IACvE,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,uBAAuB,EAAE,EAAE,CAAC,EAAE;QACzF,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,mBAAmB,CAAC,EAAE,MAAM,EAAE;KAC/E,CAAC,CAAC;AACL,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,CAAC,KAAK,UAAU,gBAAgB,CACpC,MAAsB,EACtB,IAAqE,EACrE,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;QAC7B,OAAO,WAAW,CAChB,GAAG,EACH,8DAA8D,MAAM,CAAC,QAAQ,QAAQ,MAAM,CAAC,IAAI,EAAE,EAClG,OAAO,CACR,CAAC;IACJ,CAAC;IACD,MAAM,IAAI,GAAG,CAAC,IAAI,CAAC,OAAO,IAAI,EAAE,CAA4B,CAAC;IAC7D,MAAM,IAAI,GAAG,EAAE,GAAG,IAAI,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,WAAW,EAAE,CAAC;IACxE,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IACD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QAChD,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;QAC1B,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;AACL,CAAC"}
1
+ {"version":3,"file":"backend.js","sourceRoot":"","sources":["../src/backend.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,yBAAyB,EAAE,4BAA4B,EAAE,MAAM,YAAY,CAAC;AAErF,OAAO,EAAE,aAAa,EAAE,kBAAkB,EAAE,MAAM,gBAAgB,CAAC;AAEnE;;;;;;;;GAQG;AACH,MAAM,CAAC,MAAM,mBAAmB,GAAG,0BAA0B,CAAC;AAK9D,kEAAkE;AAClE,MAAM,UAAU,WAAW,CAAC,GAAa;IACvC,MAAM,CAAC,GAAG,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,CAAC,CAAC;IAC/C,OAAO,CAAC,KAAK,UAAU,IAAI,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;AACtD,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,gBAAgB,GAAG,uBAAuB,CAAC;AAExD;;;;;;;;GAQG;AACH,MAAM,UAAU,iBAAiB,CAAC,KAAgC,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;IAClF,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,IAAI,CAAC;IAC3C,MAAM,GAAG,GAAG,KAAK,CAAC,IAAI,EAAE,CAAC;IACzB,IAAI,GAAG,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,IAAI,CAAC;IAElC,kFAAkF;IAClF,IAAI,eAAe,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QAC9B,MAAM,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,IAAI,CAAC,CAAC;QAC1C,OAAO,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IACpD,CAAC;IAED,MAAM,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAC3B,IAAI,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;QAAE,OAAO,IAAI,CAAC;IAClC,MAAM,KAAK,GAAG,EAAE,GAAG,GAAG,CAAC;IACvB,OAAO,KAAK,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC;AAClC,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,MAAsB,EACtB,IAQC,EACD,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;QAChC,MAAM,IAAI,GAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,OAAO,EAAE,IAAI,CAAC,gBAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC;QACvG,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM;YAAE,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC;QAChD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,oBAAoB;IACpB,sFAAsF;IACtF,sFAAsF;IACtF,IAAI,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;IAC3B,IAAI,CAAC;QACH,OAAO,GAAG,MAAM,kBAAkB,CAAC,OAAO,CAAC,CAAC;IAC9C,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,sFAAsF;QACtF,sFAAsF;QACtF,IAAI,CAAC,YAAY,aAAa;YAAE,OAAO,cAAc,CAAC,GAAG,EAAE,cAAc,CAAC,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;QAC/F,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC7F,CAAC;IAED,IAAI,UAAmC,CAAC;IACxC,IAAI,CAAC;QACH,UAAU,GAAG,yBAAyB,CAAC,WAAW,EAAE,QAAQ,EAAE,CAAC,OAAO,IAAI,EAAE,CAAU,CAA4B,CAAC;IACrH,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC7F,CAAC;IACD,UAAU,CAAC,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC;IAChC,UAAU,CAAC,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC;IACrC,sFAAsF;IACtF,qFAAqF;IACrF,mFAAmF;IACnF,IAAI,IAAI,CAAC,WAAW;QAAE,UAAU,CAAC,cAAc,GAAG,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC;IAE1E,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,IAA6B,EAAE,EAAE,CAC7C,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QACzC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;QAC1B,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;IAEL,IAAI,GAAG,GAAG,MAAM,IAAI,CAAC,UAAU,CAAC,CAAC;IAEjC,oFAAoF;IACpF,6EAA6E;IAC7E,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,UAAU,CAAC,cAAc,IAAI,CAAC,GAAG,CAAC,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,CAAC,EAAE,CAAC;QACvF,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,GAAG,YAAY,EAAE,GAAG,UAAU,CAAC;QAC9D,GAAG,GAAG,MAAM,IAAI,CAAC,YAAY,CAAC,CAAC;IACjC,CAAC;IAED,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;QACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC;QAC9B,6EAA6E;QAC7E,sEAAsE;QACtE,2DAA2D;QAC3D,MAAM,IAAI,GACR,GAAG,CAAC,MAAM,KAAK,GAAG;YAChB,CAAC,CAAC,aAAa,MAAM,CAAC,KAAK,gCAAgC,MAAM,CAAC,QAAQ,yDAAyD;YACnI,CAAC,CAAC,EAAE,CAAC;QACT,2FAA2F;QAC3F,2FAA2F;QAC3F,6FAA6F;QAC7F,oFAAoF;QACpF,OAAO,cAAc,CAAC,GAAG,CAAC,MAAM,EAAE,uBAAuB,GAAG,CAAC,MAAM,GAAG,IAAI,KAAK,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,EAAE,UAAU,EAAE;YAC/G,GAAG,gBAAgB,CAAC,GAAG,CAAC,OAAO,CAAC;SACjC,CAAC,CAAC;IACL,CAAC;IAED,IAAI,IAAI,CAAC,WAAW,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;QACjC,MAAM,UAAU,GAAG,4BAA4B,CAAC,GAAG,CAAC,IAAI,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC;QACjF,OAAO,IAAI,QAAQ,CAAC,UAAU,EAAE,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,mBAAmB,EAAE,EAAE,CAAC,CAAC;IAC5G,CAAC;IAED,IAAI,aAAqB,CAAC;IAC1B,IAAI,CAAC;QACH,aAAa,GAAG,yBAAyB,CAAC,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAA4B,EAAE,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC;IAC/G,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,kFAAkF;QAClF,kFAAkF;QAClF,OAAO,cAAc,CAAC,GAAG,EAAE,gCAAiC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC9F,CAAC;IACD,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,EAAE,EAAE,MAAM,EAAE,GAAG,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,EAAE,CAAC,CAAC;AACvH,CAAC;AAED,4EAA4E;AAC5E,MAAM,UAAU,yBAAyB,CAAC,CAA0B,EAAE,KAAa;IACjF,MAAM,MAAM,GAAI,CAAC,CAAC,OAAsD,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;IACpF,MAAM,GAAG,GAAI,MAAM,CAAC,OAA+C,IAAI,EAAE,CAAC;IAC1E,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,IAAI,OAAO,GAAG,CAAC,OAAO,KAAK,QAAQ,IAAI,GAAG,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;IACjH,MAAM,SAAS,GAAI,GAAG,CAAC,UAAyD,IAAI,EAAE,CAAC;IACvF,KAAK,MAAM,EAAE,IAAI,SAAS,EAAE,CAAC;QAC3B,MAAM,EAAE,GAAI,EAAE,CAAC,QAAgD,IAAI,EAAE,CAAC;QACtE,IAAI,KAAc,CAAC;QACnB,IAAI,CAAC;YAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAE,EAAE,CAAC,SAAoB,IAAI,IAAI,CAAC,CAAC;QAAC,CAAC;QAAC,MAAM,CAAC;YAAC,KAAK,GAAG,EAAE,CAAC,SAAS,IAAI,EAAE,CAAC;QAAC,CAAC;QACnG,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,UAAU,EAAE,EAAE,EAAG,EAAE,CAAC,EAAa,IAAI,IAAI,EAAE,IAAI,EAAG,EAAE,CAAC,IAAe,IAAI,EAAE,EAAE,KAAK,EAAE,CAAC,CAAC;IAC5G,CAAC;IACD,MAAM,MAAM,GAAG,MAAM,CAAC,aAAmC,CAAC;IAC1D,MAAM,UAAU,GACd,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,KAAK,QAAQ,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,IAAI,UAAU,CAAC;IACjI,MAAM,KAAK,GAAI,CAAC,CAAC,KAA4C,IAAI,EAAE,CAAC;IACpE,OAAO;QACL,EAAE,EAAG,CAAC,CAAC,EAAa,IAAI,gBAAgB;QACxC,IAAI,EAAE,SAAS;QACf,IAAI,EAAE,WAAW;QACjB,KAAK,EAAE,KAAK,IAAI,CAAE,CAAC,CAAC,KAAgB,IAAI,EAAE,CAAC;QAC3C,OAAO;QACP,WAAW,EAAE,UAAU;QACvB,aAAa,EAAE,IAAI;QACnB,KAAK,EAAE,EAAE,YAAY,EAAE,KAAK,CAAC,aAAa,IAAI,CAAC,EAAE,aAAa,EAAE,KAAK,CAAC,iBAAiB,IAAI,CAAC,EAAE;KAC/F,CAAC;AACJ,CAAC;AAED,6FAA6F;AAC7F,SAAS,gBAAgB,CAAC,EAAW;IACnC,MAAM,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC;IAChC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;AACvC,CAAC;AAED,SAAS,cAAc,CACrB,MAAc,EACd,OAAe,EACf,MAAmB,EACnB,QAAgC,EAAE;IAElC,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE,IAAI,EAAE,WAAW,EAAE,OAAO,EAAE,EAAE,CAAC,EAAE;QAC5F,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,mBAAmB,CAAC,EAAE,MAAM,EAAE,GAAG,KAAK,EAAE;KACzF,CAAC,CAAC;AACL,CAAC;AAED,SAAS,WAAW,CAAC,MAAc,EAAE,OAAe,EAAE,MAAmB;IACvE,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,uBAAuB,EAAE,EAAE,CAAC,EAAE;QACzF,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,mBAAmB,CAAC,EAAE,MAAM,EAAE;KAC/E,CAAC,CAAC;AACL,CAAC;AAED;;;;;;;;;;;;;;;;;;;GAmBG;AACH,MAAM,UAAU,wBAAwB,CAAC,IAAY,EAAE,MAAc;IACnE,IAAI,MAAe,CAAC;IACpB,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAC5B,CAAC;IAAC,MAAM,CAAC;QACP,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,0BAA0B,MAAM,qBAAqB,CAAC;QACpG,OAAO,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,gBAAgB,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE,CAAC,CAAC;IACtF,CAAC;IAED,MAAM,QAAQ,GAAG,CAAC,CAAU,EAA2C,EAAE,CACvE,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,KAAK,IAAI,IAAI,OAAQ,CAAyB,CAAC,KAAK,KAAK,QAAQ,IAAK,CAAyB,CAAC,KAAK,KAAK,IAAI,CAAC;IAE3I,IAAI,QAAQ,CAAC,MAAM,CAAC;QAAE,OAAO,IAAI,CAAC,CAAC,6BAA6B;IAEhE,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;QAC1B,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QACtC,IAAI,OAAO;YAAE,OAAO,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC;IAC9C,CAAC;IAED,OAAO,IAAI,CAAC,SAAS,CAAC;QACpB,KAAK,EAAE;YACL,OAAO,EAAE,IAAI,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,0BAA0B,MAAM,qBAAqB;YAC5F,IAAI,EAAE,gBAAgB;YACtB,IAAI,EAAE,MAAM;SACb;KACF,CAAC,CAAC;AACL,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,CAAC,KAAK,UAAU,gBAAgB,CACpC,MAAsB,EACtB,IAAqE,EACrE,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;QAC7B,OAAO,WAAW,CAChB,GAAG,EACH,8DAA8D,MAAM,CAAC,QAAQ,QAAQ,MAAM,CAAC,IAAI,EAAE,EAClG,OAAO,CACR,CAAC;IACJ,CAAC;IACD,MAAM,IAAI,GAAG,CAAC,IAAI,CAAC,OAAO,IAAI,EAAE,CAA4B,CAAC;IAC7D,MAAM,IAAI,GAAG,EAAE,GAAG,IAAI,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,WAAW,EAAE,CAAC;IACxE,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IACD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QAChD,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;QAC1B,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;AACL,CAAC"}
@@ -47,6 +47,19 @@ export interface Candidate {
47
47
  consecutiveFailures: number;
48
48
  lastStatus: number | null;
49
49
  cooldownRemainingMs: number;
50
+ /**
51
+ * Credential faults observed on real traffic, on their own axis.
52
+ *
53
+ * A member answering 401 on every call used to be indistinguishable here from a healthy
54
+ * one — verdict `Pending`, breaker `closed`, no failure count — because a 401 is
55
+ * deliberately not health data and so reached none of the fields above. It is still not
56
+ * health data; it is reported as what it is. `credentialFault` true means the router is
57
+ * currently DEMOTING this member (it is tried only after every other candidate fails),
58
+ * and it expires, so a rotated key recovers without a restart.
59
+ */
60
+ credentialFailures: number;
61
+ lastCredentialStatus: number | null;
62
+ credentialFault: boolean;
50
63
  };
51
64
  /** Observed real traffic through this proxy (not synthetic probes). */
52
65
  observed: {
@@ -127,6 +127,9 @@ export async function buildCandidates(cfg, opts = {}) {
127
127
  consecutiveFailures: state?.consecutiveFailures ?? 0,
128
128
  lastStatus: state?.lastStatus ?? null,
129
129
  cooldownRemainingMs: Math.max(0, (state?.cooldownUntil ?? 0) - nowMs),
130
+ credentialFailures: state?.credentialFailures ?? 0,
131
+ lastCredentialStatus: state?.lastCredentialStatus ?? null,
132
+ credentialFault: breaker.hasCredentialFault(spec, nowMs),
130
133
  },
131
134
  observed: obs
132
135
  ? {
@@ -1 +1 @@
1
- {"version":3,"file":"candidates.js","sourceRoot":"","sources":["../src/candidates.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAG1C,OAAO,EAAE,WAAW,EAAsB,MAAM,iBAAiB,CAAC;AAClE,OAAO,EAAE,aAAa,EAAiB,MAAM,gBAAgB,CAAC;AAC9D,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,eAAe,EAAuB,MAAM,eAAe,CAAC;AACrE,OAAO,EAAE,oBAAoB,EAAuB,MAAM,sBAAsB,CAAC;AACjF,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC;AACnE,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAmH7C,MAAM,IAAI,GACR,8FAA8F;IAC9F,yFAAyF;IACzF,0FAA0F;IAC1F,wFAAwF,CAAC;AAE3F,2FAA2F;AAC3F,SAAS,UAAU,CAAC,IAAY,EAAE,GAAW;IAC3C,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IAC5B,MAAM,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAChD,IAAI,IAAI,KAAK,WAAW;QAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IACxC,MAAM,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IAC/C,OAAO,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;AACzC,CAAC;AAED;;;;GAIG;AACH,SAAS,YAAY,CAAC,GAAW;IAC/B,MAAM,GAAG,GAAG,IAAI,GAAG,EAAwD,CAAC;IAC5E,MAAM,KAAK,GAAG,CAAC,IAAY,EAAE,EAAE;QAC7B,IAAI,CAAC,GAAG,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACtB,IAAI,CAAC,CAAC,EAAE,CAAC;YACP,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,EAAE,aAAa,EAAE,EAAE,EAAE,CAAC;YACrC,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;QACnB,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC,CAAC;IAEF,KAAK,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,KAAK,IAAI,EAAE,CAAC,EAAE,CAAC;QACpE,KAAK,MAAM,CAAC,IAAI,KAAK;YAAE,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACnD,CAAC;IACD,KAAK,MAAM,CAAC,IAAI,EAAE,IAAI,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,EAAE,CAAC,EAAE,CAAC;QACvE,KAAK,MAAM,CAAC,IAAI,UAAU,CAAC,IAAI,EAAE,GAAG,CAAC;YAAE,KAAK,CAAC,CAAC,CAAC,CAAC,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC3E,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,SAAS,SAAS,CAAC,IAAY;IAC7B,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IAC5B,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC;AAClG,CAAC;AAED,+DAA+D;AAC/D,MAAM,CAAC,KAAK,UAAU,eAAe,CACnC,GAAW,EACX,OAWI,EAAE;IAEN,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,IAAI,oBAAoB,CAAC;IACrD,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC;IACvC,MAAM,MAAM,GAAG,CAAC,IAAI,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,EAAE,MAAM,IAAI,EAAE,CAAC;IAC5F,MAAM,SAAS,GAAG,oBAAoB,EAAE,CAAC;IAEzC,MAAM,UAAU,GAAgB,EAAE,CAAC;IACnC,KAAK,MAAM,CAAC,IAAI,EAAE,UAAU,CAAC,IAAI,YAAY,CAAC,GAAG,CAAC,EAAE,CAAC;QACnD,MAAM,EAAE,QAAQ,EAAE,KAAK,EAAE,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;QAC5C,IAAI,IAAI,CAAC,QAAQ,IAAI,QAAQ,KAAK,IAAI,CAAC,QAAQ;YAAE,SAAS;QAC1D,MAAM,CAAC,GAA+B,GAAG,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QAE9D,IAAI,MAAM,GAAmB,IAAI,CAAC;QAClC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,KAAK,QAAQ,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;YACtD,IAAI,CAAC;gBACH,MAAM,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,QAAQ,EAAE,CAAC,EAAE,KAAK,CAAC,CAAC;YACtD,CAAC;YAAC,MAAM,CAAC;gBACP,MAAM,GAAG,IAAI,CAAC;YAChB,CAAC;QACH,CAAC;QAED,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,IAAI,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,eAAe,CAAC,QAAQ,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QAC/F,MAAM,KAAK,GAAG,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC;QACrC,MAAM,GAAG,GAAG,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC,MAAM,CAAC,GAAG,QAAQ,IAAI,KAAK,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;QACzE,MAAM,QAAQ,GAAG,WAAW,CAAC,IAAI,CAAC,CAAC;QACnC,MAAM,OAAO,GAAG,aAAa,CAAC,KAAK,IAAI,IAAI,EAAE,MAAM,CAAC,CAAC;QACrD,MAAM,IAAI,GAAG,OAAO,EAAE,GAAG,CAAC;QAC1B,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAE,IAAI,CAAC,CAAC,CAAY,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QAExF,uFAAuF;QACvF,4FAA4F;QAC5F,2FAA2F;QAC3F,oCAAoC;QACpC,MAAM,cAAc,GAAG,CAAC,IAAI,CAAC,CAAC,IAAI,KAAK,QAAQ,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO;YACtE,CAAC,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC;YACjE,CAAC,CAAC,IAAI,CAAC;QACT,qFAAqF;QACrF,kFAAkF;QAClF,mFAAmF;QACnF,sFAAsF;QACtF,uFAAuF;QACvF,wBAAwB;QACxB,MAAM,aAAa,GACjB,OAAO,IAAI,OAAO,CAAC,KAAK,KAAK,OAAO,CAAC,CAAC,CAAC,cAAc,OAAO,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,YAAY,CAAC;QACzF,MAAM,IAAI,GAAG,eAAe,CAAC,KAAK,IAAI,QAAQ,EAAE;YAC9C,cAAc;YACd,SAAS,EAAE;gBACT,aAAa,EAAE,GAAG,CAAC,gBAAgB,CAAC;gBACpC,mBAAmB,EAAE,GAAG,CAAC,cAAc,CAAC;gBACxC,uBAAuB,EAAE,GAAG,CAAC,kBAAkB,CAAC;gBAChD,IAAI,EAAE,aAAa;aACpB;SACF,CAAC,CAAC;QAEH,UAAU,CAAC,IAAI,CAAC;YACd,IAAI;YACJ,QAAQ;YACR,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC3B,KAAK,EAAE,UAAU,CAAC,KAAK;YACvB,aAAa,EAAE,UAAU,CAAC,aAAa;YACvC,qFAAqF;YACrF,oFAAoF;YACpF,MAAM,EAAE,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,YAAY,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI;YAChE,MAAM;YACN,eAAe,EAAE,OAAO,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,GAAG,CAAC,IAAI,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI;YAClF,MAAM,EAAE,OAAO;gBACb,CAAC,CAAC;oBACE,OAAO,EAAE,OAAO,CAAC,OAAO;oBACxB,KAAK,EAAE,OAAO,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI;oBAChD,KAAK,EAAE,OAAO,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI;oBAChD,QAAQ,EAAE,OAAO,CAAC,QAAQ;oBAC1B,SAAS,EAAE,OAAO,CAAC,SAAS;oBAC5B,YAAY,EAAE,OAAO,CAAC,YAAY;oBAClC,UAAU,EAAE,OAAO,CAAC,UAAU;iBAC/B;gBACH,CAAC,CAAC,IAAI;YACR,YAAY,EAAE,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,gBAAgB,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,IAAI;YAC7E,OAAO,EAAE;gBACP,IAAI,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,EAAE,KAAK,CAAC;gBACrC,mBAAmB,EAAE,KAAK,EAAE,mBAAmB,IAAI,CAAC;gBACpD,UAAU,EAAE,KAAK,EAAE,UAAU,IAAI,IAAI;gBACrC,mBAAmB,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,KAAK,EAAE,aAAa,IAAI,CAAC,CAAC,GAAG,KAAK,CAAC;aACtE;YACD,QAAQ,EAAE,GAAG;gBACX,CAAC,CAAC;oBACE,UAAU,EAAE,GAAG,CAAC,UAAU;oBAC1B,YAAY,EAAE,GAAG,CAAC,YAAY;oBAC9B,YAAY,EAAE,GAAG,CAAC,UAAU,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,cAAc,GAAG,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,IAAI;oBACzF,YAAY,EAAE,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,GAAG,CAAC,YAAY,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,IAAI;iBACjF;gBACH,CAAC,CAAC,IAAI;YACR,aAAa,EAAE,IAAI,CAAC,aAAa;YACjC,mBAAmB,EAAE,IAAI,CAAC,mBAAmB;YAC7C,eAAe,EAAE,IAAI,CAAC,eAAe;YACrC,qBAAqB,EAAE,IAAI,CAAC,qBAAqB;YACjD,GAAG,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC,CAAC,EAAE,qBAAqB,EAAE,IAAI,CAAC,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC5E,cAAc,EAAE,IAAI,CAAC,cAAc;YACnC,eAAe,EAAE,IAAI,CAAC,eAAe;YACrC,WAAW,EAAE,IAAI,CAAC,WAAW;YAC7B,aAAa,EAAE,OAAO,IAAI,EAAE,cAAc,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,CAAC,IAAI;YACrF,iBAAiB,EAAE,KAAK,CAAC,OAAO,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC,CAAC,CAAE,IAAI,CAAC,OAAoB,CAAC,CAAC,CAAC,EAAE;YACjF,MAAM,EAAE;gBACN,WAAW,EAAE,GAAG,CAAC,cAAc,CAAC;gBAChC,aAAa,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACrC,eAAe,EAAE,GAAG,CAAC,kBAAkB,CAAC;gBACxC,cAAc,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACtC,QAAQ,EAAE,GAAG,CAAC,WAAW,CAAC;gBAC1B,SAAS,EAAE,GAAG,CAAC,YAAY,CAAC;gBAC5B,aAAa,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACrC,eAAe,EAAE,GAAG,CAAC,mBAAmB,CAAC;gBACzC,wBAAwB,EAAE,GAAG,CAAC,8BAA8B,CAAC;gBAC7D,wBAAwB,EAAE,GAAG,CAAC,8BAA8B,CAAC;gBAC7D,WAAW,EAAE,GAAG,CAAC,cAAc,CAAC;gBAChC,SAAS,EAAE,GAAG,CAAC,YAAY,CAAC;aAC7B;YACD,UAAU,EAAE;gBACV,QAAQ,EAAE,QAAQ,CAAC,KAAK;gBACxB,aAAa,EAAE,QAAQ,CAAC,KAAK;gBAC7B,eAAe,EAAE,QAAQ,CAAC,OAAO,IAAI,EAAE;gBACvC,gBAAgB,EAAE,OAAO,CAAC,oBAAoB,CAAC,IAAI,CAAC;aACrD;SACF,CAAC,CAAC;IACL,CAAC;IAED,OAAO;QACL,YAAY,EAAE,IAAI,CAAC,GAAG,IAAI,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC,WAAW,EAAE;QACvD,eAAe,EAAE,GAAG,CAAC,OAAO,CAAC,OAAO,KAAK,IAAI;QAC7C,IAAI,EAAE,IAAI;QACV,UAAU;KACX,CAAC;AACJ,CAAC"}
1
+ {"version":3,"file":"candidates.js","sourceRoot":"","sources":["../src/candidates.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAG1C,OAAO,EAAE,WAAW,EAAsB,MAAM,iBAAiB,CAAC;AAClE,OAAO,EAAE,aAAa,EAAiB,MAAM,gBAAgB,CAAC;AAC9D,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,eAAe,EAAuB,MAAM,eAAe,CAAC;AACrE,OAAO,EAAE,oBAAoB,EAAuB,MAAM,sBAAsB,CAAC;AACjF,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC;AACnE,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAgI7C,MAAM,IAAI,GACR,8FAA8F;IAC9F,yFAAyF;IACzF,0FAA0F;IAC1F,wFAAwF,CAAC;AAE3F,2FAA2F;AAC3F,SAAS,UAAU,CAAC,IAAY,EAAE,GAAW;IAC3C,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IAC5B,MAAM,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAChD,IAAI,IAAI,KAAK,WAAW;QAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IACxC,MAAM,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IAC/C,OAAO,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;AACzC,CAAC;AAED;;;;GAIG;AACH,SAAS,YAAY,CAAC,GAAW;IAC/B,MAAM,GAAG,GAAG,IAAI,GAAG,EAAwD,CAAC;IAC5E,MAAM,KAAK,GAAG,CAAC,IAAY,EAAE,EAAE;QAC7B,IAAI,CAAC,GAAG,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACtB,IAAI,CAAC,CAAC,EAAE,CAAC;YACP,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,EAAE,aAAa,EAAE,EAAE,EAAE,CAAC;YACrC,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;QACnB,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC,CAAC;IAEF,KAAK,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,KAAK,IAAI,EAAE,CAAC,EAAE,CAAC;QACpE,KAAK,MAAM,CAAC,IAAI,KAAK;YAAE,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACnD,CAAC;IACD,KAAK,MAAM,CAAC,IAAI,EAAE,IAAI,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,EAAE,CAAC,EAAE,CAAC;QACvE,KAAK,MAAM,CAAC,IAAI,UAAU,CAAC,IAAI,EAAE,GAAG,CAAC;YAAE,KAAK,CAAC,CAAC,CAAC,CAAC,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC3E,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,SAAS,SAAS,CAAC,IAAY;IAC7B,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IAC5B,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC;AAClG,CAAC;AAED,+DAA+D;AAC/D,MAAM,CAAC,KAAK,UAAU,eAAe,CACnC,GAAW,EACX,OAWI,EAAE;IAEN,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,IAAI,oBAAoB,CAAC;IACrD,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC;IACvC,MAAM,MAAM,GAAG,CAAC,IAAI,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,EAAE,MAAM,IAAI,EAAE,CAAC;IAC5F,MAAM,SAAS,GAAG,oBAAoB,EAAE,CAAC;IAEzC,MAAM,UAAU,GAAgB,EAAE,CAAC;IACnC,KAAK,MAAM,CAAC,IAAI,EAAE,UAAU,CAAC,IAAI,YAAY,CAAC,GAAG,CAAC,EAAE,CAAC;QACnD,MAAM,EAAE,QAAQ,EAAE,KAAK,EAAE,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;QAC5C,IAAI,IAAI,CAAC,QAAQ,IAAI,QAAQ,KAAK,IAAI,CAAC,QAAQ;YAAE,SAAS;QAC1D,MAAM,CAAC,GAA+B,GAAG,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QAE9D,IAAI,MAAM,GAAmB,IAAI,CAAC;QAClC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,KAAK,QAAQ,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;YACtD,IAAI,CAAC;gBACH,MAAM,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,QAAQ,EAAE,CAAC,EAAE,KAAK,CAAC,CAAC;YACtD,CAAC;YAAC,MAAM,CAAC;gBACP,MAAM,GAAG,IAAI,CAAC;YAChB,CAAC;QACH,CAAC;QAED,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,IAAI,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,eAAe,CAAC,QAAQ,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QAC/F,MAAM,KAAK,GAAG,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC;QACrC,MAAM,GAAG,GAAG,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC,MAAM,CAAC,GAAG,QAAQ,IAAI,KAAK,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;QACzE,MAAM,QAAQ,GAAG,WAAW,CAAC,IAAI,CAAC,CAAC;QACnC,MAAM,OAAO,GAAG,aAAa,CAAC,KAAK,IAAI,IAAI,EAAE,MAAM,CAAC,CAAC;QACrD,MAAM,IAAI,GAAG,OAAO,EAAE,GAAG,CAAC;QAC1B,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAE,IAAI,CAAC,CAAC,CAAY,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QAExF,uFAAuF;QACvF,4FAA4F;QAC5F,2FAA2F;QAC3F,oCAAoC;QACpC,MAAM,cAAc,GAAG,CAAC,IAAI,CAAC,CAAC,IAAI,KAAK,QAAQ,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO;YACtE,CAAC,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC;YACjE,CAAC,CAAC,IAAI,CAAC;QACT,qFAAqF;QACrF,kFAAkF;QAClF,mFAAmF;QACnF,sFAAsF;QACtF,uFAAuF;QACvF,wBAAwB;QACxB,MAAM,aAAa,GACjB,OAAO,IAAI,OAAO,CAAC,KAAK,KAAK,OAAO,CAAC,CAAC,CAAC,cAAc,OAAO,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,YAAY,CAAC;QACzF,MAAM,IAAI,GAAG,eAAe,CAAC,KAAK,IAAI,QAAQ,EAAE;YAC9C,cAAc;YACd,SAAS,EAAE;gBACT,aAAa,EAAE,GAAG,CAAC,gBAAgB,CAAC;gBACpC,mBAAmB,EAAE,GAAG,CAAC,cAAc,CAAC;gBACxC,uBAAuB,EAAE,GAAG,CAAC,kBAAkB,CAAC;gBAChD,IAAI,EAAE,aAAa;aACpB;SACF,CAAC,CAAC;QAEH,UAAU,CAAC,IAAI,CAAC;YACd,IAAI;YACJ,QAAQ;YACR,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC3B,KAAK,EAAE,UAAU,CAAC,KAAK;YACvB,aAAa,EAAE,UAAU,CAAC,aAAa;YACvC,qFAAqF;YACrF,oFAAoF;YACpF,MAAM,EAAE,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,YAAY,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI;YAChE,MAAM;YACN,eAAe,EAAE,OAAO,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,GAAG,CAAC,IAAI,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI;YAClF,MAAM,EAAE,OAAO;gBACb,CAAC,CAAC;oBACE,OAAO,EAAE,OAAO,CAAC,OAAO;oBACxB,KAAK,EAAE,OAAO,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI;oBAChD,KAAK,EAAE,OAAO,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI;oBAChD,QAAQ,EAAE,OAAO,CAAC,QAAQ;oBAC1B,SAAS,EAAE,OAAO,CAAC,SAAS;oBAC5B,YAAY,EAAE,OAAO,CAAC,YAAY;oBAClC,UAAU,EAAE,OAAO,CAAC,UAAU;iBAC/B;gBACH,CAAC,CAAC,IAAI;YACR,YAAY,EAAE,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,gBAAgB,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,IAAI;YAC7E,OAAO,EAAE;gBACP,IAAI,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,EAAE,KAAK,CAAC;gBACrC,mBAAmB,EAAE,KAAK,EAAE,mBAAmB,IAAI,CAAC;gBACpD,UAAU,EAAE,KAAK,EAAE,UAAU,IAAI,IAAI;gBACrC,mBAAmB,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,KAAK,EAAE,aAAa,IAAI,CAAC,CAAC,GAAG,KAAK,CAAC;gBACrE,kBAAkB,EAAE,KAAK,EAAE,kBAAkB,IAAI,CAAC;gBAClD,oBAAoB,EAAE,KAAK,EAAE,oBAAoB,IAAI,IAAI;gBACzD,eAAe,EAAE,OAAO,CAAC,kBAAkB,CAAC,IAAI,EAAE,KAAK,CAAC;aACzD;YACD,QAAQ,EAAE,GAAG;gBACX,CAAC,CAAC;oBACE,UAAU,EAAE,GAAG,CAAC,UAAU;oBAC1B,YAAY,EAAE,GAAG,CAAC,YAAY;oBAC9B,YAAY,EAAE,GAAG,CAAC,UAAU,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,cAAc,GAAG,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,IAAI;oBACzF,YAAY,EAAE,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,GAAG,CAAC,YAAY,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,IAAI;iBACjF;gBACH,CAAC,CAAC,IAAI;YACR,aAAa,EAAE,IAAI,CAAC,aAAa;YACjC,mBAAmB,EAAE,IAAI,CAAC,mBAAmB;YAC7C,eAAe,EAAE,IAAI,CAAC,eAAe;YACrC,qBAAqB,EAAE,IAAI,CAAC,qBAAqB;YACjD,GAAG,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC,CAAC,EAAE,qBAAqB,EAAE,IAAI,CAAC,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC5E,cAAc,EAAE,IAAI,CAAC,cAAc;YACnC,eAAe,EAAE,IAAI,CAAC,eAAe;YACrC,WAAW,EAAE,IAAI,CAAC,WAAW;YAC7B,aAAa,EAAE,OAAO,IAAI,EAAE,cAAc,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,CAAC,IAAI;YACrF,iBAAiB,EAAE,KAAK,CAAC,OAAO,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC,CAAC,CAAE,IAAI,CAAC,OAAoB,CAAC,CAAC,CAAC,EAAE;YACjF,MAAM,EAAE;gBACN,WAAW,EAAE,GAAG,CAAC,cAAc,CAAC;gBAChC,aAAa,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACrC,eAAe,EAAE,GAAG,CAAC,kBAAkB,CAAC;gBACxC,cAAc,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACtC,QAAQ,EAAE,GAAG,CAAC,WAAW,CAAC;gBAC1B,SAAS,EAAE,GAAG,CAAC,YAAY,CAAC;gBAC5B,aAAa,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACrC,eAAe,EAAE,GAAG,CAAC,mBAAmB,CAAC;gBACzC,wBAAwB,EAAE,GAAG,CAAC,8BAA8B,CAAC;gBAC7D,wBAAwB,EAAE,GAAG,CAAC,8BAA8B,CAAC;gBAC7D,WAAW,EAAE,GAAG,CAAC,cAAc,CAAC;gBAChC,SAAS,EAAE,GAAG,CAAC,YAAY,CAAC;aAC7B;YACD,UAAU,EAAE;gBACV,QAAQ,EAAE,QAAQ,CAAC,KAAK;gBACxB,aAAa,EAAE,QAAQ,CAAC,KAAK;gBAC7B,eAAe,EAAE,QAAQ,CAAC,OAAO,IAAI,EAAE;gBACvC,gBAAgB,EAAE,OAAO,CAAC,oBAAoB,CAAC,IAAI,CAAC;aACrD;SACF,CAAC,CAAC;IACL,CAAC;IAED,OAAO;QACL,YAAY,EAAE,IAAI,CAAC,GAAG,IAAI,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC,WAAW,EAAE;QACvD,eAAe,EAAE,GAAG,CAAC,OAAO,CAAC,OAAO,KAAK,IAAI;QAC7C,IAAI,EAAE,IAAI;QACV,UAAU;KACX,CAAC;AACJ,CAAC"}
@@ -7,6 +7,19 @@ export interface CircuitState {
7
7
  lastStatus?: number | undefined;
8
8
  pings: PingRecord[];
9
9
  quotaPercent?: number | null | undefined;
10
+ /**
11
+ * Credential faults (401/403), counted SEPARATELY from health failures.
12
+ *
13
+ * A revoked key is not a sick backend: recording it as a failure would open the breaker on
14
+ * a configuration problem and hide the 401 the operator has to see behind a "target
15
+ * unhealthy" skip, and recording it as a success would launder a permanently broken
16
+ * candidate into a healthy one. So it is neither — it is its own dimension, which is what
17
+ * lets a pool step over the candidate while `/candidates` still reports exactly why.
18
+ */
19
+ credentialFailures: number;
20
+ lastCredentialStatus?: number | undefined;
21
+ /** While in the future, this target is demoted (never dropped) as unusable-for-now. */
22
+ credentialFaultUntil: number;
10
23
  }
11
24
  /**
12
25
  * The ordering value for a target nothing has been measured about.
@@ -45,7 +58,27 @@ export declare class CircuitBreaker {
45
58
  status?: number;
46
59
  quotaPercent?: number | null;
47
60
  at?: number;
61
+ /**
62
+ * The provider's own `Retry-After`, in ms, when it published one. It replaces the flat
63
+ * cooldown guess: the provider is the only party that knows when it will serve again,
64
+ * and 2 minutes is as likely to be far too long (a 20-second groq TPM window) as far
65
+ * too short. Clamped, because the value is attacker-adjacent input.
66
+ */
67
+ retryAfterMs?: number | undefined;
48
68
  }): void;
69
+ /**
70
+ * Record a 401/403 — a credential fault, which is NOT health data.
71
+ *
72
+ * This deliberately does not touch `consecutiveFailures`, `cooldownUntil` or the ping
73
+ * history, so `isHealthy()` keeps answering the question it is named for and a revoked key
74
+ * never masquerades as a flaky backend (or vice versa). What it does do is mark the target
75
+ * unusable-for-now, which `hasCredentialFault()` reports and the router uses to DEMOTE it —
76
+ * so a pool stops paying a round-trip per request for a member that cannot authenticate,
77
+ * while `/candidates` still shows the operator the 401 and its count.
78
+ */
79
+ recordCredentialFault(target: ResolvedTarget | string, status: number, at?: number): void;
80
+ /** True while a recent 401/403 makes this target a last resort. Expires, so a fixed key recovers. */
81
+ hasCredentialFault(target: ResolvedTarget | string, now?: number): boolean;
49
82
  /**
50
83
  * Stability for a target whose behaviour has actually been observed, or
51
84
  * `null` when NOTHING has been measured.
@@ -3,6 +3,17 @@ const DEFAULT_COOLDOWN_MS = 60000; // 1 minute cooldown after consecutive failur
3
3
  const RATE_LIMIT_COOLDOWN_MS = 120000; // 2 minutes cooldown on 429
4
4
  const MAX_FAILURES_BEFORE_TRIP = 2;
5
5
  const MAX_PING_HISTORY = 10;
6
+ /**
7
+ * How long a 401/403 demotes a candidate before it is tried again.
8
+ *
9
+ * Deliberately finite: a rotated key must recover without restarting the proxy. Deliberately
10
+ * not zero: without it, a pool holding seven keyless members pays seven round-trips of 401
11
+ * latency on EVERY request before reaching a live one.
12
+ */
13
+ const CREDENTIAL_FAULT_TTL_MS = 300000; // 5 minutes
14
+ /** Clamp for a provider-supplied Retry-After, so a hostile or absurd value cannot park a target. */
15
+ const MIN_RETRY_AFTER_MS = 1000;
16
+ const MAX_RETRY_AFTER_MS = 900000; // 15 minutes
6
17
  /**
7
18
  * The ordering value for a target nothing has been measured about.
8
19
  *
@@ -49,6 +60,8 @@ export class CircuitBreaker {
49
60
  lastFailureTime: 0,
50
61
  cooldownUntil: 0,
51
62
  pings: [],
63
+ credentialFailures: 0,
64
+ credentialFaultUntil: 0,
52
65
  };
53
66
  this.states.set(key, fresh);
54
67
  return fresh;
@@ -87,18 +100,52 @@ export class CircuitBreaker {
87
100
  if (outcome.ok) {
88
101
  state.consecutiveFailures = 0;
89
102
  state.cooldownUntil = 0;
103
+ // A call that actually succeeded is proof the credential works now — that is the
104
+ // recovery path for a key fixed while the proxy is running.
105
+ state.credentialFailures = 0;
106
+ state.credentialFaultUntil = 0;
90
107
  return;
91
108
  }
92
109
  state.consecutiveFailures += 1;
93
110
  state.lastFailureTime = now;
94
- // HTTP 429 (Rate Limit) trips immediately for 2 minutes
111
+ const asked = outcome.retryAfterMs !== undefined
112
+ ? Math.min(MAX_RETRY_AFTER_MS, Math.max(MIN_RETRY_AFTER_MS, outcome.retryAfterMs))
113
+ : null;
114
+ // HTTP 429 (Rate Limit) trips immediately — for exactly as long as the provider asked,
115
+ // or 2 minutes when it did not say.
95
116
  if (outcome.status === 429) {
96
- state.cooldownUntil = now + RATE_LIMIT_COOLDOWN_MS;
117
+ state.cooldownUntil = now + (asked ?? RATE_LIMIT_COOLDOWN_MS);
118
+ }
119
+ else if (asked !== null) {
120
+ // A 503 with a Retry-After is the provider scheduling us; honour it on the first
121
+ // failure rather than waiting for a second one to trip the generic cooldown.
122
+ state.cooldownUntil = now + asked;
97
123
  }
98
124
  else if (state.consecutiveFailures >= MAX_FAILURES_BEFORE_TRIP) {
99
125
  state.cooldownUntil = now + DEFAULT_COOLDOWN_MS;
100
126
  }
101
127
  }
128
+ /**
129
+ * Record a 401/403 — a credential fault, which is NOT health data.
130
+ *
131
+ * This deliberately does not touch `consecutiveFailures`, `cooldownUntil` or the ping
132
+ * history, so `isHealthy()` keeps answering the question it is named for and a revoked key
133
+ * never masquerades as a flaky backend (or vice versa). What it does do is mark the target
134
+ * unusable-for-now, which `hasCredentialFault()` reports and the router uses to DEMOTE it —
135
+ * so a pool stops paying a round-trip per request for a member that cannot authenticate,
136
+ * while `/candidates` still shows the operator the 401 and its count.
137
+ */
138
+ recordCredentialFault(target, status, at = Date.now()) {
139
+ const state = this.getOrCreate(this.getKey(target));
140
+ state.credentialFailures += 1;
141
+ state.lastCredentialStatus = status;
142
+ state.credentialFaultUntil = at + CREDENTIAL_FAULT_TTL_MS;
143
+ }
144
+ /** True while a recent 401/403 makes this target a last resort. Expires, so a fixed key recovers. */
145
+ hasCredentialFault(target, now = Date.now()) {
146
+ const state = this.states.get(this.getKey(target));
147
+ return !!state && state.credentialFaultUntil > now;
148
+ }
102
149
  /**
103
150
  * Stability for a target whose behaviour has actually been observed, or
104
151
  * `null` when NOTHING has been measured.
@@ -1 +1 @@
1
- {"version":3,"file":"circuit-breaker.js","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,iBAAiB,EAAmB,MAAM,mBAAmB,CAAC;AAWvE,MAAM,mBAAmB,GAAG,KAAK,CAAC,CAAC,+CAA+C;AAClF,MAAM,sBAAsB,GAAG,MAAM,CAAC,CAAC,4BAA4B;AACnE,MAAM,wBAAwB,GAAG,CAAC,CAAC;AACnC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AAE5B;;;;;;;;;;;;GAYG;AACH,MAAM,CAAC,MAAM,oBAAoB,GAAG,EAAE,CAAC;AAEvC;;;;;;;;GAQG;AACH,SAAS,WAAW,CAAC,OAAyC;IAC5D,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS;QAAE,OAAO,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC;IACpE,IAAI,OAAO,CAAC,MAAM,IAAI,GAAG,IAAI,OAAO,CAAC,MAAM,GAAG,GAAG;QAAE,OAAO,KAAK,CAAC;IAChE,OAAO,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC;AAChC,CAAC;AAED,MAAM,OAAO,cAAc;IACjB,MAAM,GAAG,IAAI,GAAG,EAAwB,CAAC;IAEzC,MAAM,CAAC,MAA+B;QAC5C,IAAI,OAAO,MAAM,KAAK,QAAQ;YAAE,OAAO,MAAM,CAAC;QAC9C,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC,QAAQ,IAAI,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC;IAC/E,CAAC;IAEO,WAAW,CAAC,GAAW;QAC7B,MAAM,QAAQ,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACtC,IAAI,QAAQ;YAAE,OAAO,QAAQ,CAAC;QAC9B,MAAM,KAAK,GAAiB;YAC1B,mBAAmB,EAAE,CAAC;YACtB,eAAe,EAAE,CAAC;YAClB,aAAa,EAAE,CAAC;YAChB,KAAK,EAAE,EAAE;SACV,CAAC;QACF,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;QAC5B,OAAO,KAAK,CAAC;IACf,CAAC;IAED,8EAA8E;IAC9E,SAAS,CAAC,MAA+B,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;QACzD,MAAM,GAAG,GAAG,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAChC,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACnC,IAAI,CAAC,KAAK;YAAE,OAAO,IAAI,CAAC;QAExB,IAAI,KAAK,CAAC,aAAa,GAAG,GAAG,EAAE,CAAC;YAC9B,OAAO,KAAK,CAAC,CAAC,8BAA8B;QAC9C,CAAC;QAED,OAAO,IAAI,CAAC,CAAC,8BAA8B;IAC7C,CAAC;IAED;;;;;;;;;;OAUG;IACH,aAAa,CACX,MAA+B,EAC/B,OAAuG;QAEvG,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC;QACrC,MAAM,KAAK,GAAG,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QAEpD,IAAI,OAAO,CAAC,YAAY,KAAK,SAAS;YAAE,KAAK,CAAC,YAAY,GAAG,OAAO,CAAC,YAAY,CAAC;QAClF,KAAK,CAAC,UAAU,GAAG,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;QACpE,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,WAAW,CAAC,OAAO,CAAC,EAAE,SAAS,EAAE,GAAG,EAAE,CAAC,CAAC;QACxF,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,gBAAgB;YAAE,KAAK,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QAE/D,IAAI,OAAO,CAAC,EAAE,EAAE,CAAC;YACf,KAAK,CAAC,mBAAmB,GAAG,CAAC,CAAC;YAC9B,KAAK,CAAC,aAAa,GAAG,CAAC,CAAC;YACxB,OAAO;QACT,CAAC;QAED,KAAK,CAAC,mBAAmB,IAAI,CAAC,CAAC;QAC/B,KAAK,CAAC,eAAe,GAAG,GAAG,CAAC;QAE5B,wDAAwD;QACxD,IAAI,OAAO,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YAC3B,KAAK,CAAC,aAAa,GAAG,GAAG,GAAG,sBAAsB,CAAC;QACrD,CAAC;aAAM,IAAI,KAAK,CAAC,mBAAmB,IAAI,wBAAwB,EAAE,CAAC;YACjE,KAAK,CAAC,aAAa,GAAG,GAAG,GAAG,mBAAmB,CAAC;QAClD,CAAC;IACH,CAAC;IAED;;;;;;;;OAQG;IACH,oBAAoB,CAAC,MAA+B;QAClD,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACnD,IAAI,CAAC,KAAK,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,IAAI,CAAC;QACpD,MAAM,KAAK,GAAG,iBAAiB,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QAC7C,OAAO,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IAChC,CAAC;IAED,wEAAwE;IACxE,eAAe,CAAC,MAA+B;QAC7C,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACnD,OAAO,CAAC,CAAC,KAAK,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC;IAC3C,CAAC;IAED,qFAAqF;IACrF,2FAA2F;IAC3F,wFAAwF;IACxF,4FAA4F;IAC5F,2FAA2F;IAC3F,4FAA4F;IAC5F,kFAAkF;IAElF,mCAAmC;IACnC,QAAQ,CAAC,MAA+B;QACtC,OAAO,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;IAC9C,CAAC;IAED,sCAAsC;IACtC,YAAY;QACV,OAAO,IAAI,CAAC,MAAM,CAAC;IACrB,CAAC;IAED;;;;;;;;;;;OAWG;IACH,iBAAiB,CAAC,OAAyB,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;QAC3D,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;QAC9D,MAAM,UAAU,GAAG,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC;QAC1D,OAAO,CAAC,GAAG,UAAU,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;YACnC,MAAM,EAAE,GAAG,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC;YACxC,MAAM,EAAE,GAAG,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC;YACxC,MAAM,EAAE,GAAG,EAAE,IAAI,oBAAoB,CAAC;YACtC,MAAM,EAAE,GAAG,EAAE,IAAI,oBAAoB,CAAC;YACtC,IAAI,EAAE,KAAK,EAAE;gBAAE,OAAO,EAAE,GAAG,EAAE,CAAC;YAC9B,IAAI,CAAC,EAAE,KAAK,IAAI,CAAC,KAAK,CAAC,EAAE,KAAK,IAAI,CAAC;gBAAE,OAAO,EAAE,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YACjE,OAAO,CAAC,CAAC;QACX,CAAC,CAAC,CAAC;IACL,CAAC;IAED,gCAAgC;IAChC,KAAK;QACH,IAAI,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC;IACtB,CAAC;CACF;AAED,MAAM,CAAC,MAAM,oBAAoB,GAAG,IAAI,cAAc,EAAE,CAAC"}
1
+ {"version":3,"file":"circuit-breaker.js","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,iBAAiB,EAAmB,MAAM,mBAAmB,CAAC;AAwBvE,MAAM,mBAAmB,GAAG,KAAK,CAAC,CAAC,+CAA+C;AAClF,MAAM,sBAAsB,GAAG,MAAM,CAAC,CAAC,4BAA4B;AACnE,MAAM,wBAAwB,GAAG,CAAC,CAAC;AACnC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AAE5B;;;;;;GAMG;AACH,MAAM,uBAAuB,GAAG,MAAM,CAAC,CAAC,YAAY;AAEpD,oGAAoG;AACpG,MAAM,kBAAkB,GAAG,IAAI,CAAC;AAChC,MAAM,kBAAkB,GAAG,MAAM,CAAC,CAAC,aAAa;AAEhD;;;;;;;;;;;;GAYG;AACH,MAAM,CAAC,MAAM,oBAAoB,GAAG,EAAE,CAAC;AAEvC;;;;;;;;GAQG;AACH,SAAS,WAAW,CAAC,OAAyC;IAC5D,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS;QAAE,OAAO,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC;IACpE,IAAI,OAAO,CAAC,MAAM,IAAI,GAAG,IAAI,OAAO,CAAC,MAAM,GAAG,GAAG;QAAE,OAAO,KAAK,CAAC;IAChE,OAAO,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC;AAChC,CAAC;AAED,MAAM,OAAO,cAAc;IACjB,MAAM,GAAG,IAAI,GAAG,EAAwB,CAAC;IAEzC,MAAM,CAAC,MAA+B;QAC5C,IAAI,OAAO,MAAM,KAAK,QAAQ;YAAE,OAAO,MAAM,CAAC;QAC9C,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC,QAAQ,IAAI,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC;IAC/E,CAAC;IAEO,WAAW,CAAC,GAAW;QAC7B,MAAM,QAAQ,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACtC,IAAI,QAAQ;YAAE,OAAO,QAAQ,CAAC;QAC9B,MAAM,KAAK,GAAiB;YAC1B,mBAAmB,EAAE,CAAC;YACtB,eAAe,EAAE,CAAC;YAClB,aAAa,EAAE,CAAC;YAChB,KAAK,EAAE,EAAE;YACT,kBAAkB,EAAE,CAAC;YACrB,oBAAoB,EAAE,CAAC;SACxB,CAAC;QACF,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;QAC5B,OAAO,KAAK,CAAC;IACf,CAAC;IAED,8EAA8E;IAC9E,SAAS,CAAC,MAA+B,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;QACzD,MAAM,GAAG,GAAG,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAChC,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACnC,IAAI,CAAC,KAAK;YAAE,OAAO,IAAI,CAAC;QAExB,IAAI,KAAK,CAAC,aAAa,GAAG,GAAG,EAAE,CAAC;YAC9B,OAAO,KAAK,CAAC,CAAC,8BAA8B;QAC9C,CAAC;QAED,OAAO,IAAI,CAAC,CAAC,8BAA8B;IAC7C,CAAC;IAED;;;;;;;;;;OAUG;IACH,aAAa,CACX,MAA+B,EAC/B,OAaC;QAED,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC;QACrC,MAAM,KAAK,GAAG,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QAEpD,IAAI,OAAO,CAAC,YAAY,KAAK,SAAS;YAAE,KAAK,CAAC,YAAY,GAAG,OAAO,CAAC,YAAY,CAAC;QAClF,KAAK,CAAC,UAAU,GAAG,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;QACpE,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,WAAW,CAAC,OAAO,CAAC,EAAE,SAAS,EAAE,GAAG,EAAE,CAAC,CAAC;QACxF,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,gBAAgB;YAAE,KAAK,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QAE/D,IAAI,OAAO,CAAC,EAAE,EAAE,CAAC;YACf,KAAK,CAAC,mBAAmB,GAAG,CAAC,CAAC;YAC9B,KAAK,CAAC,aAAa,GAAG,CAAC,CAAC;YACxB,iFAAiF;YACjF,4DAA4D;YAC5D,KAAK,CAAC,kBAAkB,GAAG,CAAC,CAAC;YAC7B,KAAK,CAAC,oBAAoB,GAAG,CAAC,CAAC;YAC/B,OAAO;QACT,CAAC;QAED,KAAK,CAAC,mBAAmB,IAAI,CAAC,CAAC;QAC/B,KAAK,CAAC,eAAe,GAAG,GAAG,CAAC;QAE5B,MAAM,KAAK,GACT,OAAO,CAAC,YAAY,KAAK,SAAS;YAChC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,kBAAkB,EAAE,IAAI,CAAC,GAAG,CAAC,kBAAkB,EAAE,OAAO,CAAC,YAAY,CAAC,CAAC;YAClF,CAAC,CAAC,IAAI,CAAC;QAEX,uFAAuF;QACvF,oCAAoC;QACpC,IAAI,OAAO,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YAC3B,KAAK,CAAC,aAAa,GAAG,GAAG,GAAG,CAAC,KAAK,IAAI,sBAAsB,CAAC,CAAC;QAChE,CAAC;aAAM,IAAI,KAAK,KAAK,IAAI,EAAE,CAAC;YAC1B,iFAAiF;YACjF,6EAA6E;YAC7E,KAAK,CAAC,aAAa,GAAG,GAAG,GAAG,KAAK,CAAC;QACpC,CAAC;aAAM,IAAI,KAAK,CAAC,mBAAmB,IAAI,wBAAwB,EAAE,CAAC;YACjE,KAAK,CAAC,aAAa,GAAG,GAAG,GAAG,mBAAmB,CAAC;QAClD,CAAC;IACH,CAAC;IAED;;;;;;;;;OASG;IACH,qBAAqB,CAAC,MAA+B,EAAE,MAAc,EAAE,EAAE,GAAG,IAAI,CAAC,GAAG,EAAE;QACpF,MAAM,KAAK,GAAG,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACpD,KAAK,CAAC,kBAAkB,IAAI,CAAC,CAAC;QAC9B,KAAK,CAAC,oBAAoB,GAAG,MAAM,CAAC;QACpC,KAAK,CAAC,oBAAoB,GAAG,EAAE,GAAG,uBAAuB,CAAC;IAC5D,CAAC;IAED,qGAAqG;IACrG,kBAAkB,CAAC,MAA+B,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;QAClE,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACnD,OAAO,CAAC,CAAC,KAAK,IAAI,KAAK,CAAC,oBAAoB,GAAG,GAAG,CAAC;IACrD,CAAC;IAED;;;;;;;;OAQG;IACH,oBAAoB,CAAC,MAA+B;QAClD,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACnD,IAAI,CAAC,KAAK,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,IAAI,CAAC;QACpD,MAAM,KAAK,GAAG,iBAAiB,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QAC7C,OAAO,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IAChC,CAAC;IAED,wEAAwE;IACxE,eAAe,CAAC,MAA+B;QAC7C,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACnD,OAAO,CAAC,CAAC,KAAK,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC;IAC3C,CAAC;IAED,qFAAqF;IACrF,2FAA2F;IAC3F,wFAAwF;IACxF,4FAA4F;IAC5F,2FAA2F;IAC3F,4FAA4F;IAC5F,kFAAkF;IAElF,mCAAmC;IACnC,QAAQ,CAAC,MAA+B;QACtC,OAAO,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;IAC9C,CAAC;IAED,sCAAsC;IACtC,YAAY;QACV,OAAO,IAAI,CAAC,MAAM,CAAC;IACrB,CAAC;IAED;;;;;;;;;;;OAWG;IACH,iBAAiB,CAAC,OAAyB,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;QAC3D,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;QAC9D,MAAM,UAAU,GAAG,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC;QAC1D,OAAO,CAAC,GAAG,UAAU,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;YACnC,MAAM,EAAE,GAAG,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC;YACxC,MAAM,EAAE,GAAG,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC;YACxC,MAAM,EAAE,GAAG,EAAE,IAAI,oBAAoB,CAAC;YACtC,MAAM,EAAE,GAAG,EAAE,IAAI,oBAAoB,CAAC;YACtC,IAAI,EAAE,KAAK,EAAE;gBAAE,OAAO,EAAE,GAAG,EAAE,CAAC;YAC9B,IAAI,CAAC,EAAE,KAAK,IAAI,CAAC,KAAK,CAAC,EAAE,KAAK,IAAI,CAAC;gBAAE,OAAO,EAAE,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YACjE,OAAO,CAAC,CAAC;QACX,CAAC,CAAC,CAAC;IACL,CAAC;IAED,gCAAgC;IAChC,KAAK;QACH,IAAI,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC;IACtB,CAAC;CACF;AAED,MAAM,CAAC,MAAM,oBAAoB,GAAG,IAAI,cAAc,EAAE,CAAC"}
package/dist/cli.js CHANGED
@@ -658,6 +658,19 @@ export async function runOffload(arg) {
658
658
  function fmt(v, suffix = "") {
659
659
  return v === null || v === undefined ? "-" : `${v}${suffix}`;
660
660
  }
661
+ /**
662
+ * Mean latency of this proxy's OWN traffic to a target, with the call count that backs it.
663
+ *
664
+ * Deliberately not merged into the `p95` column: that one is the synthetic probe loop's measured
665
+ * p95, and an average over real requests is a different statistic from a different sample. Shown
666
+ * in seconds past 10s because "62605" reads as noise where "62.6s" reads as a decision.
667
+ */
668
+ function obsLatency(o) {
669
+ if (!o || o.avgLatencyMs === null || o.totalCalls === 0)
670
+ return "-";
671
+ const ms = o.avgLatencyMs;
672
+ return ms >= 10000 ? `${(ms / 1000).toFixed(1)}s` : `${ms}ms`;
673
+ }
661
674
  /**
662
675
  * How much to trust the strength number. A score from 5 leaderboards and a score from "nothing is
663
676
  * known, assume neutral" must never render identically.
@@ -698,6 +711,11 @@ export async function runCandidates() {
698
711
  "$/Mout".padEnd(8) +
699
712
  "verdict".padEnd(10) +
700
713
  "p95".padEnd(8) +
714
+ // Latency actually observed on this proxy's own traffic. The synthetic-probe p95 beside it is
715
+ // routinely blank, so the table could show a rank-1 pool member with NO latency signal at all
716
+ // while the proxy had already measured it at 60+ seconds per call — which is the difference
717
+ // between a pool that suits mechanical batch work and one that does not.
718
+ "obs".padEnd(8) +
701
719
  "quota".padEnd(7) +
702
720
  "breaker".padEnd(9) +
703
721
  "ctx";
@@ -706,7 +724,15 @@ export async function runCandidates() {
706
724
  const tags = [...c.pools, ...c.subagentTiers.map((t) => `@${t}`)].join(",") || "-";
707
725
  const live = c.listed === null ? "?" : c.listed ? "yes" : "NO";
708
726
  const ctx = c.contextLength ? `${Math.round(c.contextLength / 1000)}k` : "-";
709
- const breaker = c.breaker.open ? `OPEN ${Math.round(c.breaker.cooldownRemainingMs / 1000)}s` : "closed";
727
+ // A member that answers 401 on every call read "closed" here — the same as a healthy one —
728
+ // because a credential fault is deliberately not health data and so never reached the
729
+ // breaker's failure fields. It has its own axis now, and it is shown: the reason half a
730
+ // pool can be unusable while every row looks fine is precisely this cell.
731
+ const breaker = c.breaker.open
732
+ ? `OPEN ${Math.round(c.breaker.cooldownRemainingMs / 1000)}s`
733
+ : c.breaker.credentialFault
734
+ ? `AUTH ${c.breaker.lastCredentialStatus ?? ""}`.trim()
735
+ : "closed";
710
736
  process.stdout.write(c.spec.slice(0, 31).padEnd(32) +
711
737
  tags.slice(0, 23).padEnd(24) +
712
738
  // The scalar plus how well-evidenced it is: "83.3/4" = 4 published signals behind it,
@@ -722,6 +748,7 @@ export async function runCandidates() {
722
748
  : "-").padEnd(8) +
723
749
  (c.health?.verdict ?? "-").padEnd(10) +
724
750
  fmt(c.health?.p95Ms ?? null, "ms").padEnd(8) +
751
+ obsLatency(c.observed).padEnd(8) +
725
752
  fmt(c.quotaPercent, "%").padEnd(7) +
726
753
  breaker.padEnd(9) +
727
754
  // Provenance inline: "~" = another provider's figure for this model id. A NIM row must
@@ -733,9 +760,31 @@ export async function runCandidates() {
733
760
  }
734
761
  const fuzzy = view.candidates.filter((c) => c.capabilityMatch?.match === "fuzzy");
735
762
  const srcs = [...new Set(view.candidates.flatMap((c) => c.capabilitySources))].sort();
763
+ // Say out loud how much of the roster is currently unusable. A row-by-row table makes
764
+ // "5 of 14 members can actually serve" something the reader has to notice; a pool that is
765
+ // half dead is worth stating.
766
+ const noKey = view.candidates.filter((c) => !c.hasKey);
767
+ const authFault = view.candidates.filter((c) => c.breaker.credentialFault);
768
+ const cooling = view.candidates.filter((c) => c.breaker.open);
769
+ if (noKey.length || authFault.length || cooling.length) {
770
+ process.stdout.write(`\nNot first choice right now (of ${view.candidates.length} targets):\n` +
771
+ ` ${authFault.length} auth-faulted, ${cooling.length} cooling — DEMOTED: still tried, but only\n` +
772
+ " after every other candidate has failed on that request.\n" +
773
+ ` ${noKey.length} with no key set — DROPPED from a pool entirely (resolveTargets removes a\n` +
774
+ " declared-but-unset credential), so a pool's real size is smaller than its member count.\n" +
775
+ " ⚠ That includes free providers that would serve WITHOUT a key: declaring an authEnv and\n" +
776
+ " leaving it unset excludes them from every pool they belong to.\n" +
777
+ " `llm-relay keys` says whether a credential is good; `llm-relay pools --probe` is the\n" +
778
+ " only check that proves a member can actually serve.\n");
779
+ }
736
780
  process.stdout.write("\nColumns are independent — weigh them yourself. agentic/coding/BFCL/aider/arena are\n" +
737
781
  "capability from DIFFERENT leaderboards and they disagree; verdict/p95 are live behaviour;\n" +
738
782
  "quota/breaker/$ are what it costs to use right now. A blank cell means NOT MEASURED.\n" +
783
+ ` p95 = synthetic probe loop; obs = mean of this proxy's OWN requests. Different samples,\n` +
784
+ ` so they are not merged — and a high "obs" on a top-ranked member is worth seeing\n` +
785
+ ` before pointing bulk work at that pool (str ranks capability, never speed).\n` +
786
+ ` breaker: "OPEN 42s" = cooling after failures/429; "AUTH 401" = credential fault, demoted\n` +
787
+ ` until it is retried (expires, so a rotated key recovers with no restart).\n` +
739
788
  ` str = the one scalar pool ordering needs. "83.3/4" = 4 published signals behind it;\n` +
740
789
  ` "obs" = ranked on this proxy's own traffic, "neut" = nothing known.\n` +
741
790
  ` "~" on ctx/$ = another provider's figure for the same model id (this one publishes none);\n` +