llm-relay 0.12.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -2
- package/dist/backend.d.ts +41 -1
- package/dist/backend.js +94 -4
- package/dist/backend.js.map +1 -1
- package/dist/candidates.d.ts +13 -0
- package/dist/candidates.js +3 -0
- package/dist/candidates.js.map +1 -1
- package/dist/circuit-breaker.d.ts +33 -0
- package/dist/circuit-breaker.js +49 -2
- package/dist/circuit-breaker.js.map +1 -1
- package/dist/cli.js +50 -1
- package/dist/cli.js.map +1 -1
- package/dist/registry.d.ts +1 -1
- package/dist/server.d.ts +33 -0
- package/dist/server.js +199 -88
- package/dist/server.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -295,6 +295,25 @@ Claude Code subagent frontmatter (`model:`), which accepts a full model id but n
|
|
|
295
295
|
list. A pool is the indirection that gives those callers ranking and failover. `pool` is a
|
|
296
296
|
reserved provider name; configuring a provider called `pool` fails at load.
|
|
297
297
|
|
|
298
|
+
**What failover actually does** (both `/v1/messages` and `/v1/chat/completions`):
|
|
299
|
+
|
|
300
|
+
- **429 / 5xx / 400 / 404** → the candidate is recorded as a breaker failure and the next one is
|
|
301
|
+
tried. A `Retry-After` sets that candidate's cooldown for exactly as long as the provider asked.
|
|
302
|
+
- **401 / 403** → the next candidate is tried, but the fault is recorded on its own axis rather
|
|
303
|
+
than as ill health, so `llm-relay candidates` shows it as `AUTH 401` instead of hiding it. It
|
|
304
|
+
expires after 5 minutes, so a rotated key recovers with no restart.
|
|
305
|
+
- **A genuine client 4xx** (413, 422, …) → returned as-is. Every other candidate would reject it
|
|
306
|
+
identically.
|
|
307
|
+
- **Every candidate failed** → the last real upstream error, not a synthesized one.
|
|
308
|
+
|
|
309
|
+
Responses carry **`x-llm-relay-served-by`**: the deployment that served, or on an error every
|
|
310
|
+
deployment that was tried, in order.
|
|
311
|
+
|
|
312
|
+
⚠ **A pool routes to fewer members than it lists** when some declare an `authEnv` that is unset —
|
|
313
|
+
those are dropped before ranking, so a 14-member pool can resolve to 7 and the config's *tenth*
|
|
314
|
+
entry can legitimately be the one that answers. `llm-relay candidates` reports the count.
|
|
315
|
+
Background: [docs/pool-failover.md](docs/pool-failover.md).
|
|
316
|
+
|
|
298
317
|
### Quieting the onboarding nudge (`leave_me_alone`)
|
|
299
318
|
|
|
300
319
|
`llm-relay onboard` walks every known provider and prompts for the keys you are missing. For a
|
|
@@ -518,7 +537,7 @@ silently at request time.
|
|
|
518
537
|
|
|
519
538
|
### Discovery endpoint (`GET /registry`) — for a dispatcher
|
|
520
539
|
|
|
521
|
-
For a caller that does its own selection (
|
|
540
|
+
For a caller that does its own selection (an external dispatcher weighing
|
|
522
541
|
quota / rate limits / token budget), `GET http://127.0.0.1:8791/registry` returns one
|
|
523
542
|
coherent JSON view:
|
|
524
543
|
|
|
@@ -629,7 +648,7 @@ headroom's backend-agnostic memory/learn layer. So stack it for context-fit, not
|
|
|
629
648
|
|
|
630
649
|
## Design
|
|
631
650
|
|
|
632
|
-
Consumers (
|
|
651
|
+
Consumers (an external dispatcher, plain `claude` CLI) point `ANTHROPIC_BASE_URL` at this proxy; it validates one backend per request. Target *selection* / token-prediction is a separate concern (the router/auditor), deliberately not here. For architecture, invariants, and the script inventory, see [CLAUDE.md](CLAUDE.md).
|
|
633
652
|
|
|
634
653
|
## Dev
|
|
635
654
|
|
package/dist/backend.d.ts
CHANGED
|
@@ -13,6 +13,25 @@ export declare const ERROR_ORIGIN_HEADER = "x-llm-relay-error-origin";
|
|
|
13
13
|
export type ErrorOrigin = "upstream" | "local";
|
|
14
14
|
/** Read the origin marker off a Response, when it carries one. */
|
|
15
15
|
export declare function errorOrigin(res: Response): ErrorOrigin | null;
|
|
16
|
+
/**
|
|
17
|
+
* Response header naming the deployment that actually answered — the (provider, model) left
|
|
18
|
+
* standing after pool expansion, benchmark ranking, breaker demotion and failover.
|
|
19
|
+
*
|
|
20
|
+
* A pool request's most basic debugging question is "who served this?", and until now the only
|
|
21
|
+
* way to answer it was to correlate timestamps against the proxy's own log. When every candidate
|
|
22
|
+
* fails it carries the list that was tried instead, so an exhausted pool is self-describing.
|
|
23
|
+
*/
|
|
24
|
+
export declare const SERVED_BY_HEADER = "x-llm-relay-served-by";
|
|
25
|
+
/**
|
|
26
|
+
* The provider's `Retry-After` in milliseconds, or null.
|
|
27
|
+
*
|
|
28
|
+
* Accepts both RFC 9110 forms — delta-seconds and an HTTP-date — because providers use both
|
|
29
|
+
* (groq sends seconds, some CDNs in front of a provider send a date). A date in the past, a
|
|
30
|
+
* negative delta or an unparseable value yields null rather than 0: "the provider said nothing
|
|
31
|
+
* usable" and "the provider said retry immediately" call for different cooldowns, and treating
|
|
32
|
+
* garbage as 0 would silently disable the backoff this exists to honour.
|
|
33
|
+
*/
|
|
34
|
+
export declare function parseRetryAfterMs(value: string | null | undefined, now?: number): number | null;
|
|
16
35
|
/**
|
|
17
36
|
* Fetch the resolved provider target and return an ANTHROPIC-shaped `Response`,
|
|
18
37
|
* regardless of the backend's native wire format. For kind="anthropic" this is a
|
|
@@ -32,13 +51,34 @@ export declare function fetchBackend(target: ResolvedTarget, args: {
|
|
|
32
51
|
}, fetchFn?: typeof fetch): Promise<Response>;
|
|
33
52
|
/** Map a non-streaming OpenAI chat completion into an Anthropic message. */
|
|
34
53
|
export declare function openAiResponseToAnthropic(j: Record<string, unknown>, model: string): object;
|
|
54
|
+
/**
|
|
55
|
+
* Coerce an upstream error body into the OpenAI error envelope — WITHOUT rewriting one that
|
|
56
|
+
* already conforms.
|
|
57
|
+
*
|
|
58
|
+
* The OpenAI front promises "OpenAI in, OpenAI out", but on the error path it returned whatever
|
|
59
|
+
* shape the provider chose. Gemini wraps its error in a JSON ARRAY (`[{"error":{…}}]`); a client
|
|
60
|
+
* reading `response.choices[0]` gets `undefined` from that and reports a malformed completion,
|
|
61
|
+
* so a plain 429 surfaces as "the model returned garbage" — which is exactly how a rate limit
|
|
62
|
+
* cost two days of debugging on the caller's side.
|
|
63
|
+
*
|
|
64
|
+
* Rules, in order:
|
|
65
|
+
* - already `{error:{…}}` → returned BYTE-EXACT. A conforming provider's message, code and
|
|
66
|
+
* type are its own to state, and rewriting them would lose detail.
|
|
67
|
+
* - `[{error:{…}}, …]` → unwrapped to the element. Same fields, now at the top level.
|
|
68
|
+
* - anything else (HTML, text,
|
|
69
|
+
* a bare string, empty) → wrapped, with the original preserved as the message.
|
|
70
|
+
*
|
|
71
|
+
* Returns null when the body is already conforming, so the caller can stream the original bytes
|
|
72
|
+
* rather than re-serialize them.
|
|
73
|
+
*/
|
|
74
|
+
export declare function normalizeOpenAiErrorBody(body: string, status: number): string | null;
|
|
35
75
|
/**
|
|
36
76
|
* OpenAI-compatible FRONT: an OpenAI `/chat/completions` request comes in, its `model`
|
|
37
77
|
* has already been resolved to a provider target by namespace/tier routing. For an
|
|
38
78
|
* openai-kind target this is a routing reverse-proxy — rewrite `model` to the backend
|
|
39
79
|
* id, inject the backend key, and stream the upstream OpenAI response straight back
|
|
40
80
|
* (OpenAI in, OpenAI out — no translation). This is the transport a dispatcher (e.g.
|
|
41
|
-
*
|
|
81
|
+
* an external dispatcher) consumes to reach many backends behind one endpoint.
|
|
42
82
|
*
|
|
43
83
|
* anthropic-kind targets are not served on the OpenAI front (they need OpenAI↔Anthropic
|
|
44
84
|
* translation and are not the dispatcher use case) — a clean 400, never a mistranslation.
|
package/dist/backend.js
CHANGED
|
@@ -15,6 +15,41 @@ export function errorOrigin(res) {
|
|
|
15
15
|
const v = res.headers.get(ERROR_ORIGIN_HEADER);
|
|
16
16
|
return v === "upstream" || v === "local" ? v : null;
|
|
17
17
|
}
|
|
18
|
+
/**
|
|
19
|
+
* Response header naming the deployment that actually answered — the (provider, model) left
|
|
20
|
+
* standing after pool expansion, benchmark ranking, breaker demotion and failover.
|
|
21
|
+
*
|
|
22
|
+
* A pool request's most basic debugging question is "who served this?", and until now the only
|
|
23
|
+
* way to answer it was to correlate timestamps against the proxy's own log. When every candidate
|
|
24
|
+
* fails it carries the list that was tried instead, so an exhausted pool is self-describing.
|
|
25
|
+
*/
|
|
26
|
+
export const SERVED_BY_HEADER = "x-llm-relay-served-by";
|
|
27
|
+
/**
|
|
28
|
+
* The provider's `Retry-After` in milliseconds, or null.
|
|
29
|
+
*
|
|
30
|
+
* Accepts both RFC 9110 forms — delta-seconds and an HTTP-date — because providers use both
|
|
31
|
+
* (groq sends seconds, some CDNs in front of a provider send a date). A date in the past, a
|
|
32
|
+
* negative delta or an unparseable value yields null rather than 0: "the provider said nothing
|
|
33
|
+
* usable" and "the provider said retry immediately" call for different cooldowns, and treating
|
|
34
|
+
* garbage as 0 would silently disable the backoff this exists to honour.
|
|
35
|
+
*/
|
|
36
|
+
export function parseRetryAfterMs(value, now = Date.now()) {
|
|
37
|
+
if (typeof value !== "string")
|
|
38
|
+
return null;
|
|
39
|
+
const raw = value.trim();
|
|
40
|
+
if (raw.length === 0)
|
|
41
|
+
return null;
|
|
42
|
+
// delta-seconds — integer per the RFC, but providers do emit fractions ("20.45").
|
|
43
|
+
if (/^\d+(\.\d+)?$/.test(raw)) {
|
|
44
|
+
const ms = Math.round(Number(raw) * 1000);
|
|
45
|
+
return Number.isFinite(ms) && ms >= 0 ? ms : null;
|
|
46
|
+
}
|
|
47
|
+
const at = Date.parse(raw);
|
|
48
|
+
if (Number.isNaN(at))
|
|
49
|
+
return null;
|
|
50
|
+
const delta = at - now;
|
|
51
|
+
return delta > 0 ? delta : null;
|
|
52
|
+
}
|
|
18
53
|
/**
|
|
19
54
|
* Fetch the resolved provider target and return an ANTHROPIC-shaped `Response`,
|
|
20
55
|
* regardless of the backend's native wire format. For kind="anthropic" this is a
|
|
@@ -88,7 +123,12 @@ export async function fetchBackend(target, args, fetchFn = fetch) {
|
|
|
88
123
|
? ` — model "${target.model}" is not served by provider "${target.provider}" (a model can be listed in /models and still 404 here)`
|
|
89
124
|
: "";
|
|
90
125
|
// The provider really answered with this status — the body is reworded, the origin is not.
|
|
91
|
-
|
|
126
|
+
// `Retry-After` is carried onto the synthesized error: this response is a NEW Response, so
|
|
127
|
+
// without this the one header stating when the provider will serve again was destroyed here,
|
|
128
|
+
// and neither the breaker's cooldown nor the client's backoff could ever honour it.
|
|
129
|
+
return anthropicError(res.status, `openai backend HTTP ${res.status}${hint}: ${body.slice(0, 300)}`, "upstream", {
|
|
130
|
+
...retryAfterHeader(res.headers),
|
|
131
|
+
});
|
|
92
132
|
}
|
|
93
133
|
if (args.wantsStream && res.body) {
|
|
94
134
|
const anthStream = handleUniversalStreamRequest(res.body, "openai", "anthropic");
|
|
@@ -138,10 +178,15 @@ export function openAiResponseToAnthropic(j, model) {
|
|
|
138
178
|
usage: { input_tokens: usage.prompt_tokens ?? 0, output_tokens: usage.completion_tokens ?? 0 },
|
|
139
179
|
};
|
|
140
180
|
}
|
|
141
|
-
|
|
181
|
+
/** The upstream's `Retry-After`, as a header object to spread, or `{}` when it sent none. */
|
|
182
|
+
function retryAfterHeader(hh) {
|
|
183
|
+
const v = hh.get("retry-after");
|
|
184
|
+
return v ? { "retry-after": v } : {};
|
|
185
|
+
}
|
|
186
|
+
function anthropicError(status, message, origin, extra = {}) {
|
|
142
187
|
return new Response(JSON.stringify({ type: "error", error: { type: "api_error", message } }), {
|
|
143
188
|
status,
|
|
144
|
-
headers: { "content-type": "application/json", [ERROR_ORIGIN_HEADER]: origin },
|
|
189
|
+
headers: { "content-type": "application/json", [ERROR_ORIGIN_HEADER]: origin, ...extra },
|
|
145
190
|
});
|
|
146
191
|
}
|
|
147
192
|
function openaiError(status, message, origin) {
|
|
@@ -150,13 +195,58 @@ function openaiError(status, message, origin) {
|
|
|
150
195
|
headers: { "content-type": "application/json", [ERROR_ORIGIN_HEADER]: origin },
|
|
151
196
|
});
|
|
152
197
|
}
|
|
198
|
+
/**
|
|
199
|
+
* Coerce an upstream error body into the OpenAI error envelope — WITHOUT rewriting one that
|
|
200
|
+
* already conforms.
|
|
201
|
+
*
|
|
202
|
+
* The OpenAI front promises "OpenAI in, OpenAI out", but on the error path it returned whatever
|
|
203
|
+
* shape the provider chose. Gemini wraps its error in a JSON ARRAY (`[{"error":{…}}]`); a client
|
|
204
|
+
* reading `response.choices[0]` gets `undefined` from that and reports a malformed completion,
|
|
205
|
+
* so a plain 429 surfaces as "the model returned garbage" — which is exactly how a rate limit
|
|
206
|
+
* cost two days of debugging on the caller's side.
|
|
207
|
+
*
|
|
208
|
+
* Rules, in order:
|
|
209
|
+
* - already `{error:{…}}` → returned BYTE-EXACT. A conforming provider's message, code and
|
|
210
|
+
* type are its own to state, and rewriting them would lose detail.
|
|
211
|
+
* - `[{error:{…}}, …]` → unwrapped to the element. Same fields, now at the top level.
|
|
212
|
+
* - anything else (HTML, text,
|
|
213
|
+
* a bare string, empty) → wrapped, with the original preserved as the message.
|
|
214
|
+
*
|
|
215
|
+
* Returns null when the body is already conforming, so the caller can stream the original bytes
|
|
216
|
+
* rather than re-serialize them.
|
|
217
|
+
*/
|
|
218
|
+
export function normalizeOpenAiErrorBody(body, status) {
|
|
219
|
+
let parsed;
|
|
220
|
+
try {
|
|
221
|
+
parsed = JSON.parse(body);
|
|
222
|
+
}
|
|
223
|
+
catch {
|
|
224
|
+
const message = body.trim().slice(0, 2000) || `upstream returned HTTP ${status} with an empty body`;
|
|
225
|
+
return JSON.stringify({ error: { message, type: "upstream_error", code: status } });
|
|
226
|
+
}
|
|
227
|
+
const hasError = (v) => typeof v === "object" && v !== null && typeof v.error === "object" && v.error !== null;
|
|
228
|
+
if (hasError(parsed))
|
|
229
|
+
return null; // conforms — do not touch it
|
|
230
|
+
if (Array.isArray(parsed)) {
|
|
231
|
+
const wrapped = parsed.find(hasError);
|
|
232
|
+
if (wrapped)
|
|
233
|
+
return JSON.stringify(wrapped);
|
|
234
|
+
}
|
|
235
|
+
return JSON.stringify({
|
|
236
|
+
error: {
|
|
237
|
+
message: body.trim().slice(0, 2000) || `upstream returned HTTP ${status} with an empty body`,
|
|
238
|
+
type: "upstream_error",
|
|
239
|
+
code: status,
|
|
240
|
+
},
|
|
241
|
+
});
|
|
242
|
+
}
|
|
153
243
|
/**
|
|
154
244
|
* OpenAI-compatible FRONT: an OpenAI `/chat/completions` request comes in, its `model`
|
|
155
245
|
* has already been resolved to a provider target by namespace/tier routing. For an
|
|
156
246
|
* openai-kind target this is a routing reverse-proxy — rewrite `model` to the backend
|
|
157
247
|
* id, inject the backend key, and stream the upstream OpenAI response straight back
|
|
158
248
|
* (OpenAI in, OpenAI out — no translation). This is the transport a dispatcher (e.g.
|
|
159
|
-
*
|
|
249
|
+
* an external dispatcher) consumes to reach many backends behind one endpoint.
|
|
160
250
|
*
|
|
161
251
|
* anthropic-kind targets are not served on the OpenAI front (they need OpenAI↔Anthropic
|
|
162
252
|
* translation and are not the dispatcher use case) — a clean 400, never a mistranslation.
|
package/dist/backend.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"backend.js","sourceRoot":"","sources":["../src/backend.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,yBAAyB,EAAE,4BAA4B,EAAE,MAAM,YAAY,CAAC;AAErF,OAAO,EAAE,aAAa,EAAE,kBAAkB,EAAE,MAAM,gBAAgB,CAAC;AAEnE;;;;;;;;GAQG;AACH,MAAM,CAAC,MAAM,mBAAmB,GAAG,0BAA0B,CAAC;AAK9D,kEAAkE;AAClE,MAAM,UAAU,WAAW,CAAC,GAAa;IACvC,MAAM,CAAC,GAAG,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,CAAC,CAAC;IAC/C,OAAO,CAAC,KAAK,UAAU,IAAI,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;AACtD,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,MAAsB,EACtB,IAQC,EACD,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;QAChC,MAAM,IAAI,GAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,OAAO,EAAE,IAAI,CAAC,gBAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC;QACvG,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM;YAAE,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC;QAChD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,oBAAoB;IACpB,sFAAsF;IACtF,sFAAsF;IACtF,IAAI,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;IAC3B,IAAI,CAAC;QACH,OAAO,GAAG,MAAM,kBAAkB,CAAC,OAAO,CAAC,CAAC;IAC9C,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,sFAAsF;QACtF,sFAAsF;QACtF,IAAI,CAAC,YAAY,aAAa;YAAE,OAAO,cAAc,CAAC,GAAG,EAAE,cAAc,CAAC,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;QAC/F,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC7F,CAAC;IAED,IAAI,UAAmC,CAAC;IACxC,IAAI,CAAC;QACH,UAAU,GAAG,yBAAyB,CAAC,WAAW,EAAE,QAAQ,EAAE,CAAC,OAAO,IAAI,EAAE,CAAU,CAA4B,CAAC;IACrH,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC7F,CAAC;IACD,UAAU,CAAC,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC;IAChC,UAAU,CAAC,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC;IACrC,sFAAsF;IACtF,qFAAqF;IACrF,mFAAmF;IACnF,IAAI,IAAI,CAAC,WAAW;QAAE,UAAU,CAAC,cAAc,GAAG,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC;IAE1E,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,IAA6B,EAAE,EAAE,CAC7C,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QACzC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;QAC1B,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;IAEL,IAAI,GAAG,GAAG,MAAM,IAAI,CAAC,UAAU,CAAC,CAAC;IAEjC,oFAAoF;IACpF,6EAA6E;IAC7E,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,UAAU,CAAC,cAAc,IAAI,CAAC,GAAG,CAAC,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,CAAC,EAAE,CAAC;QACvF,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,GAAG,YAAY,EAAE,GAAG,UAAU,CAAC;QAC9D,GAAG,GAAG,MAAM,IAAI,CAAC,YAAY,CAAC,CAAC;IACjC,CAAC;IAED,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;QACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC;QAC9B,6EAA6E;QAC7E,sEAAsE;QACtE,2DAA2D;QAC3D,MAAM,IAAI,GACR,GAAG,CAAC,MAAM,KAAK,GAAG;YAChB,CAAC,CAAC,aAAa,MAAM,CAAC,KAAK,gCAAgC,MAAM,CAAC,QAAQ,yDAAyD;YACnI,CAAC,CAAC,EAAE,CAAC;QACT,2FAA2F;QAC3F,OAAO,cAAc,CAAC,GAAG,CAAC,MAAM,EAAE,uBAAuB,GAAG,CAAC,MAAM,GAAG,IAAI,KAAK,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,EAAE,UAAU,CAAC,CAAC;
|
|
1
|
+
{"version":3,"file":"backend.js","sourceRoot":"","sources":["../src/backend.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,yBAAyB,EAAE,4BAA4B,EAAE,MAAM,YAAY,CAAC;AAErF,OAAO,EAAE,aAAa,EAAE,kBAAkB,EAAE,MAAM,gBAAgB,CAAC;AAEnE;;;;;;;;GAQG;AACH,MAAM,CAAC,MAAM,mBAAmB,GAAG,0BAA0B,CAAC;AAK9D,kEAAkE;AAClE,MAAM,UAAU,WAAW,CAAC,GAAa;IACvC,MAAM,CAAC,GAAG,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,CAAC,CAAC;IAC/C,OAAO,CAAC,KAAK,UAAU,IAAI,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;AACtD,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,gBAAgB,GAAG,uBAAuB,CAAC;AAExD;;;;;;;;GAQG;AACH,MAAM,UAAU,iBAAiB,CAAC,KAAgC,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;IAClF,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,IAAI,CAAC;IAC3C,MAAM,GAAG,GAAG,KAAK,CAAC,IAAI,EAAE,CAAC;IACzB,IAAI,GAAG,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,IAAI,CAAC;IAElC,kFAAkF;IAClF,IAAI,eAAe,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QAC9B,MAAM,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,IAAI,CAAC,CAAC;QAC1C,OAAO,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IACpD,CAAC;IAED,MAAM,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAC3B,IAAI,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;QAAE,OAAO,IAAI,CAAC;IAClC,MAAM,KAAK,GAAG,EAAE,GAAG,GAAG,CAAC;IACvB,OAAO,KAAK,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC;AAClC,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,MAAsB,EACtB,IAQC,EACD,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;QAChC,MAAM,IAAI,GAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,OAAO,EAAE,IAAI,CAAC,gBAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC;QACvG,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM;YAAE,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC;QAChD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,oBAAoB;IACpB,sFAAsF;IACtF,sFAAsF;IACtF,IAAI,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;IAC3B,IAAI,CAAC;QACH,OAAO,GAAG,MAAM,kBAAkB,CAAC,OAAO,CAAC,CAAC;IAC9C,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,sFAAsF;QACtF,sFAAsF;QACtF,IAAI,CAAC,YAAY,aAAa;YAAE,OAAO,cAAc,CAAC,GAAG,EAAE,cAAc,CAAC,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;QAC/F,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC7F,CAAC;IAED,IAAI,UAAmC,CAAC;IACxC,IAAI,CAAC;QACH,UAAU,GAAG,yBAAyB,CAAC,WAAW,EAAE,QAAQ,EAAE,CAAC,OAAO,IAAI,EAAE,CAAU,CAA4B,CAAC;IACrH,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC7F,CAAC;IACD,UAAU,CAAC,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC;IAChC,UAAU,CAAC,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC;IACrC,sFAAsF;IACtF,qFAAqF;IACrF,mFAAmF;IACnF,IAAI,IAAI,CAAC,WAAW;QAAE,UAAU,CAAC,cAAc,GAAG,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC;IAE1E,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,IAA6B,EAAE,EAAE,CAC7C,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QACzC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;QAC1B,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;IAEL,IAAI,GAAG,GAAG,MAAM,IAAI,CAAC,UAAU,CAAC,CAAC;IAEjC,oFAAoF;IACpF,6EAA6E;IAC7E,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,UAAU,CAAC,cAAc,IAAI,CAAC,GAAG,CAAC,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,CAAC,EAAE,CAAC;QACvF,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,GAAG,YAAY,EAAE,GAAG,UAAU,CAAC;QAC9D,GAAG,GAAG,MAAM,IAAI,CAAC,YAAY,CAAC,CAAC;IACjC,CAAC;IAED,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;QACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC;QAC9B,6EAA6E;QAC7E,sEAAsE;QACtE,2DAA2D;QAC3D,MAAM,IAAI,GACR,GAAG,CAAC,MAAM,KAAK,GAAG;YAChB,CAAC,CAAC,aAAa,MAAM,CAAC,KAAK,gCAAgC,MAAM,CAAC,QAAQ,yDAAyD;YACnI,CAAC,CAAC,EAAE,CAAC;QACT,2FAA2F;QAC3F,2FAA2F;QAC3F,6FAA6F;QAC7F,oFAAoF;QACpF,OAAO,cAAc,CAAC,GAAG,CAAC,MAAM,EAAE,uBAAuB,GAAG,CAAC,MAAM,GAAG,IAAI,KAAK,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,EAAE,UAAU,EAAE;YAC/G,GAAG,gBAAgB,CAAC,GAAG,CAAC,OAAO,CAAC;SACjC,CAAC,CAAC;IACL,CAAC;IAED,IAAI,IAAI,CAAC,WAAW,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;QACjC,MAAM,UAAU,GAAG,4BAA4B,CAAC,GAAG,CAAC,IAAI,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC;QACjF,OAAO,IAAI,QAAQ,CAAC,UAAU,EAAE,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,mBAAmB,EAAE,EAAE,CAAC,CAAC;IAC5G,CAAC;IAED,IAAI,aAAqB,CAAC;IAC1B,IAAI,CAAC;QACH,aAAa,GAAG,yBAAyB,CAAC,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAA4B,EAAE,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC;IAC/G,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,kFAAkF;QAClF,kFAAkF;QAClF,OAAO,cAAc,CAAC,GAAG,EAAE,gCAAiC,CAAW,CAAC,OAAO,EAAE,EAAE,OAAO,CAAC,CAAC;IAC9F,CAAC;IACD,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,EAAE,EAAE,MAAM,EAAE,GAAG,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,EAAE,CAAC,CAAC;AACvH,CAAC;AAED,4EAA4E;AAC5E,MAAM,UAAU,yBAAyB,CAAC,CAA0B,EAAE,KAAa;IACjF,MAAM,MAAM,GAAI,CAAC,CAAC,OAAsD,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;IACpF,MAAM,GAAG,GAAI,MAAM,CAAC,OAA+C,IAAI,EAAE,CAAC;IAC1E,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,IAAI,OAAO,GAAG,CAAC,OAAO,KAAK,QAAQ,IAAI,GAAG,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;IACjH,MAAM,SAAS,GAAI,GAAG,CAAC,UAAyD,IAAI,EAAE,CAAC;IACvF,KAAK,MAAM,EAAE,IAAI,SAAS,EAAE,CAAC;QAC3B,MAAM,EAAE,GAAI,EAAE,CAAC,QAAgD,IAAI,EAAE,CAAC;QACtE,IAAI,KAAc,CAAC;QACnB,IAAI,CAAC;YAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAE,EAAE,CAAC,SAAoB,IAAI,IAAI,CAAC,CAAC;QAAC,CAAC;QAAC,MAAM,CAAC;YAAC,KAAK,GAAG,EAAE,CAAC,SAAS,IAAI,EAAE,CAAC;QAAC,CAAC;QACnG,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,UAAU,EAAE,EAAE,EAAG,EAAE,CAAC,EAAa,IAAI,IAAI,EAAE,IAAI,EAAG,EAAE,CAAC,IAAe,IAAI,EAAE,EAAE,KAAK,EAAE,CAAC,CAAC;IAC5G,CAAC;IACD,MAAM,MAAM,GAAG,MAAM,CAAC,aAAmC,CAAC;IAC1D,MAAM,UAAU,GACd,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,KAAK,QAAQ,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,IAAI,UAAU,CAAC;IACjI,MAAM,KAAK,GAAI,CAAC,CAAC,KAA4C,IAAI,EAAE,CAAC;IACpE,OAAO;QACL,EAAE,EAAG,CAAC,CAAC,EAAa,IAAI,gBAAgB;QACxC,IAAI,EAAE,SAAS;QACf,IAAI,EAAE,WAAW;QACjB,KAAK,EAAE,KAAK,IAAI,CAAE,CAAC,CAAC,KAAgB,IAAI,EAAE,CAAC;QAC3C,OAAO;QACP,WAAW,EAAE,UAAU;QACvB,aAAa,EAAE,IAAI;QACnB,KAAK,EAAE,EAAE,YAAY,EAAE,KAAK,CAAC,aAAa,IAAI,CAAC,EAAE,aAAa,EAAE,KAAK,CAAC,iBAAiB,IAAI,CAAC,EAAE;KAC/F,CAAC;AACJ,CAAC;AAED,6FAA6F;AAC7F,SAAS,gBAAgB,CAAC,EAAW;IACnC,MAAM,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC;IAChC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;AACvC,CAAC;AAED,SAAS,cAAc,CACrB,MAAc,EACd,OAAe,EACf,MAAmB,EACnB,QAAgC,EAAE;IAElC,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE,IAAI,EAAE,WAAW,EAAE,OAAO,EAAE,EAAE,CAAC,EAAE;QAC5F,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,mBAAmB,CAAC,EAAE,MAAM,EAAE,GAAG,KAAK,EAAE;KACzF,CAAC,CAAC;AACL,CAAC;AAED,SAAS,WAAW,CAAC,MAAc,EAAE,OAAe,EAAE,MAAmB;IACvE,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,uBAAuB,EAAE,EAAE,CAAC,EAAE;QACzF,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,mBAAmB,CAAC,EAAE,MAAM,EAAE;KAC/E,CAAC,CAAC;AACL,CAAC;AAED;;;;;;;;;;;;;;;;;;;GAmBG;AACH,MAAM,UAAU,wBAAwB,CAAC,IAAY,EAAE,MAAc;IACnE,IAAI,MAAe,CAAC;IACpB,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAC5B,CAAC;IAAC,MAAM,CAAC;QACP,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,0BAA0B,MAAM,qBAAqB,CAAC;QACpG,OAAO,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,gBAAgB,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE,CAAC,CAAC;IACtF,CAAC;IAED,MAAM,QAAQ,GAAG,CAAC,CAAU,EAA2C,EAAE,CACvE,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,KAAK,IAAI,IAAI,OAAQ,CAAyB,CAAC,KAAK,KAAK,QAAQ,IAAK,CAAyB,CAAC,KAAK,KAAK,IAAI,CAAC;IAE3I,IAAI,QAAQ,CAAC,MAAM,CAAC;QAAE,OAAO,IAAI,CAAC,CAAC,6BAA6B;IAEhE,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;QAC1B,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QACtC,IAAI,OAAO;YAAE,OAAO,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC;IAC9C,CAAC;IAED,OAAO,IAAI,CAAC,SAAS,CAAC;QACpB,KAAK,EAAE;YACL,OAAO,EAAE,IAAI,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,0BAA0B,MAAM,qBAAqB;YAC5F,IAAI,EAAE,gBAAgB;YACtB,IAAI,EAAE,MAAM;SACb;KACF,CAAC,CAAC;AACL,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,CAAC,KAAK,UAAU,gBAAgB,CACpC,MAAsB,EACtB,IAAqE,EACrE,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;QAC7B,OAAO,WAAW,CAChB,GAAG,EACH,8DAA8D,MAAM,CAAC,QAAQ,QAAQ,MAAM,CAAC,IAAI,EAAE,EAClG,OAAO,CACR,CAAC;IACJ,CAAC;IACD,MAAM,IAAI,GAAG,CAAC,IAAI,CAAC,OAAO,IAAI,EAAE,CAA4B,CAAC;IAC7D,MAAM,IAAI,GAAG,EAAE,GAAG,IAAI,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,WAAW,EAAE,CAAC;IACxE,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IACD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QAChD,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;QAC1B,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;AACL,CAAC"}
|
package/dist/candidates.d.ts
CHANGED
|
@@ -47,6 +47,19 @@ export interface Candidate {
|
|
|
47
47
|
consecutiveFailures: number;
|
|
48
48
|
lastStatus: number | null;
|
|
49
49
|
cooldownRemainingMs: number;
|
|
50
|
+
/**
|
|
51
|
+
* Credential faults observed on real traffic, on their own axis.
|
|
52
|
+
*
|
|
53
|
+
* A member answering 401 on every call used to be indistinguishable here from a healthy
|
|
54
|
+
* one — verdict `Pending`, breaker `closed`, no failure count — because a 401 is
|
|
55
|
+
* deliberately not health data and so reached none of the fields above. It is still not
|
|
56
|
+
* health data; it is reported as what it is. `credentialFault` true means the router is
|
|
57
|
+
* currently DEMOTING this member (it is tried only after every other candidate fails),
|
|
58
|
+
* and it expires, so a rotated key recovers without a restart.
|
|
59
|
+
*/
|
|
60
|
+
credentialFailures: number;
|
|
61
|
+
lastCredentialStatus: number | null;
|
|
62
|
+
credentialFault: boolean;
|
|
50
63
|
};
|
|
51
64
|
/** Observed real traffic through this proxy (not synthetic probes). */
|
|
52
65
|
observed: {
|
package/dist/candidates.js
CHANGED
|
@@ -127,6 +127,9 @@ export async function buildCandidates(cfg, opts = {}) {
|
|
|
127
127
|
consecutiveFailures: state?.consecutiveFailures ?? 0,
|
|
128
128
|
lastStatus: state?.lastStatus ?? null,
|
|
129
129
|
cooldownRemainingMs: Math.max(0, (state?.cooldownUntil ?? 0) - nowMs),
|
|
130
|
+
credentialFailures: state?.credentialFailures ?? 0,
|
|
131
|
+
lastCredentialStatus: state?.lastCredentialStatus ?? null,
|
|
132
|
+
credentialFault: breaker.hasCredentialFault(spec, nowMs),
|
|
130
133
|
},
|
|
131
134
|
observed: obs
|
|
132
135
|
? {
|
package/dist/candidates.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"candidates.js","sourceRoot":"","sources":["../src/candidates.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAG1C,OAAO,EAAE,WAAW,EAAsB,MAAM,iBAAiB,CAAC;AAClE,OAAO,EAAE,aAAa,EAAiB,MAAM,gBAAgB,CAAC;AAC9D,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,eAAe,EAAuB,MAAM,eAAe,CAAC;AACrE,OAAO,EAAE,oBAAoB,EAAuB,MAAM,sBAAsB,CAAC;AACjF,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC;AACnE,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;
|
|
1
|
+
{"version":3,"file":"candidates.js","sourceRoot":"","sources":["../src/candidates.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAG1C,OAAO,EAAE,WAAW,EAAsB,MAAM,iBAAiB,CAAC;AAClE,OAAO,EAAE,aAAa,EAAiB,MAAM,gBAAgB,CAAC;AAC9D,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,eAAe,EAAuB,MAAM,eAAe,CAAC;AACrE,OAAO,EAAE,oBAAoB,EAAuB,MAAM,sBAAsB,CAAC;AACjF,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC;AACnE,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAgI7C,MAAM,IAAI,GACR,8FAA8F;IAC9F,yFAAyF;IACzF,0FAA0F;IAC1F,wFAAwF,CAAC;AAE3F,2FAA2F;AAC3F,SAAS,UAAU,CAAC,IAAY,EAAE,GAAW;IAC3C,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IAC5B,MAAM,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAChD,IAAI,IAAI,KAAK,WAAW;QAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IACxC,MAAM,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IAC/C,OAAO,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;AACzC,CAAC;AAED;;;;GAIG;AACH,SAAS,YAAY,CAAC,GAAW;IAC/B,MAAM,GAAG,GAAG,IAAI,GAAG,EAAwD,CAAC;IAC5E,MAAM,KAAK,GAAG,CAAC,IAAY,EAAE,EAAE;QAC7B,IAAI,CAAC,GAAG,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACtB,IAAI,CAAC,CAAC,EAAE,CAAC;YACP,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,EAAE,aAAa,EAAE,EAAE,EAAE,CAAC;YACrC,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;QACnB,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC,CAAC;IAEF,KAAK,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,KAAK,IAAI,EAAE,CAAC,EAAE,CAAC;QACpE,KAAK,MAAM,CAAC,IAAI,KAAK;YAAE,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACnD,CAAC;IACD,KAAK,MAAM,CAAC,IAAI,EAAE,IAAI,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,EAAE,CAAC,EAAE,CAAC;QACvE,KAAK,MAAM,CAAC,IAAI,UAAU,CAAC,IAAI,EAAE,GAAG,CAAC;YAAE,KAAK,CAAC,CAAC,CAAC,CAAC,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC3E,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,SAAS,SAAS,CAAC,IAAY;IAC7B,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IAC5B,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC;AAClG,CAAC;AAED,+DAA+D;AAC/D,MAAM,CAAC,KAAK,UAAU,eAAe,CACnC,GAAW,EACX,OAWI,EAAE;IAEN,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,IAAI,oBAAoB,CAAC;IACrD,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC;IACvC,MAAM,MAAM,GAAG,CAAC,IAAI,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,EAAE,MAAM,IAAI,EAAE,CAAC;IAC5F,MAAM,SAAS,GAAG,oBAAoB,EAAE,CAAC;IAEzC,MAAM,UAAU,GAAgB,EAAE,CAAC;IACnC,KAAK,MAAM,CAAC,IAAI,EAAE,UAAU,CAAC,IAAI,YAAY,CAAC,GAAG,CAAC,EAAE,CAAC;QACnD,MAAM,EAAE,QAAQ,EAAE,KAAK,EAAE,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;QAC5C,IAAI,IAAI,CAAC,QAAQ,IAAI,QAAQ,KAAK,IAAI,CAAC,QAAQ;YAAE,SAAS;QAC1D,MAAM,CAAC,GAA+B,GAAG,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QAE9D,IAAI,MAAM,GAAmB,IAAI,CAAC;QAClC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,KAAK,QAAQ,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;YACtD,IAAI,CAAC;gBACH,MAAM,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,QAAQ,EAAE,CAAC,EAAE,KAAK,CAAC,CAAC;YACtD,CAAC;YAAC,MAAM,CAAC;gBACP,MAAM,GAAG,IAAI,CAAC;YAChB,CAAC;QACH,CAAC;QAED,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,IAAI,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,eAAe,CAAC,QAAQ,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QAC/F,MAAM,KAAK,GAAG,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC;QACrC,MAAM,GAAG,GAAG,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC,MAAM,CAAC,GAAG,QAAQ,IAAI,KAAK,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;QACzE,MAAM,QAAQ,GAAG,WAAW,CAAC,IAAI,CAAC,CAAC;QACnC,MAAM,OAAO,GAAG,aAAa,CAAC,KAAK,IAAI,IAAI,EAAE,MAAM,CAAC,CAAC;QACrD,MAAM,IAAI,GAAG,OAAO,EAAE,GAAG,CAAC;QAC1B,MAAM,GAAG,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAE,IAAI,CAAC,CAAC,CAAY,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QAExF,uFAAuF;QACvF,4FAA4F;QAC5F,2FAA2F;QAC3F,oCAAoC;QACpC,MAAM,cAAc,GAAG,CAAC,IAAI,CAAC,CAAC,IAAI,KAAK,QAAQ,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO;YACtE,CAAC,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC,EAAE,KAAK,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC;YACjE,CAAC,CAAC,IAAI,CAAC;QACT,qFAAqF;QACrF,kFAAkF;QAClF,mFAAmF;QACnF,sFAAsF;QACtF,uFAAuF;QACvF,wBAAwB;QACxB,MAAM,aAAa,GACjB,OAAO,IAAI,OAAO,CAAC,KAAK,KAAK,OAAO,CAAC,CAAC,CAAC,cAAc,OAAO,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,YAAY,CAAC;QACzF,MAAM,IAAI,GAAG,eAAe,CAAC,KAAK,IAAI,QAAQ,EAAE;YAC9C,cAAc;YACd,SAAS,EAAE;gBACT,aAAa,EAAE,GAAG,CAAC,gBAAgB,CAAC;gBACpC,mBAAmB,EAAE,GAAG,CAAC,cAAc,CAAC;gBACxC,uBAAuB,EAAE,GAAG,CAAC,kBAAkB,CAAC;gBAChD,IAAI,EAAE,aAAa;aACpB;SACF,CAAC,CAAC;QAEH,UAAU,CAAC,IAAI,CAAC;YACd,IAAI;YACJ,QAAQ;YACR,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC3B,KAAK,EAAE,UAAU,CAAC,KAAK;YACvB,aAAa,EAAE,UAAU,CAAC,aAAa;YACvC,qFAAqF;YACrF,oFAAoF;YACpF,MAAM,EAAE,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,YAAY,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI;YAChE,MAAM;YACN,eAAe,EAAE,OAAO,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,GAAG,CAAC,IAAI,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI;YAClF,MAAM,EAAE,OAAO;gBACb,CAAC,CAAC;oBACE,OAAO,EAAE,OAAO,CAAC,OAAO;oBACxB,KAAK,EAAE,OAAO,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI;oBAChD,KAAK,EAAE,OAAO,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI;oBAChD,QAAQ,EAAE,OAAO,CAAC,QAAQ;oBAC1B,SAAS,EAAE,OAAO,CAAC,SAAS;oBAC5B,YAAY,EAAE,OAAO,CAAC,YAAY;oBAClC,UAAU,EAAE,OAAO,CAAC,UAAU;iBAC/B;gBACH,CAAC,CAAC,IAAI;YACR,YAAY,EAAE,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,gBAAgB,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,IAAI;YAC7E,OAAO,EAAE;gBACP,IAAI,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,EAAE,KAAK,CAAC;gBACrC,mBAAmB,EAAE,KAAK,EAAE,mBAAmB,IAAI,CAAC;gBACpD,UAAU,EAAE,KAAK,EAAE,UAAU,IAAI,IAAI;gBACrC,mBAAmB,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,KAAK,EAAE,aAAa,IAAI,CAAC,CAAC,GAAG,KAAK,CAAC;gBACrE,kBAAkB,EAAE,KAAK,EAAE,kBAAkB,IAAI,CAAC;gBAClD,oBAAoB,EAAE,KAAK,EAAE,oBAAoB,IAAI,IAAI;gBACzD,eAAe,EAAE,OAAO,CAAC,kBAAkB,CAAC,IAAI,EAAE,KAAK,CAAC;aACzD;YACD,QAAQ,EAAE,GAAG;gBACX,CAAC,CAAC;oBACE,UAAU,EAAE,GAAG,CAAC,UAAU;oBAC1B,YAAY,EAAE,GAAG,CAAC,YAAY;oBAC9B,YAAY,EAAE,GAAG,CAAC,UAAU,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,cAAc,GAAG,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,IAAI;oBACzF,YAAY,EAAE,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,GAAG,CAAC,YAAY,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,IAAI;iBACjF;gBACH,CAAC,CAAC,IAAI;YACR,aAAa,EAAE,IAAI,CAAC,aAAa;YACjC,mBAAmB,EAAE,IAAI,CAAC,mBAAmB;YAC7C,eAAe,EAAE,IAAI,CAAC,eAAe;YACrC,qBAAqB,EAAE,IAAI,CAAC,qBAAqB;YACjD,GAAG,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC,CAAC,EAAE,qBAAqB,EAAE,IAAI,CAAC,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC5E,cAAc,EAAE,IAAI,CAAC,cAAc;YACnC,eAAe,EAAE,IAAI,CAAC,eAAe;YACrC,WAAW,EAAE,IAAI,CAAC,WAAW;YAC7B,aAAa,EAAE,OAAO,IAAI,EAAE,cAAc,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,CAAC,IAAI;YACrF,iBAAiB,EAAE,KAAK,CAAC,OAAO,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC,CAAC,CAAE,IAAI,CAAC,OAAoB,CAAC,CAAC,CAAC,EAAE;YACjF,MAAM,EAAE;gBACN,WAAW,EAAE,GAAG,CAAC,cAAc,CAAC;gBAChC,aAAa,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACrC,eAAe,EAAE,GAAG,CAAC,kBAAkB,CAAC;gBACxC,cAAc,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACtC,QAAQ,EAAE,GAAG,CAAC,WAAW,CAAC;gBAC1B,SAAS,EAAE,GAAG,CAAC,YAAY,CAAC;gBAC5B,aAAa,EAAE,GAAG,CAAC,iBAAiB,CAAC;gBACrC,eAAe,EAAE,GAAG,CAAC,mBAAmB,CAAC;gBACzC,wBAAwB,EAAE,GAAG,CAAC,8BAA8B,CAAC;gBAC7D,wBAAwB,EAAE,GAAG,CAAC,8BAA8B,CAAC;gBAC7D,WAAW,EAAE,GAAG,CAAC,cAAc,CAAC;gBAChC,SAAS,EAAE,GAAG,CAAC,YAAY,CAAC;aAC7B;YACD,UAAU,EAAE;gBACV,QAAQ,EAAE,QAAQ,CAAC,KAAK;gBACxB,aAAa,EAAE,QAAQ,CAAC,KAAK;gBAC7B,eAAe,EAAE,QAAQ,CAAC,OAAO,IAAI,EAAE;gBACvC,gBAAgB,EAAE,OAAO,CAAC,oBAAoB,CAAC,IAAI,CAAC;aACrD;SACF,CAAC,CAAC;IACL,CAAC;IAED,OAAO;QACL,YAAY,EAAE,IAAI,CAAC,GAAG,IAAI,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC,WAAW,EAAE;QACvD,eAAe,EAAE,GAAG,CAAC,OAAO,CAAC,OAAO,KAAK,IAAI;QAC7C,IAAI,EAAE,IAAI;QACV,UAAU;KACX,CAAC;AACJ,CAAC"}
|
|
@@ -7,6 +7,19 @@ export interface CircuitState {
|
|
|
7
7
|
lastStatus?: number | undefined;
|
|
8
8
|
pings: PingRecord[];
|
|
9
9
|
quotaPercent?: number | null | undefined;
|
|
10
|
+
/**
|
|
11
|
+
* Credential faults (401/403), counted SEPARATELY from health failures.
|
|
12
|
+
*
|
|
13
|
+
* A revoked key is not a sick backend: recording it as a failure would open the breaker on
|
|
14
|
+
* a configuration problem and hide the 401 the operator has to see behind a "target
|
|
15
|
+
* unhealthy" skip, and recording it as a success would launder a permanently broken
|
|
16
|
+
* candidate into a healthy one. So it is neither — it is its own dimension, which is what
|
|
17
|
+
* lets a pool step over the candidate while `/candidates` still reports exactly why.
|
|
18
|
+
*/
|
|
19
|
+
credentialFailures: number;
|
|
20
|
+
lastCredentialStatus?: number | undefined;
|
|
21
|
+
/** While in the future, this target is demoted (never dropped) as unusable-for-now. */
|
|
22
|
+
credentialFaultUntil: number;
|
|
10
23
|
}
|
|
11
24
|
/**
|
|
12
25
|
* The ordering value for a target nothing has been measured about.
|
|
@@ -45,7 +58,27 @@ export declare class CircuitBreaker {
|
|
|
45
58
|
status?: number;
|
|
46
59
|
quotaPercent?: number | null;
|
|
47
60
|
at?: number;
|
|
61
|
+
/**
|
|
62
|
+
* The provider's own `Retry-After`, in ms, when it published one. It replaces the flat
|
|
63
|
+
* cooldown guess: the provider is the only party that knows when it will serve again,
|
|
64
|
+
* and 2 minutes is as likely to be far too long (a 20-second groq TPM window) as far
|
|
65
|
+
* too short. Clamped, because the value is attacker-adjacent input.
|
|
66
|
+
*/
|
|
67
|
+
retryAfterMs?: number | undefined;
|
|
48
68
|
}): void;
|
|
69
|
+
/**
|
|
70
|
+
* Record a 401/403 — a credential fault, which is NOT health data.
|
|
71
|
+
*
|
|
72
|
+
* This deliberately does not touch `consecutiveFailures`, `cooldownUntil` or the ping
|
|
73
|
+
* history, so `isHealthy()` keeps answering the question it is named for and a revoked key
|
|
74
|
+
* never masquerades as a flaky backend (or vice versa). What it does do is mark the target
|
|
75
|
+
* unusable-for-now, which `hasCredentialFault()` reports and the router uses to DEMOTE it —
|
|
76
|
+
* so a pool stops paying a round-trip per request for a member that cannot authenticate,
|
|
77
|
+
* while `/candidates` still shows the operator the 401 and its count.
|
|
78
|
+
*/
|
|
79
|
+
recordCredentialFault(target: ResolvedTarget | string, status: number, at?: number): void;
|
|
80
|
+
/** True while a recent 401/403 makes this target a last resort. Expires, so a fixed key recovers. */
|
|
81
|
+
hasCredentialFault(target: ResolvedTarget | string, now?: number): boolean;
|
|
49
82
|
/**
|
|
50
83
|
* Stability for a target whose behaviour has actually been observed, or
|
|
51
84
|
* `null` when NOTHING has been measured.
|
package/dist/circuit-breaker.js
CHANGED
|
@@ -3,6 +3,17 @@ const DEFAULT_COOLDOWN_MS = 60000; // 1 minute cooldown after consecutive failur
|
|
|
3
3
|
const RATE_LIMIT_COOLDOWN_MS = 120000; // 2 minutes cooldown on 429
|
|
4
4
|
const MAX_FAILURES_BEFORE_TRIP = 2;
|
|
5
5
|
const MAX_PING_HISTORY = 10;
|
|
6
|
+
/**
|
|
7
|
+
* How long a 401/403 demotes a candidate before it is tried again.
|
|
8
|
+
*
|
|
9
|
+
* Deliberately finite: a rotated key must recover without restarting the proxy. Deliberately
|
|
10
|
+
* not zero: without it, a pool holding seven keyless members pays seven round-trips of 401
|
|
11
|
+
* latency on EVERY request before reaching a live one.
|
|
12
|
+
*/
|
|
13
|
+
const CREDENTIAL_FAULT_TTL_MS = 300000; // 5 minutes
|
|
14
|
+
/** Clamp for a provider-supplied Retry-After, so a hostile or absurd value cannot park a target. */
|
|
15
|
+
const MIN_RETRY_AFTER_MS = 1000;
|
|
16
|
+
const MAX_RETRY_AFTER_MS = 900000; // 15 minutes
|
|
6
17
|
/**
|
|
7
18
|
* The ordering value for a target nothing has been measured about.
|
|
8
19
|
*
|
|
@@ -49,6 +60,8 @@ export class CircuitBreaker {
|
|
|
49
60
|
lastFailureTime: 0,
|
|
50
61
|
cooldownUntil: 0,
|
|
51
62
|
pings: [],
|
|
63
|
+
credentialFailures: 0,
|
|
64
|
+
credentialFaultUntil: 0,
|
|
52
65
|
};
|
|
53
66
|
this.states.set(key, fresh);
|
|
54
67
|
return fresh;
|
|
@@ -87,18 +100,52 @@ export class CircuitBreaker {
|
|
|
87
100
|
if (outcome.ok) {
|
|
88
101
|
state.consecutiveFailures = 0;
|
|
89
102
|
state.cooldownUntil = 0;
|
|
103
|
+
// A call that actually succeeded is proof the credential works now — that is the
|
|
104
|
+
// recovery path for a key fixed while the proxy is running.
|
|
105
|
+
state.credentialFailures = 0;
|
|
106
|
+
state.credentialFaultUntil = 0;
|
|
90
107
|
return;
|
|
91
108
|
}
|
|
92
109
|
state.consecutiveFailures += 1;
|
|
93
110
|
state.lastFailureTime = now;
|
|
94
|
-
|
|
111
|
+
const asked = outcome.retryAfterMs !== undefined
|
|
112
|
+
? Math.min(MAX_RETRY_AFTER_MS, Math.max(MIN_RETRY_AFTER_MS, outcome.retryAfterMs))
|
|
113
|
+
: null;
|
|
114
|
+
// HTTP 429 (Rate Limit) trips immediately — for exactly as long as the provider asked,
|
|
115
|
+
// or 2 minutes when it did not say.
|
|
95
116
|
if (outcome.status === 429) {
|
|
96
|
-
state.cooldownUntil = now + RATE_LIMIT_COOLDOWN_MS;
|
|
117
|
+
state.cooldownUntil = now + (asked ?? RATE_LIMIT_COOLDOWN_MS);
|
|
118
|
+
}
|
|
119
|
+
else if (asked !== null) {
|
|
120
|
+
// A 503 with a Retry-After is the provider scheduling us; honour it on the first
|
|
121
|
+
// failure rather than waiting for a second one to trip the generic cooldown.
|
|
122
|
+
state.cooldownUntil = now + asked;
|
|
97
123
|
}
|
|
98
124
|
else if (state.consecutiveFailures >= MAX_FAILURES_BEFORE_TRIP) {
|
|
99
125
|
state.cooldownUntil = now + DEFAULT_COOLDOWN_MS;
|
|
100
126
|
}
|
|
101
127
|
}
|
|
128
|
+
/**
|
|
129
|
+
* Record a 401/403 — a credential fault, which is NOT health data.
|
|
130
|
+
*
|
|
131
|
+
* This deliberately does not touch `consecutiveFailures`, `cooldownUntil` or the ping
|
|
132
|
+
* history, so `isHealthy()` keeps answering the question it is named for and a revoked key
|
|
133
|
+
* never masquerades as a flaky backend (or vice versa). What it does do is mark the target
|
|
134
|
+
* unusable-for-now, which `hasCredentialFault()` reports and the router uses to DEMOTE it —
|
|
135
|
+
* so a pool stops paying a round-trip per request for a member that cannot authenticate,
|
|
136
|
+
* while `/candidates` still shows the operator the 401 and its count.
|
|
137
|
+
*/
|
|
138
|
+
recordCredentialFault(target, status, at = Date.now()) {
|
|
139
|
+
const state = this.getOrCreate(this.getKey(target));
|
|
140
|
+
state.credentialFailures += 1;
|
|
141
|
+
state.lastCredentialStatus = status;
|
|
142
|
+
state.credentialFaultUntil = at + CREDENTIAL_FAULT_TTL_MS;
|
|
143
|
+
}
|
|
144
|
+
/** True while a recent 401/403 makes this target a last resort. Expires, so a fixed key recovers. */
|
|
145
|
+
hasCredentialFault(target, now = Date.now()) {
|
|
146
|
+
const state = this.states.get(this.getKey(target));
|
|
147
|
+
return !!state && state.credentialFaultUntil > now;
|
|
148
|
+
}
|
|
102
149
|
/**
|
|
103
150
|
* Stability for a target whose behaviour has actually been observed, or
|
|
104
151
|
* `null` when NOTHING has been measured.
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"circuit-breaker.js","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,iBAAiB,EAAmB,MAAM,mBAAmB,CAAC;
|
|
1
|
+
{"version":3,"file":"circuit-breaker.js","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,iBAAiB,EAAmB,MAAM,mBAAmB,CAAC;AAwBvE,MAAM,mBAAmB,GAAG,KAAK,CAAC,CAAC,+CAA+C;AAClF,MAAM,sBAAsB,GAAG,MAAM,CAAC,CAAC,4BAA4B;AACnE,MAAM,wBAAwB,GAAG,CAAC,CAAC;AACnC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AAE5B;;;;;;GAMG;AACH,MAAM,uBAAuB,GAAG,MAAM,CAAC,CAAC,YAAY;AAEpD,oGAAoG;AACpG,MAAM,kBAAkB,GAAG,IAAI,CAAC;AAChC,MAAM,kBAAkB,GAAG,MAAM,CAAC,CAAC,aAAa;AAEhD;;;;;;;;;;;;GAYG;AACH,MAAM,CAAC,MAAM,oBAAoB,GAAG,EAAE,CAAC;AAEvC;;;;;;;;GAQG;AACH,SAAS,WAAW,CAAC,OAAyC;IAC5D,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS;QAAE,OAAO,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC;IACpE,IAAI,OAAO,CAAC,MAAM,IAAI,GAAG,IAAI,OAAO,CAAC,MAAM,GAAG,GAAG;QAAE,OAAO,KAAK,CAAC;IAChE,OAAO,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC;AAChC,CAAC;AAED,MAAM,OAAO,cAAc;IACjB,MAAM,GAAG,IAAI,GAAG,EAAwB,CAAC;IAEzC,MAAM,CAAC,MAA+B;QAC5C,IAAI,OAAO,MAAM,KAAK,QAAQ;YAAE,OAAO,MAAM,CAAC;QAC9C,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC,QAAQ,IAAI,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC;IAC/E,CAAC;IAEO,WAAW,CAAC,GAAW;QAC7B,MAAM,QAAQ,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACtC,IAAI,QAAQ;YAAE,OAAO,QAAQ,CAAC;QAC9B,MAAM,KAAK,GAAiB;YAC1B,mBAAmB,EAAE,CAAC;YACtB,eAAe,EAAE,CAAC;YAClB,aAAa,EAAE,CAAC;YAChB,KAAK,EAAE,EAAE;YACT,kBAAkB,EAAE,CAAC;YACrB,oBAAoB,EAAE,CAAC;SACxB,CAAC;QACF,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;QAC5B,OAAO,KAAK,CAAC;IACf,CAAC;IAED,8EAA8E;IAC9E,SAAS,CAAC,MAA+B,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;QACzD,MAAM,GAAG,GAAG,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAChC,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACnC,IAAI,CAAC,KAAK;YAAE,OAAO,IAAI,CAAC;QAExB,IAAI,KAAK,CAAC,aAAa,GAAG,GAAG,EAAE,CAAC;YAC9B,OAAO,KAAK,CAAC,CAAC,8BAA8B;QAC9C,CAAC;QAED,OAAO,IAAI,CAAC,CAAC,8BAA8B;IAC7C,CAAC;IAED;;;;;;;;;;OAUG;IACH,aAAa,CACX,MAA+B,EAC/B,OAaC;QAED,MAAM,GAAG,GAAG,OAAO,CAAC,EAAE,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC;QACrC,MAAM,KAAK,GAAG,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QAEpD,IAAI,OAAO,CAAC,YAAY,KAAK,SAAS;YAAE,KAAK,CAAC,YAAY,GAAG,OAAO,CAAC,YAAY,CAAC;QAClF,KAAK,CAAC,UAAU,GAAG,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;QACpE,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,WAAW,CAAC,OAAO,CAAC,EAAE,SAAS,EAAE,GAAG,EAAE,CAAC,CAAC;QACxF,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,gBAAgB;YAAE,KAAK,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QAE/D,IAAI,OAAO,CAAC,EAAE,EAAE,CAAC;YACf,KAAK,CAAC,mBAAmB,GAAG,CAAC,CAAC;YAC9B,KAAK,CAAC,aAAa,GAAG,CAAC,CAAC;YACxB,iFAAiF;YACjF,4DAA4D;YAC5D,KAAK,CAAC,kBAAkB,GAAG,CAAC,CAAC;YAC7B,KAAK,CAAC,oBAAoB,GAAG,CAAC,CAAC;YAC/B,OAAO;QACT,CAAC;QAED,KAAK,CAAC,mBAAmB,IAAI,CAAC,CAAC;QAC/B,KAAK,CAAC,eAAe,GAAG,GAAG,CAAC;QAE5B,MAAM,KAAK,GACT,OAAO,CAAC,YAAY,KAAK,SAAS;YAChC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,kBAAkB,EAAE,IAAI,CAAC,GAAG,CAAC,kBAAkB,EAAE,OAAO,CAAC,YAAY,CAAC,CAAC;YAClF,CAAC,CAAC,IAAI,CAAC;QAEX,uFAAuF;QACvF,oCAAoC;QACpC,IAAI,OAAO,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YAC3B,KAAK,CAAC,aAAa,GAAG,GAAG,GAAG,CAAC,KAAK,IAAI,sBAAsB,CAAC,CAAC;QAChE,CAAC;aAAM,IAAI,KAAK,KAAK,IAAI,EAAE,CAAC;YAC1B,iFAAiF;YACjF,6EAA6E;YAC7E,KAAK,CAAC,aAAa,GAAG,GAAG,GAAG,KAAK,CAAC;QACpC,CAAC;aAAM,IAAI,KAAK,CAAC,mBAAmB,IAAI,wBAAwB,EAAE,CAAC;YACjE,KAAK,CAAC,aAAa,GAAG,GAAG,GAAG,mBAAmB,CAAC;QAClD,CAAC;IACH,CAAC;IAED;;;;;;;;;OASG;IACH,qBAAqB,CAAC,MAA+B,EAAE,MAAc,EAAE,EAAE,GAAG,IAAI,CAAC,GAAG,EAAE;QACpF,MAAM,KAAK,GAAG,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACpD,KAAK,CAAC,kBAAkB,IAAI,CAAC,CAAC;QAC9B,KAAK,CAAC,oBAAoB,GAAG,MAAM,CAAC;QACpC,KAAK,CAAC,oBAAoB,GAAG,EAAE,GAAG,uBAAuB,CAAC;IAC5D,CAAC;IAED,qGAAqG;IACrG,kBAAkB,CAAC,MAA+B,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;QAClE,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACnD,OAAO,CAAC,CAAC,KAAK,IAAI,KAAK,CAAC,oBAAoB,GAAG,GAAG,CAAC;IACrD,CAAC;IAED;;;;;;;;OAQG;IACH,oBAAoB,CAAC,MAA+B;QAClD,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACnD,IAAI,CAAC,KAAK,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,IAAI,CAAC;QACpD,MAAM,KAAK,GAAG,iBAAiB,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QAC7C,OAAO,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IAChC,CAAC;IAED,wEAAwE;IACxE,eAAe,CAAC,MAA+B;QAC7C,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;QACnD,OAAO,CAAC,CAAC,KAAK,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC;IAC3C,CAAC;IAED,qFAAqF;IACrF,2FAA2F;IAC3F,wFAAwF;IACxF,4FAA4F;IAC5F,2FAA2F;IAC3F,4FAA4F;IAC5F,kFAAkF;IAElF,mCAAmC;IACnC,QAAQ,CAAC,MAA+B;QACtC,OAAO,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;IAC9C,CAAC;IAED,sCAAsC;IACtC,YAAY;QACV,OAAO,IAAI,CAAC,MAAM,CAAC;IACrB,CAAC;IAED;;;;;;;;;;;OAWG;IACH,iBAAiB,CAAC,OAAyB,EAAE,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE;QAC3D,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;QAC9D,MAAM,UAAU,GAAG,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC;QAC1D,OAAO,CAAC,GAAG,UAAU,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;YACnC,MAAM,EAAE,GAAG,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC;YACxC,MAAM,EAAE,GAAG,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC;YACxC,MAAM,EAAE,GAAG,EAAE,IAAI,oBAAoB,CAAC;YACtC,MAAM,EAAE,GAAG,EAAE,IAAI,oBAAoB,CAAC;YACtC,IAAI,EAAE,KAAK,EAAE;gBAAE,OAAO,EAAE,GAAG,EAAE,CAAC;YAC9B,IAAI,CAAC,EAAE,KAAK,IAAI,CAAC,KAAK,CAAC,EAAE,KAAK,IAAI,CAAC;gBAAE,OAAO,EAAE,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YACjE,OAAO,CAAC,CAAC;QACX,CAAC,CAAC,CAAC;IACL,CAAC;IAED,gCAAgC;IAChC,KAAK;QACH,IAAI,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC;IACtB,CAAC;CACF;AAED,MAAM,CAAC,MAAM,oBAAoB,GAAG,IAAI,cAAc,EAAE,CAAC"}
|
package/dist/cli.js
CHANGED
|
@@ -658,6 +658,19 @@ export async function runOffload(arg) {
|
|
|
658
658
|
function fmt(v, suffix = "") {
|
|
659
659
|
return v === null || v === undefined ? "-" : `${v}${suffix}`;
|
|
660
660
|
}
|
|
661
|
+
/**
|
|
662
|
+
* Mean latency of this proxy's OWN traffic to a target, with the call count that backs it.
|
|
663
|
+
*
|
|
664
|
+
* Deliberately not merged into the `p95` column: that one is the synthetic probe loop's measured
|
|
665
|
+
* p95, and an average over real requests is a different statistic from a different sample. Shown
|
|
666
|
+
* in seconds past 10s because "62605" reads as noise where "62.6s" reads as a decision.
|
|
667
|
+
*/
|
|
668
|
+
function obsLatency(o) {
|
|
669
|
+
if (!o || o.avgLatencyMs === null || o.totalCalls === 0)
|
|
670
|
+
return "-";
|
|
671
|
+
const ms = o.avgLatencyMs;
|
|
672
|
+
return ms >= 10000 ? `${(ms / 1000).toFixed(1)}s` : `${ms}ms`;
|
|
673
|
+
}
|
|
661
674
|
/**
|
|
662
675
|
* How much to trust the strength number. A score from 5 leaderboards and a score from "nothing is
|
|
663
676
|
* known, assume neutral" must never render identically.
|
|
@@ -698,6 +711,11 @@ export async function runCandidates() {
|
|
|
698
711
|
"$/Mout".padEnd(8) +
|
|
699
712
|
"verdict".padEnd(10) +
|
|
700
713
|
"p95".padEnd(8) +
|
|
714
|
+
// Latency actually observed on this proxy's own traffic. The synthetic-probe p95 beside it is
|
|
715
|
+
// routinely blank, so the table could show a rank-1 pool member with NO latency signal at all
|
|
716
|
+
// while the proxy had already measured it at 60+ seconds per call — which is the difference
|
|
717
|
+
// between a pool that suits mechanical batch work and one that does not.
|
|
718
|
+
"obs".padEnd(8) +
|
|
701
719
|
"quota".padEnd(7) +
|
|
702
720
|
"breaker".padEnd(9) +
|
|
703
721
|
"ctx";
|
|
@@ -706,7 +724,15 @@ export async function runCandidates() {
|
|
|
706
724
|
const tags = [...c.pools, ...c.subagentTiers.map((t) => `@${t}`)].join(",") || "-";
|
|
707
725
|
const live = c.listed === null ? "?" : c.listed ? "yes" : "NO";
|
|
708
726
|
const ctx = c.contextLength ? `${Math.round(c.contextLength / 1000)}k` : "-";
|
|
709
|
-
|
|
727
|
+
// A member that answers 401 on every call read "closed" here — the same as a healthy one —
|
|
728
|
+
// because a credential fault is deliberately not health data and so never reached the
|
|
729
|
+
// breaker's failure fields. It has its own axis now, and it is shown: the reason half a
|
|
730
|
+
// pool can be unusable while every row looks fine is precisely this cell.
|
|
731
|
+
const breaker = c.breaker.open
|
|
732
|
+
? `OPEN ${Math.round(c.breaker.cooldownRemainingMs / 1000)}s`
|
|
733
|
+
: c.breaker.credentialFault
|
|
734
|
+
? `AUTH ${c.breaker.lastCredentialStatus ?? ""}`.trim()
|
|
735
|
+
: "closed";
|
|
710
736
|
process.stdout.write(c.spec.slice(0, 31).padEnd(32) +
|
|
711
737
|
tags.slice(0, 23).padEnd(24) +
|
|
712
738
|
// The scalar plus how well-evidenced it is: "83.3/4" = 4 published signals behind it,
|
|
@@ -722,6 +748,7 @@ export async function runCandidates() {
|
|
|
722
748
|
: "-").padEnd(8) +
|
|
723
749
|
(c.health?.verdict ?? "-").padEnd(10) +
|
|
724
750
|
fmt(c.health?.p95Ms ?? null, "ms").padEnd(8) +
|
|
751
|
+
obsLatency(c.observed).padEnd(8) +
|
|
725
752
|
fmt(c.quotaPercent, "%").padEnd(7) +
|
|
726
753
|
breaker.padEnd(9) +
|
|
727
754
|
// Provenance inline: "~" = another provider's figure for this model id. A NIM row must
|
|
@@ -733,9 +760,31 @@ export async function runCandidates() {
|
|
|
733
760
|
}
|
|
734
761
|
const fuzzy = view.candidates.filter((c) => c.capabilityMatch?.match === "fuzzy");
|
|
735
762
|
const srcs = [...new Set(view.candidates.flatMap((c) => c.capabilitySources))].sort();
|
|
763
|
+
// Say out loud how much of the roster is currently unusable. A row-by-row table makes
|
|
764
|
+
// "5 of 14 members can actually serve" something the reader has to notice; a pool that is
|
|
765
|
+
// half dead is worth stating.
|
|
766
|
+
const noKey = view.candidates.filter((c) => !c.hasKey);
|
|
767
|
+
const authFault = view.candidates.filter((c) => c.breaker.credentialFault);
|
|
768
|
+
const cooling = view.candidates.filter((c) => c.breaker.open);
|
|
769
|
+
if (noKey.length || authFault.length || cooling.length) {
|
|
770
|
+
process.stdout.write(`\nNot first choice right now (of ${view.candidates.length} targets):\n` +
|
|
771
|
+
` ${authFault.length} auth-faulted, ${cooling.length} cooling — DEMOTED: still tried, but only\n` +
|
|
772
|
+
" after every other candidate has failed on that request.\n" +
|
|
773
|
+
` ${noKey.length} with no key set — DROPPED from a pool entirely (resolveTargets removes a\n` +
|
|
774
|
+
" declared-but-unset credential), so a pool's real size is smaller than its member count.\n" +
|
|
775
|
+
" ⚠ That includes free providers that would serve WITHOUT a key: declaring an authEnv and\n" +
|
|
776
|
+
" leaving it unset excludes them from every pool they belong to.\n" +
|
|
777
|
+
" `llm-relay keys` says whether a credential is good; `llm-relay pools --probe` is the\n" +
|
|
778
|
+
" only check that proves a member can actually serve.\n");
|
|
779
|
+
}
|
|
736
780
|
process.stdout.write("\nColumns are independent — weigh them yourself. agentic/coding/BFCL/aider/arena are\n" +
|
|
737
781
|
"capability from DIFFERENT leaderboards and they disagree; verdict/p95 are live behaviour;\n" +
|
|
738
782
|
"quota/breaker/$ are what it costs to use right now. A blank cell means NOT MEASURED.\n" +
|
|
783
|
+
` p95 = synthetic probe loop; obs = mean of this proxy's OWN requests. Different samples,\n` +
|
|
784
|
+
` so they are not merged — and a high "obs" on a top-ranked member is worth seeing\n` +
|
|
785
|
+
` before pointing bulk work at that pool (str ranks capability, never speed).\n` +
|
|
786
|
+
` breaker: "OPEN 42s" = cooling after failures/429; "AUTH 401" = credential fault, demoted\n` +
|
|
787
|
+
` until it is retried (expires, so a rotated key recovers with no restart).\n` +
|
|
739
788
|
` str = the one scalar pool ordering needs. "83.3/4" = 4 published signals behind it;\n` +
|
|
740
789
|
` "obs" = ranked on this proxy's own traffic, "neut" = nothing known.\n` +
|
|
741
790
|
` "~" on ctx/$ = another provider's figure for the same model id (this one publishes none);\n` +
|