ldrouter 1.17.3 → 1.17.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -4,6 +4,20 @@ All notable changes to this project are documented here. The format follows
4
4
  [Keep a Changelog](https://keepachangelog.com/) and the project adheres to
5
5
  [Semantic Versioning](https://semver.org/).
6
6
 
7
+ ## [1.17.4] - 2026-09-18
8
+
9
+ ### Fixed
10
+
11
+ - **An account that ran out of quota kept being retried instead of leaving the pool.** A quota refusal was classified `unknown`, which is not a routing decision: `isUpstreamHealthFailure` excludes it and `shouldFallback` answers false for it, so the pool selected the same spent account on every request and answered "usage limited". Live production data confirmed the shape — 49 quota refusals (`Codex upstream HTTP 429: The usage limit has been reached`, `Qoder account is out of quota`) recorded over 14 days, every one of them `failure_reason=unknown`, with the same `codex_account_id` / `qoder_account_id` retried each time. Quota is now its own failure class, checked before the error-type switch (a 429 reaches the classifier as `upstream_rate_limit`, which previously returned `http_status` before the quota test could run).
12
+ - **The exhausted account is now disabled, not just marked degraded.** Disabling (`enabled=0`) is the write that actually stops the churn: every selection path — `expandCodexAccountCandidates`, `expandQoderAccountCandidates`, `getCodexAccountForProvider`, `findEligibleQoderAccount` — filters on it, whereas `health_state` alone was overwritten to `healthy` by the next successful request. A Qoder billing block now disables the account too, instead of only degrading it.
13
+ - **A direct model never advanced to the next account.** The retry was gated on `comboPlan ? ... : false`, so the reported case — a direct `qoder/...` model, logged 33 times in production — never retried at all. A quota failure now always advances to the next candidate, combo or not; that is safe precisely because the exhausted account was just disabled, so the retry cannot land on it again.
14
+ - **The upstream 429 body was discarded, hiding the reason.** Quota exhaustion arrives as a plain 429 on OpenAI-compatible providers, with the wording ("you exceeded your current quota") only in the body — which `upstreamHttpError` dropped, leaving an account unclassifiable. The redacted body excerpt is now carried on a 429 too. A 429 without quota wording is still treated as a transient throttle: it retries the same account and does not disable it.
15
+ - **Re-enabling an account in the admin UI did not put it back in the pool.** The enable toggle flipped `enabled` but left `health_state=down`, which every selection filter excludes, so the account stayed unroutable while the panel showed it as enabled. Both the Codex and Qoder routes now clear the health verdict when re-enabling.
16
+
17
+ ### Added
18
+
19
+ - **Usage and Credits are refreshed as the pool is used.** Both the Codex usage snapshot and the Qoder credits snapshot are re-read in the background after a successful request, throttled per account so a burst of traffic costs at most one extra upstream call, and never awaited — the response path is unaffected.
20
+
7
21
  ## [1.17.3] - 2026-09-17
8
22
 
9
23
  ### Added
package/README.md CHANGED
@@ -65,6 +65,8 @@ ZIP upload and automatic Codex CLI config-file generation/mutation are not inclu
65
65
 
66
66
  The account panel shows the 5-hour and weekly quota windows with reset countdowns, per-account and bulk usage refresh, weekly reset credits, an opt-in 5-hour window auto-start, and a **Test** control that probes the upstream account without exposing tokens. Routing order is set by dragging rows; the saved order is the fallback order the router uses. **Delete** is permanent and erases the stored encrypted credentials — past request logs are kept but lose the account reference.
67
67
 
68
+ **A quota-exhausted account leaves the pool on its own.** When an account refuses for quota reasons — Codex `429 The usage limit has been reached`, or Qoder `out of credits` — it is marked `down` *and disabled*, so the very next request picks a different account in the pool rather than retrying the spent one. This applies to direct `codex/...` and `qoder/...` models too, not only combos. Re-enable the account in the panel once its window resets. Both Codex usage and Qoder credits are also refreshed in the background after a successful request (throttled, and never on the response path), so the panel figures track real usage without pressing **Refresh**.
69
+
68
70
  ### Qoder setup
69
71
 
70
72
  Qoder providers use a personal access token (PAT) instead of an API key. Create one at `https://qoder.com/account/integrations`; it starts with `pt-`.
@@ -155,11 +155,16 @@ export function persistQoderJobToken(id, update) {
155
155
  health_state='healthy', last_error=NULL, updated_at=? WHERE id=?`)
156
156
  .run(jobToken.ciphertext, jobToken.nonce, jobToken.version, update.expiresAt, update.catalogJson ?? null, update.catalogFetchedAt ?? null, nextUpdatedAt(id), id);
157
157
  }
158
- export function setQoderAccountHealth(id, healthState, lastError = null) {
158
+ /** `enabled` mirrors the Codex setter: pass `false` to take an exhausted account out of the pool. */
159
+ export function setQoderAccountHealth(id, healthState, lastError = null, enabled) {
159
160
  const failures = healthState === 'healthy' ? 0 : null;
160
- getRawDb()
161
- .prepare(`UPDATE qoder_accounts SET health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), updated_at=? WHERE id=?`)
162
- .run(healthState, lastError, failures, nextUpdatedAt(id), id);
161
+ const fields = enabled === undefined
162
+ ? 'health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), updated_at=?'
163
+ : 'health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), enabled=?, updated_at=?';
164
+ const values = enabled === undefined
165
+ ? [healthState, lastError, failures, nextUpdatedAt(id), id]
166
+ : [healthState, lastError, failures, enabled ? 1 : 0, nextUpdatedAt(id), id];
167
+ getRawDb().prepare(`UPDATE qoder_accounts SET ${fields} WHERE id=?`).run(...values);
163
168
  }
164
169
  export function readQoderCatalog(id) {
165
170
  const row = getRawDb()
@@ -172,6 +177,11 @@ export function saveQoderCatalog(id, catalogJson, fetchedAt) {
172
177
  .prepare('UPDATE qoder_accounts SET catalog_json=?, catalog_fetched_at=?, updated_at=? WHERE id=?')
173
178
  .run(catalogJson, fetchedAt ?? new Date().toISOString(), nextUpdatedAt(id), id);
174
179
  }
180
+ /** When the Credits snapshot was last written; null when it never has been. */
181
+ export function readQoderCreditsUpdatedAt(id) {
182
+ const row = getRawDb().prepare('SELECT credits_updated_at AS at FROM qoder_accounts WHERE id=?').get(id);
183
+ return row?.at ?? null;
184
+ }
175
185
  /** Credits are observability, not routing state: a failed fetch is recorded without touching health. */
176
186
  export function saveQoderCredits(id, creditsJson, error, fetchedAt) {
177
187
  getRawDb()
@@ -8,7 +8,9 @@ import { loadCombo, selectCandidates, orderCandidates, shouldFallback, expandCod
8
8
  import { listCodexAccountsForProvider, setCodexAccountHealth } from '../db/repositories/codex-accounts.js';
9
9
  import { listQoderAccountsForProvider, setQoderAccountHealth } from '../db/repositories/qoder-accounts.js';
10
10
  import { qoderAttemptFailure, qoderCredentialsFor } from '../providers/qoder/credentials.js';
11
+ import { refreshQoderCreditsIfStale } from '../providers/qoder/credits.js';
11
12
  import { callQoderNonStreaming, callQoderStreaming } from '../providers/qoder/client.js';
13
+ import { refreshCodexUsageIfStale } from '../providers/codex-autostart.js';
12
14
  import { getEffectiveState, isOpen, recordSuccess, recordFailure, halfOpenProbeAllowed } from '../routing/circuit.js';
13
15
  import { checkRpm, checkTpm, acquireConcurrent, releaseConcurrent } from '../routing/ratelimit.js';
14
16
  import { checkDailyMonthly, consumeUsage } from '../routing/quota.js';
@@ -279,6 +281,13 @@ export class GatewayRunner {
279
281
  setCodexAccountHealth(candidate.codexAccountId, 'healthy');
280
282
  if (candidate.qoderAccountId)
281
283
  setQoderAccountHealth(candidate.qoderAccountId, 'healthy');
284
+ // Refresh the quota snapshot in the background so the dashboard reflects real usage.
285
+ // Never awaited: it is a second upstream call, and the response is already complete.
286
+ // Throttled inside each helper, so a burst of requests costs at most one refresh.
287
+ if (candidate.codexAccountId)
288
+ void refreshCodexUsageIfStale(candidate.codexAccountId).catch(() => { });
289
+ if (candidate.qoderAccountId)
290
+ void refreshQoderCreditsIfStale(candidate.qoderAccountId).catch(() => { });
282
291
  getDb().update(schema.providers).set({ healthState: 'healthy', updatedAt: new Date().toISOString() }).where(eq(schema.providers.id, provider.id)).run();
283
292
  attempts.push(attempt);
284
293
  sentToClient = out.streamStarted ?? false;
@@ -286,15 +295,19 @@ export class GatewayRunner {
286
295
  }
287
296
  catch (e) {
288
297
  const err = e instanceof GatewayError ? e : new GatewayError('upstream_error', e.message, { cause: e });
298
+ const quotaFailure = isQuotaFailure(err);
289
299
  debugUpstream(ctx.requestId, 'ATTEMPT ERROR', [
290
300
  `attempt=${i + 1}`,
291
301
  `provider=${provider.name}`,
292
302
  `model=${candidate.publicModelId}`,
293
303
  `type=${err.type}`,
294
304
  `status=${err.status}`,
295
- `willFallback=${comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false}`,
305
+ `willFallback=${shouldRetryAttempt(comboPlan, err)}`,
296
306
  ]);
297
- const shouldRetry = comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false;
307
+ // Quota failures always advance (see shouldRetryAttempt): the next candidate for a pool
308
+ // provider is the next account, and the exhausted one was just disabled so the retry cannot
309
+ // land on it again.
310
+ const shouldRetry = shouldRetryAttempt(comboPlan, err);
298
311
  attempt.statusCode = err.status;
299
312
  attempt.success = false;
300
313
  attempt.latencyMs = Date.now() - attemptStart;
@@ -312,6 +325,10 @@ export class GatewayRunner {
312
325
  }
313
326
  attempts.push(attempt);
314
327
  lastError = err;
328
+ // An exhausted account leaves the pool outright: without this the combo retried the same
329
+ // dead account on every request for hours (verified live) and answered "usage limited".
330
+ if (quotaFailure)
331
+ markQuotaExhausted(candidate, err.message);
315
332
  if (isUpstreamHealthFailure(err)) {
316
333
  recordFailure(provider.id, provider.cbFailureThreshold, provider.cbCooldownSeconds);
317
334
  if (candidate.codexAccountId)
@@ -971,7 +988,71 @@ export class GatewayRunner {
971
988
  export function isUpstreamHealthFailure(err) {
972
989
  return ['connection_error', 'connect_timeout', 'first_token_timeout', 'http_status', 'upstream_rate_limit'].includes(classifyFailure(err));
973
990
  }
991
+ /**
992
+ * An exhausted quota is a durable, per-account verdict — the account cannot serve anything again
993
+ * until its window resets, so it must leave the pool instead of being retried.
994
+ *
995
+ * Deliberately NOT part of `isUpstreamHealthFailure`: that path opens the provider's circuit
996
+ * breaker, which here would take down every sibling account of a healthy pool because one member
997
+ * ran dry. The exhaustion is applied account-by-account by `markQuotaExhausted` instead.
998
+ *
999
+ * Detection needs both signals because the two upstreams refuse differently, and neither reports a
1000
+ * status the router can trust alone:
1001
+ * - Codex answers 429 but the credential layer re-wraps it (status 502, cause 429), and a plain
1002
+ * rate limit is also a 429 — the "usage limit"/"quota" wording is what distinguishes "come back
1003
+ * in an hour" from "this account is spent".
1004
+ * - Qoder refuses with a billing envelope carrying status 403/code 112, surfaced as the
1005
+ * `qoder_billing_block` code.
1006
+ * Wording is only consulted when the status is quota-shaped, so a 400 that merely mentions the word
1007
+ * "quota" cannot disable a working account.
1008
+ */
1009
+ export function isQuotaFailure(err) {
1010
+ if (err.code === 'qoder_billing_block')
1011
+ return true;
1012
+ const cause = err.cause;
1013
+ const upstreamStatus = typeof cause?.status === 'number' ? cause.status : err.status;
1014
+ if (upstreamStatus !== 429 && upstreamStatus !== 403)
1015
+ return false;
1016
+ return /usage[_ -]?limit|quota|out of credits|insufficient[_ -]?(credit|balance)|exceeded your current quota|billing/i.test(err.message);
1017
+ }
1018
+ /**
1019
+ * Take an exhausted account out of the pool: mark it down and disable it.
1020
+ *
1021
+ * `enabled=0` (not just `health_state='down'`) is what the operator asked for and what actually
1022
+ * stops the churn: every selection path — `expandCodexAccountCandidates`,
1023
+ * `expandQoderAccountCandidates`, `getCodexAccountForProvider`, `findEligibleQoderAccount` — filters
1024
+ * on `enabled`, so disabling is the one write that makes the next request pick a different account.
1025
+ * The account stays visible in the admin UI, where re-enabling it after the window resets is a
1026
+ * one-click action.
1027
+ */
1028
+ export function markQuotaExhausted(candidate, message) {
1029
+ const reason = redactString(message);
1030
+ if (candidate.codexAccountId)
1031
+ setCodexAccountHealth(candidate.codexAccountId, 'down', reason, false);
1032
+ if (candidate.qoderAccountId)
1033
+ setQoderAccountHealth(candidate.qoderAccountId, 'down', reason, false);
1034
+ }
1035
+ /**
1036
+ * Whether a failed attempt should advance to the next candidate.
1037
+ *
1038
+ * A quota failure always advances, with or without a combo plan. For a pool provider the next
1039
+ * candidate is the next *account*, so requiring a combo plan left a direct `codex/…` or `qoder/…`
1040
+ * request pinned to the exhausted account — the reported bug ("does not switch to a new account,
1041
+ * just answers usage limited"). Safe because the exhausted account was just disabled: the retry
1042
+ * cannot land on it again. Everything else keeps the combo's configured triggers, where no combo
1043
+ * plan means no fallback.
1044
+ */
1045
+ export function shouldRetryAttempt(comboPlan, err) {
1046
+ if (isQuotaFailure(err))
1047
+ return true;
1048
+ return comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false;
1049
+ }
974
1050
  export function classifyFailure(err) {
1051
+ // Checked first, before the type switch: a quota refusal arrives as `upstream_rate_limit` (a
1052
+ // 429) or `upstream_error` (Codex rewraps it as a 502 with a 429 cause; Qoder uses code 112) and
1053
+ // must not fall through to `http_status`/`unknown`, neither of which is a routing decision.
1054
+ if (isQuotaFailure(err))
1055
+ return 'quota';
975
1056
  switch (err.type) {
976
1057
  case 'timeout_error':
977
1058
  return err.message.includes('first token') ? 'first_token_timeout' : 'connect_timeout';
@@ -16,6 +16,27 @@ function providerFor(providerId) {
16
16
  function accountFor(accountId) {
17
17
  return getRawDb().prepare('SELECT id,chatgpt_account_id FROM codex_accounts WHERE id=?').get(accountId) ?? null;
18
18
  }
19
+ /**
20
+ * Keep the quota snapshot honest while the account is in use.
21
+ *
22
+ * The usage API is a second upstream call, so this is throttled rather than run per request: a
23
+ * snapshot younger than the interval is left alone. Before this, usage only refreshed when an admin
24
+ * pressed the button or on the 10-minute autostart tick for opted-in accounts, so a dashboard could
25
+ * show hours-old consumption for an account actively serving traffic.
26
+ */
27
+ export const CODEX_USAGE_REFRESH_INTERVAL_MS = 5 * 60 * 1000;
28
+ export async function refreshCodexUsageIfStale(accountId, now = new Date()) {
29
+ const row = getRawDb().prepare('SELECT provider_id AS providerId,chatgpt_account_id AS chatgptAccountId,codex_usage_updated_at AS at FROM codex_accounts WHERE id=?').get(accountId);
30
+ if (!row)
31
+ return;
32
+ const provider = providerFor(row.providerId);
33
+ if (!provider)
34
+ return;
35
+ const last = row.at ? Date.parse(row.at) : NaN;
36
+ if (Number.isFinite(last) && now.getTime() - last < CODEX_USAGE_REFRESH_INTERVAL_MS)
37
+ return;
38
+ await refreshStoredCodexUsage(accountId, provider, { id: accountId, chatgpt_account_id: row.chatgptAccountId });
39
+ }
19
40
  /** Reads fresh usage; stores the snapshot or a sanitized reason on failure. */
20
41
  export async function refreshStoredCodexUsage(accountId, provider, account) {
21
42
  try {
@@ -106,7 +106,11 @@ export async function qoderAttemptFailure(accountRecordId, error) {
106
106
  // otherwise be classified as a rejected credential: that mislabels the cause, force-re-exchanges
107
107
  // a perfectly good PAT on every request, and hides the real answer (the account is out of quota).
108
108
  if (billing) {
109
- setQoderAccountHealth(accountRecordId, 'degraded', 'upstream reported a quota or billing block');
109
+ // Disabled, not merely degraded: a depleted account cannot serve paid models until its Credits
110
+ // refill, and leaving it eligible made the pool retry it on every request while the client saw
111
+ // "usage limited". The operator re-enables it (or adds Credits) from the admin UI; Credits
112
+ // refresh keeps the snapshot honest in the meantime.
113
+ setQoderAccountHealth(accountRecordId, 'down', 'out of Credits — re-enable after topping up', false);
110
114
  throw new GatewayError('upstream_error', 'Qoder account is out of quota', { status: 502, code: 'qoder_billing_block' });
111
115
  }
112
116
  if (status === 401 || status === 403) {
@@ -3,6 +3,8 @@
3
3
  // consumption. `is_free` on a catalog entry is the separate signal that a model is covered
4
4
  // by a promotion (e.g. Qwen3.8-Max) and therefore does not draw on Credits at all.
5
5
  import { QODER_CREDITS_URL } from './constants.js';
6
+ import { qoderCredentialsFor } from './credentials.js';
7
+ import { readQoderCreditsUpdatedAt, saveQoderCredits } from '../../db/repositories/qoder-accounts.js';
6
8
  const FETCH_TIMEOUT_MS = 10_000;
7
9
  const numberOr = (value, fallback) => {
8
10
  const parsed = Number(value);
@@ -70,3 +72,30 @@ export async function fetchQoderCredits(creds, freeModels, deps = {}) {
70
72
  clearTimeout(timer);
71
73
  }
72
74
  }
75
+ /**
76
+ * Fetch and persist one account's Credits snapshot. Free models come from the account's cached
77
+ * catalog because `is_free` is a catalog property, not a quota one — it is why an account at zero
78
+ * Credits can still serve a promotional model.
79
+ *
80
+ * Lives here rather than in the admin route so the gateway can reuse it: Credits are the only
81
+ * upstream truth about consumption (Qoder's chat stream reports no usage at all), so the snapshot
82
+ * has to be kept fresh as accounts are actually used, not only when an admin opens the page.
83
+ */
84
+ export async function refreshStoredQoderCredits(accountRecordId) {
85
+ const { config } = await qoderCredentialsFor(accountRecordId);
86
+ const freeModels = config.catalog ? [...config.catalog.entries.values()].filter((entry) => entry.isFree).map((entry) => entry.key) : [];
87
+ const credits = await fetchQoderCredits({ jobToken: config.jobToken }, freeModels, { timeoutMs: FETCH_TIMEOUT_MS });
88
+ saveQoderCredits(accountRecordId, credits.unavailable ? null : JSON.stringify(credits), credits.unavailable ?? null, credits.fetchedAt);
89
+ return credits;
90
+ }
91
+ /** Same throttle as Codex usage: the Credits endpoint is a second upstream call per account. */
92
+ export const QODER_CREDITS_REFRESH_INTERVAL_MS = 5 * 60 * 1000;
93
+ export async function refreshQoderCreditsIfStale(accountRecordId, now = new Date()) {
94
+ const row = readQoderCreditsUpdatedAt(accountRecordId);
95
+ const last = row ? Date.parse(row) : NaN;
96
+ if (Number.isFinite(last) && now.getTime() - last < QODER_CREDITS_REFRESH_INTERVAL_MS)
97
+ return;
98
+ // Never throws, matching `refreshStoredCodexUsage`: this runs in the background after a response
99
+ // has already been delivered, so a dead credential or unreachable quota API must not surface.
100
+ await refreshStoredQoderCredits(accountRecordId).catch(() => { });
101
+ }
@@ -274,6 +274,12 @@ export async function registerCodexRoutes(app) {
274
274
  values.push(typeof value === 'boolean' ? (value ? 1 : 0) : value);
275
275
  }
276
276
  }
277
+ // Re-enabling must also clear a quota verdict: every selection path excludes `health_state='down'`,
278
+ // so flipping `enabled` alone would leave the account unreachable while the UI showed it as on.
279
+ if (body.enabled === true) {
280
+ fields.push('health_state=?');
281
+ values.push('unknown');
282
+ }
277
283
  fields.push('updated_at=?');
278
284
  values.push(new Date().toISOString(), id);
279
285
  raw.prepare(`UPDATE codex_accounts SET ${fields.join(',')} WHERE id=?`).run(...values);
@@ -4,10 +4,10 @@ import { requireAdminAuth, requireAdminCsrf } from '../../auth/middleware.js';
4
4
  import { recordAudit } from '../../db/repositories/audit.js';
5
5
  import { getRawDb } from '../../db/index.js';
6
6
  import { uuid } from '../../auth/ids.js';
7
- import { listQoderAccountSummaries, getQoderAccountDetailById, setQoderAccountHealth, upsertQoderAccount, findQoderAccountForImport, saveQoderCatalog, saveQoderCredits, toQoderAccountSummary, } from '../../db/repositories/qoder-accounts.js';
7
+ import { listQoderAccountSummaries, getQoderAccountDetailById, setQoderAccountHealth, upsertQoderAccount, findQoderAccountForImport, saveQoderCatalog, toQoderAccountSummary, } from '../../db/repositories/qoder-accounts.js';
8
8
  import { exchangeQoderPat, fetchQoderCatalog, serializeCatalog } from '../../providers/qoder/catalog.js';
9
9
  import { probeQoderInference } from '../../providers/qoder/client.js';
10
- import { fetchQoderCredits } from '../../providers/qoder/credits.js';
10
+ import { refreshStoredQoderCredits } from '../../providers/qoder/credits.js';
11
11
  import { qoderCredentialsFor, withQoderCredentials } from '../../providers/qoder/credentials.js';
12
12
  import { parseQoderImportText, toQoderPreview, qoderTokenFingerprint } from '../../providers/qoder/qoder-import.js';
13
13
  import { redactString } from '../../security/redact.js';
@@ -87,17 +87,6 @@ function summaryOrThrow(id) {
87
87
  return toQoderAccountSummary(row);
88
88
  }
89
89
  /**
90
- * Fetch and persist the Credits snapshot. Free models come from the account's cached catalog
91
- * because `is_free` is a catalog property, not a quota one — it is why an account at zero
92
- * Credits can still serve a promotional model.
93
- */
94
- async function refreshQoderCredits(id, config) {
95
- const resolved = config ?? (await qoderCredentialsFor(id)).config;
96
- const freeModels = resolved.catalog ? [...resolved.catalog.entries.values()].filter((entry) => entry.isFree).map((entry) => entry.key) : [];
97
- const credits = await fetchQoderCredits({ jobToken: resolved.jobToken }, freeModels, { timeoutMs: 10_000 });
98
- saveQoderCredits(id, credits.unavailable ? null : JSON.stringify(credits), credits.unavailable ?? null, credits.fetchedAt);
99
- return credits;
100
- }
101
90
  /**
102
91
  * A PAT that works but a catalog that 502s is recoverable — the operator retries later — so the
103
92
  * exchange result is stored even when the catalog fetch fails. When the exchange itself fails
@@ -147,7 +136,7 @@ export async function registerQoderRoutes(app) {
147
136
  if (catalogError)
148
137
  setQoderAccountHealth(result.id, 'unknown', catalogError);
149
138
  // Best effort: an account that cannot report Credits is still a usable account.
150
- await refreshQoderCredits(result.id).catch(() => { });
139
+ await refreshStoredQoderCredits(result.id).catch(() => { });
151
140
  recordAudit({ action: 'qoder.accounts.add', success: true, targetType: 'qoder_account', targetId: result.id, targetName: record.label ?? undefined, ip: req.ip, metadata: { status: result.status, catalogError: catalogError ? 'yes' : 'no' } });
152
141
  return { account: summaryOrThrow(result.id), status: result.status, catalogError };
153
142
  });
@@ -247,6 +236,12 @@ export async function registerQoderRoutes(app) {
247
236
  values.push(typeof value === 'boolean' ? (value ? 1 : 0) : value);
248
237
  }
249
238
  }
239
+ // Re-enabling must also clear a quota verdict: every selection path excludes `health_state='down'`,
240
+ // so flipping `enabled` alone would leave the account unreachable while the UI showed it as on.
241
+ if (body.enabled === true) {
242
+ fields.push('health_state=?');
243
+ values.push('unknown');
244
+ }
250
245
  fields.push('updated_at=?');
251
246
  values.push(new Date().toISOString(), id);
252
247
  raw.prepare(`UPDATE qoder_accounts SET ${fields.join(',')} WHERE id=?`).run(...values);
@@ -281,7 +276,9 @@ export async function registerQoderRoutes(app) {
281
276
  if (!getQoderAccountDetailById(id))
282
277
  throw new GatewayError('invalid_request_error', 'Qoder account not found', { status: 404 });
283
278
  const { config } = await qoderCredentialsFor(id, { force: true });
284
- await refreshQoderCredits(id, config).catch(() => { });
279
+ // The exchange above was forced and persisted, so the refresh reuses that live job token
280
+ // rather than churning another one.
281
+ await refreshStoredQoderCredits(id).catch(() => { });
285
282
  const keys = config.catalog ? [...config.catalog.entries.keys()] : [];
286
283
  recordAudit({ action: 'qoder.accounts.catalog', success: Boolean(config.catalog), targetType: 'qoder_account', targetId: id, ip: req.ip, metadata: { modelCount: keys.length } });
287
284
  return { modelCount: keys.length, modelKeys: keys, catalogFetchedAt: config.catalog?.fetchedAt ?? null };
@@ -296,7 +293,7 @@ export async function registerQoderRoutes(app) {
296
293
  throw new GatewayError('invalid_request_error', 'Qoder account not found', { status: 404 });
297
294
  // Deliberately not forced: a fresh exchange invalidates the job token in flight, so reading
298
295
  // Credits must reuse the live one rather than churn it.
299
- const credits = await refreshQoderCredits(id);
296
+ const credits = await refreshStoredQoderCredits(id);
300
297
  recordAudit({ action: 'qoder.accounts.credits', success: !credits.unavailable, targetType: 'qoder_account', targetId: id, ip: req.ip, metadata: { freeModels: credits.freeModels.length } });
301
298
  return { account: summaryOrThrow(id) };
302
299
  });
@@ -127,6 +127,13 @@ export function shouldFallback(combo, reason) {
127
127
  return combo.trigger.connectTimeout;
128
128
  case 'first_token_timeout':
129
129
  return combo.trigger.firstTokenTimeout;
130
+ case 'quota':
131
+ // An exhausted account is the clearest case for trying the next candidate: retrying the same
132
+ // one cannot succeed until its window resets. Unconditional because a dead account is not a
133
+ // provider problem the admin's 429/5xx toggles were meant to gate — the account is already
134
+ // disabled by the runner, and stopping here would answer "usage limited" instead of routing
135
+ // to a healthy account.
136
+ return true;
130
137
  case 'http_status':
131
138
  if (reason.status === 408)
132
139
  return combo.trigger.on408;
@@ -75,8 +75,11 @@ export function providerToUpstreamConfig(p, codexAccountId) {
75
75
  export function upstreamHttpError(status, bodyExcerpt, cause) {
76
76
  if (status >= 500)
77
77
  return new GatewayError('upstream_error', `Upstream HTTP ${status}`, { status: 502, cause, code: `upstream_http_${status}` });
78
+ // Keep the body on a 429: it is the only place a provider says *why*, and the wording decides
79
+ // whether this is a transient throttle (retry) or an exhausted account (disable it and move on).
80
+ // `isQuotaFailure` matches on that wording. Callers pass an already-redacted excerpt.
78
81
  if (status === 429)
79
- return new GatewayError('upstream_rate_limit', `Upstream rate limited (HTTP ${status})`, { status: 429, cause, code: 'upstream_http_429' });
82
+ return new GatewayError('upstream_rate_limit', `Upstream rate limited (HTTP ${status})${bodyExcerpt ? `: ${bodyExcerpt}` : ''}`, { status: 429, cause, code: 'upstream_http_429' });
80
83
  if (status === 401 || status === 403)
81
84
  return new GatewayError('upstream_auth_error', `Upstream authentication failed (HTTP ${status})`, { status: 502, cause, code: `upstream_http_${status}` });
82
85
  return new GatewayError('upstream_error', `Upstream HTTP ${status}: ${bodyExcerpt}`, { status: 502, cause, code: `upstream_http_${status}` });
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "ldrouter",
3
- "version": "1.17.3",
3
+ "version": "1.17.4",
4
4
  "description": "LateDev Router — lightweight self-hosted LLM gateway with admin UI",
5
5
  "type": "module",
6
6
  "license": "MIT",