ldrouter 1.17.3 → 1.17.5

package/CHANGELOG.md

@@ -4,6 +4,28 @@ All notable changes to this project are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/) and the project adheres to
 [Semantic Versioning](https://semver.org/).
 
+## [1.17.5] - 2026-09-19
+
+### Fixed
+
+- **An account whose session was revoked kept being selected instead of leaving the Codex pool.** Auth0 answers `refresh_token_invalidated` ("Your session has ended. Please log in again.") for an account whose login was revoked; the account cannot serve anything again until it is re-imported. The credential layer surfaced that as a bare `oauth_refresh_failed`, which the router classified `unknown` — and `unknown` is not a routing decision, so `shouldFallback` answered false, the retry never advanced to a sibling account, and the request answered `502`. Live production data confirmed the shape: two accounts holding invalidated refresh tokens sat at the head of the pool, so every request landed on one of them, and 226 of 241 failed requests carried `attempts_count=1` while healthy accounts sat idle behind them. Credential failure is now its own failure class, checked before the error-type switch, and it always advances to the next candidate — with or without a combo plan, since a direct `codex1/...` model has no fallback triggers to consult.
+- **The account with the dead credential is now disabled, not merely degraded.** Marking down *and* disabling (`enabled=0`) is what actually stops the churn, mirroring quota exhaustion: every selection path — `expandCodexAccountCandidates`, `getCodexAccountForProvider` — filters on `enabled`. Deliberately kept out of `isUpstreamHealthFailure`, because one re-imported account must not open the provider circuit breaker for its healthy siblings.
+- **`codexCredentialError` discarded the raw error code.** Its wrapping message is deliberately generic, so without the cause the routing layer could not distinguish a dead account from any other `authentication_error`. The raw error is now carried as `cause`.
+
+## [1.17.4] - 2026-09-18
+
+### Fixed
+
+- **An account that ran out of quota kept being retried instead of leaving the pool.** A quota refusal was classified `unknown`, which is not a routing decision: `isUpstreamHealthFailure` excludes it and `shouldFallback` answers false for it, so the pool selected the same spent account on every request and answered "usage limited". Live production data confirmed the shape — 49 quota refusals (`Codex upstream HTTP 429: The usage limit has been reached`, `Qoder account is out of quota`) recorded over 14 days, every one of them `failure_reason=unknown`, with the same `codex_account_id` / `qoder_account_id` retried each time. Quota is now its own failure class, checked before the error-type switch (a 429 reaches the classifier as `upstream_rate_limit`, which previously returned `http_status` before the quota test could run).
+- **The exhausted account is now disabled, not just marked degraded.** Disabling (`enabled=0`) is the write that actually stops the churn: every selection path — `expandCodexAccountCandidates`, `expandQoderAccountCandidates`, `getCodexAccountForProvider`, `findEligibleQoderAccount` — filters on it, whereas `health_state` alone was overwritten to `healthy` by the next successful request. A Qoder billing block now disables the account too, instead of only degrading it.
+- **A direct model never advanced to the next account.** The retry was gated on `comboPlan ? ... : false`, so the reported case — a direct `qoder/...` model, logged 33 times in production — never retried at all. A quota failure now always advances to the next candidate, combo or not; that is safe precisely because the exhausted account was just disabled, so the retry cannot land on it again.
+- **The upstream 429 body was discarded, hiding the reason.** Quota exhaustion arrives as a plain 429 on OpenAI-compatible providers, with the wording ("you exceeded your current quota") only in the body — which `upstreamHttpError` dropped, leaving an account unclassifiable. The redacted body excerpt is now carried on a 429 too. A 429 without quota wording is still treated as a transient throttle: it retries the same account and does not disable it.
+- **Re-enabling an account in the admin UI did not put it back in the pool.** The enable toggle flipped `enabled` but left `health_state=down`, which every selection filter excludes, so the account stayed unroutable while the panel showed it as enabled. Both the Codex and Qoder routes now clear the health verdict when re-enabling.
+
+### Added
+
+- **Usage and Credits are refreshed as the pool is used.** Both the Codex usage snapshot and the Qoder credits snapshot are re-read in the background after a successful request, throttled per account so a burst of traffic costs at most one extra upstream call, and never awaited — the response path is unaffected.
+
 ## [1.17.3] - 2026-09-17
 
 ### Added

package/README.md

@@ -65,6 +65,8 @@ ZIP upload and automatic Codex CLI config-file generation/mutation are not inclu
 
 The account panel shows the 5-hour and weekly quota windows with reset countdowns, per-account and bulk usage refresh, weekly reset credits, an opt-in 5-hour window auto-start, and a **Test** control that probes the upstream account without exposing tokens. Routing order is set by dragging rows; the saved order is the fallback order the router uses. **Delete** is permanent and erases the stored encrypted credentials — past request logs are kept but lose the account reference.
 
+**A quota-exhausted account leaves the pool on its own.** When an account refuses for quota reasons — Codex `429 The usage limit has been reached`, or Qoder `out of credits` — it is marked `down` *and disabled*, so the very next request picks a different account in the pool rather than retrying the spent one. This applies to direct `codex/...` and `qoder/...` models too, not only combos. Re-enable the account in the panel once its window resets. Both Codex usage and Qoder credits are also refreshed in the background after a successful request (throttled, and never on the response path), so the panel figures track real usage without pressing **Refresh**.
+
 ### Qoder setup
 
 Qoder providers use a personal access token (PAT) instead of an API key. Create one at `https://qoder.com/account/integrations`; it starts with `pt-`.

package/dist/server/db/repositories/qoder-accounts.js

@@ -155,11 +155,16 @@ export function persistQoderJobToken(id, update) {
        health_state='healthy', last_error=NULL, updated_at=? WHERE id=?`)
         .run(jobToken.ciphertext, jobToken.nonce, jobToken.version, update.expiresAt, update.catalogJson ?? null, update.catalogFetchedAt ?? null, nextUpdatedAt(id), id);
 }
-export function setQoderAccountHealth(id, healthState, lastError = null) {
+/** `enabled` mirrors the Codex setter: pass `false` to take an exhausted account out of the pool. */
+export function setQoderAccountHealth(id, healthState, lastError = null, enabled) {
     const failures = healthState === 'healthy' ? 0 : null;
-    getRawDb()
-        .prepare(`UPDATE qoder_accounts SET health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), updated_at=? WHERE id=?`)
-        .run(healthState, lastError, failures, nextUpdatedAt(id), id);
+    const fields = enabled === undefined
+        ? 'health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), updated_at=?'
+        : 'health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), enabled=?, updated_at=?';
+    const values = enabled === undefined
+        ? [healthState, lastError, failures, nextUpdatedAt(id), id]
+        : [healthState, lastError, failures, enabled ? 1 : 0, nextUpdatedAt(id), id];
+    getRawDb().prepare(`UPDATE qoder_accounts SET ${fields} WHERE id=?`).run(...values);
 }
 export function readQoderCatalog(id) {
     const row = getRawDb()
@@ -172,6 +177,11 @@ export function saveQoderCatalog(id, catalogJson, fetchedAt) {
         .prepare('UPDATE qoder_accounts SET catalog_json=?, catalog_fetched_at=?, updated_at=? WHERE id=?')
         .run(catalogJson, fetchedAt ?? new Date().toISOString(), nextUpdatedAt(id), id);
 }
+/** When the Credits snapshot was last written; null when it never has been. */
+export function readQoderCreditsUpdatedAt(id) {
+    const row = getRawDb().prepare('SELECT credits_updated_at AS at FROM qoder_accounts WHERE id=?').get(id);
+    return row?.at ?? null;
+}
 /** Credits are observability, not routing state: a failed fetch is recorded without touching health. */
 export function saveQoderCredits(id, creditsJson, error, fetchedAt) {
     getRawDb()

package/dist/server/gateway/runner.js

@@ -8,7 +8,9 @@ import { loadCombo, selectCandidates, orderCandidates, shouldFallback, expandCod
 import { listCodexAccountsForProvider, setCodexAccountHealth } from '../db/repositories/codex-accounts.js';
 import { listQoderAccountsForProvider, setQoderAccountHealth } from '../db/repositories/qoder-accounts.js';
 import { qoderAttemptFailure, qoderCredentialsFor } from '../providers/qoder/credentials.js';
+import { refreshQoderCreditsIfStale } from '../providers/qoder/credits.js';
 import { callQoderNonStreaming, callQoderStreaming } from '../providers/qoder/client.js';
+import { refreshCodexUsageIfStale } from '../providers/codex-autostart.js';
 import { getEffectiveState, isOpen, recordSuccess, recordFailure, halfOpenProbeAllowed } from '../routing/circuit.js';
 import { checkRpm, checkTpm, acquireConcurrent, releaseConcurrent } from '../routing/ratelimit.js';
 import { checkDailyMonthly, consumeUsage } from '../routing/quota.js';
@@ -279,6 +281,13 @@ export class GatewayRunner {
                         setCodexAccountHealth(candidate.codexAccountId, 'healthy');
                     if (candidate.qoderAccountId)
                         setQoderAccountHealth(candidate.qoderAccountId, 'healthy');
+                    // Refresh the quota snapshot in the background so the dashboard reflects real usage.
+                    // Never awaited: it is a second upstream call, and the response is already complete.
+                    // Throttled inside each helper, so a burst of requests costs at most one refresh.
+                    if (candidate.codexAccountId)
+                        void refreshCodexUsageIfStale(candidate.codexAccountId).catch(() => { });
+                    if (candidate.qoderAccountId)
+                        void refreshQoderCreditsIfStale(candidate.qoderAccountId).catch(() => { });
                     getDb().update(schema.providers).set({ healthState: 'healthy', updatedAt: new Date().toISOString() }).where(eq(schema.providers.id, provider.id)).run();
                     attempts.push(attempt);
                     sentToClient = out.streamStarted ?? false;
@@ -286,15 +295,20 @@ export class GatewayRunner {
                 }
                 catch (e) {
                     const err = e instanceof GatewayError ? e : new GatewayError('upstream_error', e.message, { cause: e });
+                    const quotaFailure = isQuotaFailure(err);
+                    const credentialFailure = isCredentialFailure(err);
                     debugUpstream(ctx.requestId, 'ATTEMPT ERROR', [
                         `attempt=${i + 1}`,
                         `provider=${provider.name}`,
                         `model=${candidate.publicModelId}`,
                         `type=${err.type}`,
                         `status=${err.status}`,
-                        `willFallback=${comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false}`,
+                        `willFallback=${shouldRetryAttempt(comboPlan, err)}`,
                     ]);
-                    const shouldRetry = comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false;
+                    // Quota failures always advance (see shouldRetryAttempt): the next candidate for a pool
+                    // provider is the next account, and the exhausted one was just disabled so the retry cannot
+                    // land on it again.
+                    const shouldRetry = shouldRetryAttempt(comboPlan, err);
                     attempt.statusCode = err.status;
                     attempt.success = false;
                     attempt.latencyMs = Date.now() - attemptStart;
@@ -312,6 +326,15 @@ export class GatewayRunner {
                     }
                     attempts.push(attempt);
                     lastError = err;
+                    // An exhausted account leaves the pool outright: without this the combo retried the same
+                    // dead account on every request for hours (verified live) and answered "usage limited".
+                    if (quotaFailure)
+                        markQuotaExhausted(candidate, err.message);
+                    // A dead refresh token also leaves the pool outright: the account cannot serve anything
+                    // again until it is re-imported, so a retry may only be spent on a sibling account. Without
+                    // this the retry landed on the same dead account and the request answered 502.
+                    if (credentialFailure)
+                        markCredentialsDead(candidate, err.message);
                     if (isUpstreamHealthFailure(err)) {
                         recordFailure(provider.id, provider.cbFailureThreshold, provider.cbCooldownSeconds);
                         if (candidate.codexAccountId)
@@ -971,7 +994,110 @@ export class GatewayRunner {
 export function isUpstreamHealthFailure(err) {
     return ['connection_error', 'connect_timeout', 'first_token_timeout', 'http_status', 'upstream_rate_limit'].includes(classifyFailure(err));
 }
+/**
+ * An exhausted quota is a durable, per-account verdict — the account cannot serve anything again
+ * until its window resets, so it must leave the pool instead of being retried.
+ *
+ * Deliberately NOT part of `isUpstreamHealthFailure`: that path opens the provider's circuit
+ * breaker, which here would take down every sibling account of a healthy pool because one member
+ * ran dry. The exhaustion is applied account-by-account by `markQuotaExhausted` instead.
+ *
+ * Detection needs both signals because the two upstreams refuse differently, and neither reports a
+ * status the router can trust alone:
+ *  - Codex answers 429 but the credential layer re-wraps it (status 502, cause 429), and a plain
+ *    rate limit is also a 429 — the "usage limit"/"quota" wording is what distinguishes "come back
+ *    in an hour" from "this account is spent".
+ *  - Qoder refuses with a billing envelope carrying status 403/code 112, surfaced as the
+ *    `qoder_billing_block` code.
+ * Wording is only consulted when the status is quota-shaped, so a 400 that merely mentions the word
+ * "quota" cannot disable a working account.
+ */
+export function isQuotaFailure(err) {
+    if (err.code === 'qoder_billing_block')
+        return true;
+    const cause = err.cause;
+    const upstreamStatus = typeof cause?.status === 'number' ? cause.status : err.status;
+    if (upstreamStatus !== 429 && upstreamStatus !== 403)
+        return false;
+    return /usage[_ -]?limit|quota|out of credits|insufficient[_ -]?(credit|balance)|exceeded your current quota|billing/i.test(err.message);
+}
+/**
+ * Take an exhausted account out of the pool: mark it down and disable it.
+ *
+ * `enabled=0` (not just `health_state='down'`) is what the operator asked for and what actually
+ * stops the churn: every selection path — `expandCodexAccountCandidates`,
+ * `expandQoderAccountCandidates`, `getCodexAccountForProvider`, `findEligibleQoderAccount` — filters
+ * on `enabled`, so disabling is the one write that makes the next request pick a different account.
+ * The account stays visible in the admin UI, where re-enabling it after the window resets is a
+ * one-click action.
+ */
+export function markQuotaExhausted(candidate, message) {
+    const reason = redactString(message);
+    if (candidate.codexAccountId)
+        setCodexAccountHealth(candidate.codexAccountId, 'down', reason, false);
+    if (candidate.qoderAccountId)
+        setQoderAccountHealth(candidate.qoderAccountId, 'down', reason, false);
+}
+/**
+ * A credential failure is a durable, per-account verdict — the same class as quota exhaustion.
+ *
+ * `withCodexCredentials` throws these as bare codes when the account's refresh token is dead
+ * (Auth0 answers `refresh_token_invalidated`, "Your session has ended. Please log in again.").
+ * The account cannot serve anything again until it is re-imported, so retrying it is pure waste
+ * and leaving it in the pool answers 502 to every client.
+ *
+ * Live evidence (production, 2026-09-19): two accounts holding invalidated refresh tokens sat at
+ * the head of the pool, so every request landed on one of them — `oauth_refresh_failed`,
+ * classified `unknown`, `attempts_count=1`, 502 straight to the client (226 of 241 failed
+ * requests had exactly one attempt while healthy accounts sat idle behind them).
+ *
+ * Deliberately NOT part of `isUpstreamHealthFailure`: one re-imported account must not open the
+ * provider circuit breaker for its healthy siblings. The verdict is applied per account by
+ * `markCredentialsDead` instead.
+ *
+ * Both spellings are checked because the code travels either raw (from `withCodexCredentials`
+ * inside the gateway) or wrapped by `codexCredentialError`, which now keeps it as `cause`.
+ */
+export function isCredentialFailure(err) {
+    const codes = /^(oauth_refresh_failed|invalid_refresh_response|credential_unavailable|account_not_found)$/;
+    const cause = err.cause;
+    return codes.test(err.message) || (typeof cause?.message === 'string' && codes.test(cause.message));
+}
+/** Takes an account with a dead refresh token out of the pool. `enabled=0` is what stops the churn. */
+export function markCredentialsDead(candidate, message) {
+    const reason = redactString(message);
+    if (candidate.codexAccountId)
+        setCodexAccountHealth(candidate.codexAccountId, 'down', reason, false);
+    if (candidate.qoderAccountId)
+        setQoderAccountHealth(candidate.qoderAccountId, 'down', reason, false);
+}
+/**
+ * Whether a failed attempt should advance to the next candidate.
+ *
+ * A quota failure always advances, with or without a combo plan. For a pool provider the next
+ * candidate is the next *account*, so requiring a combo plan left a direct `codex/…` or `qoder/…`
+ * request pinned to the exhausted account — the reported bug ("does not switch to a new account,
+ * just answers usage limited"). Safe because the exhausted account was just disabled: the retry
+ * cannot land on it again. Everything else keeps the combo's configured triggers, where no combo
+ * plan means no fallback.
+ */
+export function shouldRetryAttempt(comboPlan, err) {
+    if (isCredentialFailure(err))
+        return true;
+    if (isQuotaFailure(err))
+        return true;
+    return comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false;
+}
 export function classifyFailure(err) {
+    // Checked before the type switch for the same reason as quota: a dead account is not a routing
+    // decision the combo's triggers were meant to gate, and `unknown` is not a routing decision at all.
+    if (isCredentialFailure(err))
+        return 'credential';
+    // Checked first, before the type switch: a quota refusal arrives as `upstream_rate_limit` (a
+    // 429) or `upstream_error` (Codex rewraps it as a 502 with a 429 cause; Qoder uses code 112) and
+    // must not fall through to `http_status`/`unknown`, neither of which is a routing decision.
+    if (isQuotaFailure(err))
+        return 'quota';
     switch (err.type) {
         case 'timeout_error':
             return err.message.includes('first token') ? 'first_token_timeout' : 'connect_timeout';

package/dist/server/providers/codex-autostart.js

@@ -16,6 +16,27 @@ function providerFor(providerId) {
 function accountFor(accountId) {
     return getRawDb().prepare('SELECT id,chatgpt_account_id FROM codex_accounts WHERE id=?').get(accountId) ?? null;
 }
+/**
+ * Keep the quota snapshot honest while the account is in use.
+ *
+ * The usage API is a second upstream call, so this is throttled rather than run per request: a
+ * snapshot younger than the interval is left alone. Before this, usage only refreshed when an admin
+ * pressed the button or on the 10-minute autostart tick for opted-in accounts, so a dashboard could
+ * show hours-old consumption for an account actively serving traffic.
+ */
+export const CODEX_USAGE_REFRESH_INTERVAL_MS = 5 * 60 * 1000;
+export async function refreshCodexUsageIfStale(accountId, now = new Date()) {
+    const row = getRawDb().prepare('SELECT provider_id AS providerId,chatgpt_account_id AS chatgptAccountId,codex_usage_updated_at AS at FROM codex_accounts WHERE id=?').get(accountId);
+    if (!row)
+        return;
+    const provider = providerFor(row.providerId);
+    if (!provider)
+        return;
+    const last = row.at ? Date.parse(row.at) : NaN;
+    if (Number.isFinite(last) && now.getTime() - last < CODEX_USAGE_REFRESH_INTERVAL_MS)
+        return;
+    await refreshStoredCodexUsage(accountId, provider, { id: accountId, chatgpt_account_id: row.chatgptAccountId });
+}
 /** Reads fresh usage; stores the snapshot or a sanitized reason on failure. */
 export async function refreshStoredCodexUsage(accountId, provider, account) {
     try {

package/dist/server/providers/codex-refresh.js

@@ -11,7 +11,9 @@ export function codexCredentialError(error) {
     if (code === 'account_not_found')
         return new GatewayError('invalid_request_error', 'Codex account not found', { status: 404 });
     if (code === 'oauth_refresh_failed' || code === 'invalid_refresh_response' || code === 'credential_unavailable') {
-        return new GatewayError('authentication_error', 'Codex credentials could not be refreshed — re-import the account', { status: 401 });
+        // `cause` carries the raw credential code: the wrapping message is deliberately generic, so
+        // without it the routing layer cannot tell a dead account from any other authentication_error.
+        return new GatewayError('authentication_error', 'Codex credentials could not be refreshed — re-import the account', { status: 401, cause: error });
     }
     return error;
 }

package/dist/server/providers/qoder/credentials.js

@@ -106,7 +106,11 @@ export async function qoderAttemptFailure(accountRecordId, error) {
     // otherwise be classified as a rejected credential: that mislabels the cause, force-re-exchanges
     // a perfectly good PAT on every request, and hides the real answer (the account is out of quota).
     if (billing) {
-        setQoderAccountHealth(accountRecordId, 'degraded', 'upstream reported a quota or billing block');
+        // Disabled, not merely degraded: a depleted account cannot serve paid models until its Credits
+        // refill, and leaving it eligible made the pool retry it on every request while the client saw
+        // "usage limited". The operator re-enables it (or adds Credits) from the admin UI; Credits
+        // refresh keeps the snapshot honest in the meantime.
+        setQoderAccountHealth(accountRecordId, 'down', 'out of Credits — re-enable after topping up', false);
         throw new GatewayError('upstream_error', 'Qoder account is out of quota', { status: 502, code: 'qoder_billing_block' });
     }
     if (status === 401 || status === 403) {

package/dist/server/providers/qoder/credits.js

@@ -3,6 +3,8 @@
 // consumption. `is_free` on a catalog entry is the separate signal that a model is covered
 // by a promotion (e.g. Qwen3.8-Max) and therefore does not draw on Credits at all.
 import { QODER_CREDITS_URL } from './constants.js';
+import { qoderCredentialsFor } from './credentials.js';
+import { readQoderCreditsUpdatedAt, saveQoderCredits } from '../../db/repositories/qoder-accounts.js';
 const FETCH_TIMEOUT_MS = 10_000;
 const numberOr = (value, fallback) => {
     const parsed = Number(value);
@@ -70,3 +72,30 @@ export async function fetchQoderCredits(creds, freeModels, deps = {}) {
         clearTimeout(timer);
     }
 }
+/**
+ * Fetch and persist one account's Credits snapshot. Free models come from the account's cached
+ * catalog because `is_free` is a catalog property, not a quota one — it is why an account at zero
+ * Credits can still serve a promotional model.
+ *
+ * Lives here rather than in the admin route so the gateway can reuse it: Credits are the only
+ * upstream truth about consumption (Qoder's chat stream reports no usage at all), so the snapshot
+ * has to be kept fresh as accounts are actually used, not only when an admin opens the page.
+ */
+export async function refreshStoredQoderCredits(accountRecordId) {
+    const { config } = await qoderCredentialsFor(accountRecordId);
+    const freeModels = config.catalog ? [...config.catalog.entries.values()].filter((entry) => entry.isFree).map((entry) => entry.key) : [];
+    const credits = await fetchQoderCredits({ jobToken: config.jobToken }, freeModels, { timeoutMs: FETCH_TIMEOUT_MS });
+    saveQoderCredits(accountRecordId, credits.unavailable ? null : JSON.stringify(credits), credits.unavailable ?? null, credits.fetchedAt);
+    return credits;
+}
+/** Same throttle as Codex usage: the Credits endpoint is a second upstream call per account. */
+export const QODER_CREDITS_REFRESH_INTERVAL_MS = 5 * 60 * 1000;
+export async function refreshQoderCreditsIfStale(accountRecordId, now = new Date()) {
+    const row = readQoderCreditsUpdatedAt(accountRecordId);
+    const last = row ? Date.parse(row) : NaN;
+    if (Number.isFinite(last) && now.getTime() - last < QODER_CREDITS_REFRESH_INTERVAL_MS)
+        return;
+    // Never throws, matching `refreshStoredCodexUsage`: this runs in the background after a response
+    // has already been delivered, so a dead credential or unreachable quota API must not surface.
+    await refreshStoredQoderCredits(accountRecordId).catch(() => { });
+}

package/dist/server/routes/admin/codex.js

@@ -274,6 +274,12 @@ export async function registerCodexRoutes(app) {
                 values.push(typeof value === 'boolean' ? (value ? 1 : 0) : value);
             }
         }
+        // Re-enabling must also clear a quota verdict: every selection path excludes `health_state='down'`,
+        // so flipping `enabled` alone would leave the account unreachable while the UI showed it as on.
+        if (body.enabled === true) {
+            fields.push('health_state=?');
+            values.push('unknown');
+        }
         fields.push('updated_at=?');
         values.push(new Date().toISOString(), id);
         raw.prepare(`UPDATE codex_accounts SET ${fields.join(',')} WHERE id=?`).run(...values);

package/dist/server/routes/admin/qoder.js

@@ -4,10 +4,10 @@ import { requireAdminAuth, requireAdminCsrf } from '../../auth/middleware.js';
 import { recordAudit } from '../../db/repositories/audit.js';
 import { getRawDb } from '../../db/index.js';
 import { uuid } from '../../auth/ids.js';
-import { listQoderAccountSummaries, getQoderAccountDetailById, setQoderAccountHealth, upsertQoderAccount, findQoderAccountForImport, saveQoderCatalog, saveQoderCredits, toQoderAccountSummary, } from '../../db/repositories/qoder-accounts.js';
+import { listQoderAccountSummaries, getQoderAccountDetailById, setQoderAccountHealth, upsertQoderAccount, findQoderAccountForImport, saveQoderCatalog, toQoderAccountSummary, } from '../../db/repositories/qoder-accounts.js';
 import { exchangeQoderPat, fetchQoderCatalog, serializeCatalog } from '../../providers/qoder/catalog.js';
 import { probeQoderInference } from '../../providers/qoder/client.js';
-import { fetchQoderCredits } from '../../providers/qoder/credits.js';
+import { refreshStoredQoderCredits } from '../../providers/qoder/credits.js';
 import { qoderCredentialsFor, withQoderCredentials } from '../../providers/qoder/credentials.js';
 import { parseQoderImportText, toQoderPreview, qoderTokenFingerprint } from '../../providers/qoder/qoder-import.js';
 import { redactString } from '../../security/redact.js';
@@ -87,17 +87,6 @@ function summaryOrThrow(id) {
     return toQoderAccountSummary(row);
 }
 /**
- * Fetch and persist the Credits snapshot. Free models come from the account's cached catalog
- * because `is_free` is a catalog property, not a quota one — it is why an account at zero
- * Credits can still serve a promotional model.
- */
-async function refreshQoderCredits(id, config) {
-    const resolved = config ?? (await qoderCredentialsFor(id)).config;
-    const freeModels = resolved.catalog ? [...resolved.catalog.entries.values()].filter((entry) => entry.isFree).map((entry) => entry.key) : [];
-    const credits = await fetchQoderCredits({ jobToken: resolved.jobToken }, freeModels, { timeoutMs: 10_000 });
-    saveQoderCredits(id, credits.unavailable ? null : JSON.stringify(credits), credits.unavailable ?? null, credits.fetchedAt);
-    return credits;
-}
 /**
  * A PAT that works but a catalog that 502s is recoverable — the operator retries later — so the
  * exchange result is stored even when the catalog fetch fails. When the exchange itself fails
@@ -147,7 +136,7 @@ export async function registerQoderRoutes(app) {
         if (catalogError)
             setQoderAccountHealth(result.id, 'unknown', catalogError);
         // Best effort: an account that cannot report Credits is still a usable account.
-        await refreshQoderCredits(result.id).catch(() => { });
+        await refreshStoredQoderCredits(result.id).catch(() => { });
         recordAudit({ action: 'qoder.accounts.add', success: true, targetType: 'qoder_account', targetId: result.id, targetName: record.label ?? undefined, ip: req.ip, metadata: { status: result.status, catalogError: catalogError ? 'yes' : 'no' } });
         return { account: summaryOrThrow(result.id), status: result.status, catalogError };
     });
@@ -247,6 +236,12 @@ export async function registerQoderRoutes(app) {
                 values.push(typeof value === 'boolean' ? (value ? 1 : 0) : value);
             }
         }
+        // Re-enabling must also clear a quota verdict: every selection path excludes `health_state='down'`,
+        // so flipping `enabled` alone would leave the account unreachable while the UI showed it as on.
+        if (body.enabled === true) {
+            fields.push('health_state=?');
+            values.push('unknown');
+        }
         fields.push('updated_at=?');
         values.push(new Date().toISOString(), id);
         raw.prepare(`UPDATE qoder_accounts SET ${fields.join(',')} WHERE id=?`).run(...values);
@@ -281,7 +276,9 @@ export async function registerQoderRoutes(app) {
         if (!getQoderAccountDetailById(id))
             throw new GatewayError('invalid_request_error', 'Qoder account not found', { status: 404 });
         const { config } = await qoderCredentialsFor(id, { force: true });
-        await refreshQoderCredits(id, config).catch(() => { });
+        // The exchange above was forced and persisted, so the refresh reuses that live job token
+        // rather than churning another one.
+        await refreshStoredQoderCredits(id).catch(() => { });
         const keys = config.catalog ? [...config.catalog.entries.keys()] : [];
         recordAudit({ action: 'qoder.accounts.catalog', success: Boolean(config.catalog), targetType: 'qoder_account', targetId: id, ip: req.ip, metadata: { modelCount: keys.length } });
         return { modelCount: keys.length, modelKeys: keys, catalogFetchedAt: config.catalog?.fetchedAt ?? null };
@@ -296,7 +293,7 @@ export async function registerQoderRoutes(app) {
             throw new GatewayError('invalid_request_error', 'Qoder account not found', { status: 404 });
         // Deliberately not forced: a fresh exchange invalidates the job token in flight, so reading
         // Credits must reuse the live one rather than churn it.
-        const credits = await refreshQoderCredits(id);
+        const credits = await refreshStoredQoderCredits(id);
         recordAudit({ action: 'qoder.accounts.credits', success: !credits.unavailable, targetType: 'qoder_account', targetId: id, ip: req.ip, metadata: { freeModels: credits.freeModels.length } });
         return { account: summaryOrThrow(id) };
     });

package/dist/server/routing/combo.js

@@ -127,6 +127,13 @@ export function shouldFallback(combo, reason) {
             return combo.trigger.connectTimeout;
         case 'first_token_timeout':
             return combo.trigger.firstTokenTimeout;
+        case 'quota':
+            // An exhausted account is the clearest case for trying the next candidate: retrying the same
+            // one cannot succeed until its window resets. Unconditional because a dead account is not a
+            // provider problem the admin's 429/5xx toggles were meant to gate — the account is already
+            // disabled by the runner, and stopping here would answer "usage limited" instead of routing
+            // to a healthy account.
+            return true;
         case 'http_status':
             if (reason.status === 408)
                 return combo.trigger.on408;

package/dist/server/upstream/client.js

@@ -75,8 +75,11 @@ export function providerToUpstreamConfig(p, codexAccountId) {
 export function upstreamHttpError(status, bodyExcerpt, cause) {
     if (status >= 500)
         return new GatewayError('upstream_error', `Upstream HTTP ${status}`, { status: 502, cause, code: `upstream_http_${status}` });
+    // Keep the body on a 429: it is the only place a provider says *why*, and the wording decides
+    // whether this is a transient throttle (retry) or an exhausted account (disable it and move on).
+    // `isQuotaFailure` matches on that wording. Callers pass an already-redacted excerpt.
     if (status === 429)
-        return new GatewayError('upstream_rate_limit', `Upstream rate limited (HTTP ${status})`, { status: 429, cause, code: 'upstream_http_429' });
+        return new GatewayError('upstream_rate_limit', `Upstream rate limited (HTTP ${status})${bodyExcerpt ? `: ${bodyExcerpt}` : ''}`, { status: 429, cause, code: 'upstream_http_429' });
     if (status === 401 || status === 403)
         return new GatewayError('upstream_auth_error', `Upstream authentication failed (HTTP ${status})`, { status: 502, cause, code: `upstream_http_${status}` });
     return new GatewayError('upstream_error', `Upstream HTTP ${status}: ${bodyExcerpt}`, { status: 502, cause, code: `upstream_http_${status}` });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ldrouter",
-  "version": "1.17.3",
+  "version": "1.17.5",
   "description": "LateDev Router — lightweight self-hosted LLM gateway with admin UI",
   "type": "module",
   "license": "MIT",