ldrouter 1.17.2 → 1.17.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -4,6 +4,32 @@ All notable changes to this project are documented here. The format follows
4
4
  [Keep a Changelog](https://keepachangelog.com/) and the project adheres to
5
5
  [Semantic Versioning](https://semver.org/).
6
6
 
7
+ ## [1.17.4] - 2026-09-18
8
+
9
+ ### Fixed
10
+
11
+ - **An account that ran out of quota kept being retried instead of leaving the pool.** A quota refusal was classified `unknown`, which is not a routing decision: `isUpstreamHealthFailure` excludes it and `shouldFallback` answers false for it, so the pool selected the same spent account on every request and answered "usage limited". Live production data confirmed the shape — 49 quota refusals (`Codex upstream HTTP 429: The usage limit has been reached`, `Qoder account is out of quota`) recorded over 14 days, every one of them `failure_reason=unknown`, with the same `codex_account_id` / `qoder_account_id` retried each time. Quota is now its own failure class, checked before the error-type switch (a 429 reaches the classifier as `upstream_rate_limit`, which previously returned `http_status` before the quota test could run).
12
+ - **The exhausted account is now disabled, not just marked degraded.** Disabling (`enabled=0`) is the write that actually stops the churn: every selection path — `expandCodexAccountCandidates`, `expandQoderAccountCandidates`, `getCodexAccountForProvider`, `findEligibleQoderAccount` — filters on it, whereas `health_state` alone was overwritten to `healthy` by the next successful request. A Qoder billing block now disables the account too, instead of only degrading it.
13
+ - **A direct model never advanced to the next account.** The retry was gated on `comboPlan ? ... : false`, so the reported case — a direct `qoder/...` model, logged 33 times in production — never retried at all. A quota failure now always advances to the next candidate, combo or not; that is safe precisely because the exhausted account was just disabled, so the retry cannot land on it again.
14
+ - **The upstream 429 body was discarded, hiding the reason.** Quota exhaustion arrives as a plain 429 on OpenAI-compatible providers, with the wording ("you exceeded your current quota") only in the body — which `upstreamHttpError` dropped, leaving an account unclassifiable. The redacted body excerpt is now carried on a 429 too. A 429 without quota wording is still treated as a transient throttle: it retries the same account and does not disable it.
15
+ - **Re-enabling an account in the admin UI did not put it back in the pool.** The enable toggle flipped `enabled` but left `health_state=down`, which every selection filter excludes, so the account stayed unroutable while the panel showed it as enabled. Both the Codex and Qoder routes now clear the health verdict when re-enabling.
16
+
17
+ ### Added
18
+
19
+ - **Usage and Credits are refreshed as the pool is used.** Both the Codex usage snapshot and the Qoder credits snapshot are re-read in the background after a successful request, throttled per account so a burst of traffic costs at most one extra upstream call, and never awaited — the response path is unaffected.
20
+
21
+ ## [1.17.3] - 2026-09-17
22
+
23
+ ### Added
24
+
25
+ - **Qoder accounts now show Credits — the account's real usage.** Qoder does not bill in tokens: its chat stream carries no usage block at all, so every Qoder request logged zero tokens and the panel had no usage figure to show. Credits are now read from Qoder's quota API (`GET /api/v2/quota/usage`) and shown per account: plan / add-on / org buckets, percentage used, expiry, and whether the quota is exhausted. A **Refresh credits** action re-reads them for every account.
26
+ - **Promotion-covered models are shown next to Credits.** Qoder keeps a model free by marking its catalog entry `is_free` — currently `qmodel_38max` (Qwen3.8-Max). Such a model spends no Credits, so an account at zero Credits still answers on it while every other model is refused. The panel now pairs **No plan credits** with **Free now: <model>** so a partly-working account reads as intended instead of looking like a router fault.
27
+
28
+ ### Fixed
29
+
30
+ - **A Qoder quota refusal was reported as a token rejection.** The upstream returns HTTP 200 with a `403 code 112` envelope inside the streamed body for quota/billing refusals; the attempt classifier checked `401/403` before billing, so a depleted account was labelled "job token was rejected after a refresh" and the account was force-re-exchanged on every request. Billing-shaped refusals are no longer mistaken for credential failures.
31
+ - **The account Test button reported a blocked account as working.** It only probed the model catalog, which proves the PAT exchanges and nothing else — the refusal arrives as a `403` envelope inside an HTTP 200 body, so a check that never sends a message cannot see it. Test now sends one real request, preferring a promotion-covered model so testing never spends Credits, and names the free models that still work when the account is out of them.
32
+
7
33
  ## [1.17.2] - 2026-09-17
8
34
 
9
35
  ### Fixed
package/README.md CHANGED
@@ -65,6 +65,8 @@ ZIP upload and automatic Codex CLI config-file generation/mutation are not inclu
65
65
 
66
66
  The account panel shows the 5-hour and weekly quota windows with reset countdowns, per-account and bulk usage refresh, weekly reset credits, an opt-in 5-hour window auto-start, and a **Test** control that probes the upstream account without exposing tokens. Routing order is set by dragging rows; the saved order is the fallback order the router uses. **Delete** is permanent and erases the stored encrypted credentials — past request logs are kept but lose the account reference.
67
67
 
68
+ **A quota-exhausted account leaves the pool on its own.** When an account refuses for quota reasons — Codex `429 The usage limit has been reached`, or Qoder `out of credits` — it is marked `down` *and disabled*, so the very next request picks a different account in the pool rather than retrying the spent one. This applies to direct `codex/...` and `qoder/...` models too, not only combos. Re-enable the account in the panel once its window resets. Both Codex usage and Qoder credits are also refreshed in the background after a successful request (throttled, and never on the response path), so the panel figures track real usage without pressing **Refresh**.
69
+
68
70
  ### Qoder setup
69
71
 
70
72
  Qoder providers use a personal access token (PAT) instead of an API key. Create one at `https://qoder.com/account/integrations`; it starts with `pt-`.
@@ -81,6 +83,30 @@ Because the gateway serves model discovery from that live catalog, a new account
81
83
 
82
84
  The panel shows the masked user id, job-token expiry, when the catalog was last fetched, health, and an enabled toggle. Routing order is set by dragging rows, exactly as with Codex.
83
85
 
86
+ ### Qoder usage, Credits and promotions
87
+
88
+ Qoder does not meter models in tokens: its chat stream carries no usage block at all, so request logs
89
+ show zero tokens for every Qoder model and the panel's **Credits** column is the account's real
90
+ usage. It is read from Qoder's quota API (`GET /api/v2/quota/usage` on `openapi.qoder.sh`) and shows
91
+ the plan / add-on / org buckets, the percentage used, and whether the account is exhausted.
92
+
93
+ **Credits and free models are different things, and the panel shows both.** Qoder keeps promoting a
94
+ model — currently `qmodel_38max`, the Qwen3.8-Max route — by marking its catalog entry `is_free`.
95
+ Such a model spends no Credits, so an account at **zero Credits** still answers on it while every
96
+ other model returns an envelope `403 code 112` with a `pricingUrl`. That pairing is what makes an
97
+ "out of credits" account look partly broken: the panel labels the column **No plan credits** next to
98
+ **Free now: qmodel_38max** so the contradiction is visible instead of looking like a router fault.
99
+
100
+ There is an upstream edge worth knowing when reading logs: a quota refusal arrives as **HTTP 200**
101
+ with a `403` envelope inside the streamed body, so it is not an HTTP error at the transport layer.
102
+
103
+ Use **Refresh credits** to re-read the snapshot for every account. It deliberately reuses the live
104
+ `job token` instead of exchanging the PAT again — a fresh exchange invalidates the token a
105
+ concurrent request may be using.
106
+
107
+ **Test** sends one real message, because a catalog probe cannot see a quota refusal. It picks a
108
+ promotion-covered model when the account has one, so testing never spends Credits.
109
+
84
110
  ## Environment variables
85
111
 
86
112
  | Variable | Description | Default |
@@ -5,6 +5,18 @@ import { decryptSecret, encryptSecret } from '../../auth/crypto.js';
5
5
  import { uuid } from '../../auth/ids.js';
6
6
  import { createHash } from 'node:crypto';
7
7
  import { getRawDb } from '../index.js';
8
+ /** A corrupted or absent snapshot reads as "no data", never as a thrown error on a list call. */
9
+ function parseStoredCredits(json) {
10
+ if (!json)
11
+ return null;
12
+ try {
13
+ const parsed = JSON.parse(json);
14
+ return parsed && typeof parsed === 'object' ? parsed : null;
15
+ }
16
+ catch {
17
+ return null;
18
+ }
19
+ }
8
20
  export function maskQoderValue(value) {
9
21
  if (!value)
10
22
  return null;
@@ -18,7 +30,7 @@ export function identityFromQoderRecord(record) {
18
30
  label: record.label,
19
31
  };
20
32
  }
21
- const SUMMARY_COLUMNS = 'id,provider_id AS providerId,label,email,qoder_user_id AS qoderUserId,machine_id AS machineId,enabled,health_state AS healthState,priority,job_token_expires_at AS jobTokenExpiresAt,catalog_fetched_at AS catalogFetchedAt,last_error AS lastError,consecutive_failures AS consecutiveFailures,created_at AS createdAt,updated_at AS updatedAt';
33
+ const SUMMARY_COLUMNS = 'id,provider_id AS providerId,label,email,qoder_user_id AS qoderUserId,machine_id AS machineId,enabled,health_state AS healthState,priority,job_token_expires_at AS jobTokenExpiresAt,catalog_fetched_at AS catalogFetchedAt,credits_json AS creditsJson,credits_updated_at AS creditsUpdatedAt,credits_error AS creditsError,last_error AS lastError,consecutive_failures AS consecutiveFailures,created_at AS createdAt,updated_at AS updatedAt';
22
34
  export function toQoderAccountSummary(row) {
23
35
  return {
24
36
  id: row.id,
@@ -31,6 +43,9 @@ export function toQoderAccountSummary(row) {
31
43
  priority: row.priority,
32
44
  jobTokenExpiresAt: row.jobTokenExpiresAt,
33
45
  catalogFetchedAt: row.catalogFetchedAt,
46
+ credits: (row.credits ?? null) || parseStoredCredits(row.creditsJson ?? null),
47
+ creditsUpdatedAt: row.creditsUpdatedAt ?? null,
48
+ creditsError: row.creditsError ?? null,
34
49
  lastError: row.lastError,
35
50
  consecutiveFailures: row.consecutiveFailures,
36
51
  createdAt: row.createdAt,
@@ -140,11 +155,16 @@ export function persistQoderJobToken(id, update) {
140
155
  health_state='healthy', last_error=NULL, updated_at=? WHERE id=?`)
141
156
  .run(jobToken.ciphertext, jobToken.nonce, jobToken.version, update.expiresAt, update.catalogJson ?? null, update.catalogFetchedAt ?? null, nextUpdatedAt(id), id);
142
157
  }
143
- export function setQoderAccountHealth(id, healthState, lastError = null) {
158
+ /** `enabled` mirrors the Codex setter: pass `false` to take an exhausted account out of the pool. */
159
+ export function setQoderAccountHealth(id, healthState, lastError = null, enabled) {
144
160
  const failures = healthState === 'healthy' ? 0 : null;
145
- getRawDb()
146
- .prepare(`UPDATE qoder_accounts SET health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), updated_at=? WHERE id=?`)
147
- .run(healthState, lastError, failures, nextUpdatedAt(id), id);
161
+ const fields = enabled === undefined
162
+ ? 'health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), updated_at=?'
163
+ : 'health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), enabled=?, updated_at=?';
164
+ const values = enabled === undefined
165
+ ? [healthState, lastError, failures, nextUpdatedAt(id), id]
166
+ : [healthState, lastError, failures, enabled ? 1 : 0, nextUpdatedAt(id), id];
167
+ getRawDb().prepare(`UPDATE qoder_accounts SET ${fields} WHERE id=?`).run(...values);
148
168
  }
149
169
  export function readQoderCatalog(id) {
150
170
  const row = getRawDb()
@@ -157,6 +177,17 @@ export function saveQoderCatalog(id, catalogJson, fetchedAt) {
157
177
  .prepare('UPDATE qoder_accounts SET catalog_json=?, catalog_fetched_at=?, updated_at=? WHERE id=?')
158
178
  .run(catalogJson, fetchedAt ?? new Date().toISOString(), nextUpdatedAt(id), id);
159
179
  }
180
+ /** When the Credits snapshot was last written; null when it never has been. */
181
+ export function readQoderCreditsUpdatedAt(id) {
182
+ const row = getRawDb().prepare('SELECT credits_updated_at AS at FROM qoder_accounts WHERE id=?').get(id);
183
+ return row?.at ?? null;
184
+ }
185
+ /** Credits are observability, not routing state: a failed fetch is recorded without touching health. */
186
+ export function saveQoderCredits(id, creditsJson, error, fetchedAt) {
187
+ getRawDb()
188
+ .prepare('UPDATE qoder_accounts SET credits_json=COALESCE(?, credits_json), credits_updated_at=?, credits_error=?, updated_at=? WHERE id=?')
189
+ .run(creditsJson, fetchedAt, error, nextUpdatedAt(id), id);
190
+ }
160
191
  // Keeps `updated_at` strictly increasing even within the same millisecond, so a caller can
161
192
  // detect a write it just made. Copied from the Codex repository for consistency.
162
193
  function nextUpdatedAt(id) {
@@ -8,7 +8,9 @@ import { loadCombo, selectCandidates, orderCandidates, shouldFallback, expandCod
8
8
  import { listCodexAccountsForProvider, setCodexAccountHealth } from '../db/repositories/codex-accounts.js';
9
9
  import { listQoderAccountsForProvider, setQoderAccountHealth } from '../db/repositories/qoder-accounts.js';
10
10
  import { qoderAttemptFailure, qoderCredentialsFor } from '../providers/qoder/credentials.js';
11
+ import { refreshQoderCreditsIfStale } from '../providers/qoder/credits.js';
11
12
  import { callQoderNonStreaming, callQoderStreaming } from '../providers/qoder/client.js';
13
+ import { refreshCodexUsageIfStale } from '../providers/codex-autostart.js';
12
14
  import { getEffectiveState, isOpen, recordSuccess, recordFailure, halfOpenProbeAllowed } from '../routing/circuit.js';
13
15
  import { checkRpm, checkTpm, acquireConcurrent, releaseConcurrent } from '../routing/ratelimit.js';
14
16
  import { checkDailyMonthly, consumeUsage } from '../routing/quota.js';
@@ -279,6 +281,13 @@ export class GatewayRunner {
279
281
  setCodexAccountHealth(candidate.codexAccountId, 'healthy');
280
282
  if (candidate.qoderAccountId)
281
283
  setQoderAccountHealth(candidate.qoderAccountId, 'healthy');
284
+ // Refresh the quota snapshot in the background so the dashboard reflects real usage.
285
+ // Never awaited: it is a second upstream call, and the response is already complete.
286
+ // Throttled inside each helper, so a burst of requests costs at most one refresh.
287
+ if (candidate.codexAccountId)
288
+ void refreshCodexUsageIfStale(candidate.codexAccountId).catch(() => { });
289
+ if (candidate.qoderAccountId)
290
+ void refreshQoderCreditsIfStale(candidate.qoderAccountId).catch(() => { });
282
291
  getDb().update(schema.providers).set({ healthState: 'healthy', updatedAt: new Date().toISOString() }).where(eq(schema.providers.id, provider.id)).run();
283
292
  attempts.push(attempt);
284
293
  sentToClient = out.streamStarted ?? false;
@@ -286,15 +295,19 @@ export class GatewayRunner {
286
295
  }
287
296
  catch (e) {
288
297
  const err = e instanceof GatewayError ? e : new GatewayError('upstream_error', e.message, { cause: e });
298
+ const quotaFailure = isQuotaFailure(err);
289
299
  debugUpstream(ctx.requestId, 'ATTEMPT ERROR', [
290
300
  `attempt=${i + 1}`,
291
301
  `provider=${provider.name}`,
292
302
  `model=${candidate.publicModelId}`,
293
303
  `type=${err.type}`,
294
304
  `status=${err.status}`,
295
- `willFallback=${comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false}`,
305
+ `willFallback=${shouldRetryAttempt(comboPlan, err)}`,
296
306
  ]);
297
- const shouldRetry = comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false;
307
+ // Quota failures always advance (see shouldRetryAttempt): the next candidate for a pool
308
+ // provider is the next account, and the exhausted one was just disabled so the retry cannot
309
+ // land on it again.
310
+ const shouldRetry = shouldRetryAttempt(comboPlan, err);
298
311
  attempt.statusCode = err.status;
299
312
  attempt.success = false;
300
313
  attempt.latencyMs = Date.now() - attemptStart;
@@ -312,6 +325,10 @@ export class GatewayRunner {
312
325
  }
313
326
  attempts.push(attempt);
314
327
  lastError = err;
328
+ // An exhausted account leaves the pool outright: without this the combo retried the same
329
+ // dead account on every request for hours (verified live) and answered "usage limited".
330
+ if (quotaFailure)
331
+ markQuotaExhausted(candidate, err.message);
315
332
  if (isUpstreamHealthFailure(err)) {
316
333
  recordFailure(provider.id, provider.cbFailureThreshold, provider.cbCooldownSeconds);
317
334
  if (candidate.codexAccountId)
@@ -971,7 +988,71 @@ export class GatewayRunner {
971
988
  export function isUpstreamHealthFailure(err) {
972
989
  return ['connection_error', 'connect_timeout', 'first_token_timeout', 'http_status', 'upstream_rate_limit'].includes(classifyFailure(err));
973
990
  }
991
+ /**
992
+ * An exhausted quota is a durable, per-account verdict — the account cannot serve anything again
993
+ * until its window resets, so it must leave the pool instead of being retried.
994
+ *
995
+ * Deliberately NOT part of `isUpstreamHealthFailure`: that path opens the provider's circuit
996
+ * breaker, which here would take down every sibling account of a healthy pool because one member
997
+ * ran dry. The exhaustion is applied account-by-account by `markQuotaExhausted` instead.
998
+ *
999
+ * Detection needs both signals because the two upstreams refuse differently, and neither reports a
1000
+ * status the router can trust alone:
1001
+ * - Codex answers 429 but the credential layer re-wraps it (status 502, cause 429), and a plain
1002
+ * rate limit is also a 429 — the "usage limit"/"quota" wording is what distinguishes "come back
1003
+ * in an hour" from "this account is spent".
1004
+ * - Qoder refuses with a billing envelope carrying status 403/code 112, surfaced as the
1005
+ * `qoder_billing_block` code.
1006
+ * Wording is only consulted when the status is quota-shaped, so a 400 that merely mentions the word
1007
+ * "quota" cannot disable a working account.
1008
+ */
1009
+ export function isQuotaFailure(err) {
1010
+ if (err.code === 'qoder_billing_block')
1011
+ return true;
1012
+ const cause = err.cause;
1013
+ const upstreamStatus = typeof cause?.status === 'number' ? cause.status : err.status;
1014
+ if (upstreamStatus !== 429 && upstreamStatus !== 403)
1015
+ return false;
1016
+ return /usage[_ -]?limit|quota|out of credits|insufficient[_ -]?(credit|balance)|exceeded your current quota|billing/i.test(err.message);
1017
+ }
1018
+ /**
1019
+ * Take an exhausted account out of the pool: mark it down and disable it.
1020
+ *
1021
+ * `enabled=0` (not just `health_state='down'`) is what the operator asked for and what actually
1022
+ * stops the churn: every selection path — `expandCodexAccountCandidates`,
1023
+ * `expandQoderAccountCandidates`, `getCodexAccountForProvider`, `findEligibleQoderAccount` — filters
1024
+ * on `enabled`, so disabling is the one write that makes the next request pick a different account.
1025
+ * The account stays visible in the admin UI, where re-enabling it after the window resets is a
1026
+ * one-click action.
1027
+ */
1028
+ export function markQuotaExhausted(candidate, message) {
1029
+ const reason = redactString(message);
1030
+ if (candidate.codexAccountId)
1031
+ setCodexAccountHealth(candidate.codexAccountId, 'down', reason, false);
1032
+ if (candidate.qoderAccountId)
1033
+ setQoderAccountHealth(candidate.qoderAccountId, 'down', reason, false);
1034
+ }
1035
+ /**
1036
+ * Whether a failed attempt should advance to the next candidate.
1037
+ *
1038
+ * A quota failure always advances, with or without a combo plan. For a pool provider the next
1039
+ * candidate is the next *account*, so requiring a combo plan left a direct `codex/…` or `qoder/…`
1040
+ * request pinned to the exhausted account — the reported bug ("does not switch to a new account,
1041
+ * just answers usage limited"). Safe because the exhausted account was just disabled: the retry
1042
+ * cannot land on it again. Everything else keeps the combo's configured triggers, where no combo
1043
+ * plan means no fallback.
1044
+ */
1045
+ export function shouldRetryAttempt(comboPlan, err) {
1046
+ if (isQuotaFailure(err))
1047
+ return true;
1048
+ return comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false;
1049
+ }
974
1050
  export function classifyFailure(err) {
1051
+ // Checked first, before the type switch: a quota refusal arrives as `upstream_rate_limit` (a
1052
+ // 429) or `upstream_error` (Codex rewraps it as a 502 with a 429 cause; Qoder uses code 112) and
1053
+ // must not fall through to `http_status`/`unknown`, neither of which is a routing decision.
1054
+ if (isQuotaFailure(err))
1055
+ return 'quota';
975
1056
  switch (err.type) {
976
1057
  case 'timeout_error':
977
1058
  return err.message.includes('first token') ? 'first_token_timeout' : 'connect_timeout';
@@ -16,6 +16,27 @@ function providerFor(providerId) {
16
16
  function accountFor(accountId) {
17
17
  return getRawDb().prepare('SELECT id,chatgpt_account_id FROM codex_accounts WHERE id=?').get(accountId) ?? null;
18
18
  }
19
+ /**
20
+ * Keep the quota snapshot honest while the account is in use.
21
+ *
22
+ * The usage API is a second upstream call, so this is throttled rather than run per request: a
23
+ * snapshot younger than the interval is left alone. Before this, usage only refreshed when an admin
24
+ * pressed the button or on the 10-minute autostart tick for opted-in accounts, so a dashboard could
25
+ * show hours-old consumption for an account actively serving traffic.
26
+ */
27
+ export const CODEX_USAGE_REFRESH_INTERVAL_MS = 5 * 60 * 1000;
28
+ export async function refreshCodexUsageIfStale(accountId, now = new Date()) {
29
+ const row = getRawDb().prepare('SELECT provider_id AS providerId,chatgpt_account_id AS chatgptAccountId,codex_usage_updated_at AS at FROM codex_accounts WHERE id=?').get(accountId);
30
+ if (!row)
31
+ return;
32
+ const provider = providerFor(row.providerId);
33
+ if (!provider)
34
+ return;
35
+ const last = row.at ? Date.parse(row.at) : NaN;
36
+ if (Number.isFinite(last) && now.getTime() - last < CODEX_USAGE_REFRESH_INTERVAL_MS)
37
+ return;
38
+ await refreshStoredCodexUsage(accountId, provider, { id: accountId, chatgpt_account_id: row.chatgptAccountId });
39
+ }
19
40
  /** Reads fresh usage; stores the snapshot or a sanitized reason on failure. */
20
41
  export async function refreshStoredCodexUsage(accountId, provider, account) {
21
42
  try {
@@ -6,7 +6,7 @@ import { listCodexAccountSummaries, chatgptAccountIdOf } from '../db/repositorie
6
6
  import { listQoderAccountSummaries } from '../db/repositories/qoder-accounts.js';
7
7
  import { probeCodex, codexModels, CODEX_BASE_URL } from './codex.js';
8
8
  import { withCodexCredentials, codexCredentialError } from './codex-refresh.js';
9
- import { probeQoder, qoderModels } from './qoder/client.js';
9
+ import { probeQoderInference, qoderModels } from './qoder/client.js';
10
10
  import { withQoderCredentials } from './qoder/credentials.js';
11
11
  import { QODER_INFERENCE_BASE } from './qoder/constants.js';
12
12
  export const POOLED_PROVIDER_TYPES = ['codex', 'qoder'];
@@ -49,7 +49,9 @@ const qoderStrategy = {
49
49
  // through the credential seam so a stale job token is refreshed before the upstream is asked.
50
50
  probe: async (provider) => {
51
51
  const account = firstEligible(qoderStrategy.listAccounts(provider.id));
52
- return withQoderCredentials(account.id, (config) => probeQoder({ ...config, accountRecordId: account.id, totalTimeoutMs: Math.min(provider.totalTimeoutMs, 20_000) }))
52
+ // Inference, not just the catalog — see probeQoderInference: a catalog probe cannot see the
53
+ // billing envelope that Qoder returns with HTTP 200 when an account is out of Credits.
54
+ return withQoderCredentials(account.id, (config) => probeQoderInference({ ...config, accountRecordId: account.id, totalTimeoutMs: Math.min(provider.totalTimeoutMs, 30_000) }))
53
55
  .catch((error) => { throw qoderCredentialError(error); });
54
56
  },
55
57
  discover: async (provider) => {
@@ -43,6 +43,7 @@ export function parseCatalog(body, fetchedAt) {
43
43
  enabled: raw.enable !== false,
44
44
  isReasoning: raw.is_reasoning === true,
45
45
  isVl: raw.is_vl === true,
46
+ isFree: raw.is_free === true,
46
47
  maxInputTokens: numberOr(raw.max_input_tokens, 0),
47
48
  maxOutputTokens: numberOr(raw.max_output_tokens, 0),
48
49
  raw,
@@ -65,9 +66,13 @@ export function deserializeCatalog(raw) {
65
66
  const parsed = JSON.parse(raw);
66
67
  const entries = new Map();
67
68
  if (Array.isArray(parsed.entries)) {
68
- for (const entry of parsed.entries)
69
- if (entry?.key)
70
- entries.set(entry.key, entry);
69
+ for (const entry of parsed.entries) {
70
+ if (!entry?.key)
71
+ continue;
72
+ // Catalogs cached before `isFree` existed carry only `raw.is_free`, so re-derive it
73
+ // rather than showing an empty free-model list until the next catalog refresh.
74
+ entries.set(entry.key, entry.isFree === undefined ? { ...entry, isFree: entry.raw?.is_free === true } : entry);
75
+ }
71
76
  }
72
77
  return { entries, fetchedAt: typeof parsed.fetchedAt === 'string' ? parsed.fetchedAt : '' };
73
78
  }
@@ -229,6 +229,52 @@ export async function probeQoder(cfg) {
229
229
  }
230
230
  return { ok: true, detail: 'Connected (catalog loaded)', latencyMs: Date.now() - started, modelCount: catalogue.entries.size };
231
231
  }
232
+ /**
233
+ * A catalog probe proves the PAT exchanges, nothing more. Qoder refuses inference with an
234
+ * envelope-level 403 (code 112, billing) on an HTTP 200 response, so an account can pass the
235
+ * catalog probe while every chat it serves is refused — which reads as "the API key is fine"
236
+ * right up until a real request fails. This probe sends one real message to settle it.
237
+ *
238
+ * It prefers a catalog entry marked `is_free` (a promotion spends no Credits) so testing an
239
+ * account never consumes paid quota.
240
+ */
241
+ export async function probeQoderInference(cfg) {
242
+ const started = Date.now();
243
+ const catalogue = cfg.catalog;
244
+ if (!catalogue || catalogue.entries.size === 0) {
245
+ return { ok: false, detail: 'Model catalog is empty — check the personal access token', latencyMs: Date.now() - started };
246
+ }
247
+ const entries = [...catalogue.entries.values()];
248
+ const model = entries.find((entry) => entry.isFree) ?? entries[0];
249
+ const probe = {
250
+ model: model.key,
251
+ messages: [{ role: 'user', content: [{ type: 'text', text: 'ping' }] }],
252
+ maxOutputTokens: 16,
253
+ stream: false,
254
+ };
255
+ try {
256
+ await callQoderNonStreaming({ ...cfg, totalTimeoutMs: Math.min(cfg.totalTimeoutMs, 30_000) }, probe);
257
+ return { ok: true, detail: `Inference OK (${model.key}${model.isFree ? ', free' : ''})`, latencyMs: Date.now() - started, modelCount: entries.length };
258
+ }
259
+ catch (error) {
260
+ if (error instanceof QoderUpstreamError && error.billing) {
261
+ const free = freeModelHint(entries);
262
+ return {
263
+ ok: false,
264
+ detail: `Account is out of Credits — ${model.key} was refused as billing. ${free ?? 'No promotion-covered model is available on this account, so nothing it serves will work until Credits are added.'}`,
265
+ latencyMs: Date.now() - started,
266
+ modelCount: entries.length,
267
+ };
268
+ }
269
+ const detail = error instanceof QoderUpstreamError ? `HTTP ${error.status}: ${error.message}` : error instanceof Error ? error.message : 'inference probe failed';
270
+ return { ok: false, detail: redactString(detail), latencyMs: Date.now() - started, modelCount: entries.length };
271
+ }
272
+ }
273
+ /** Null when the account has no promotion-covered model — the caller words that case itself. */
274
+ function freeModelHint(entries) {
275
+ const free = entries.filter((entry) => entry.isFree).map((entry) => entry.key);
276
+ return free.length > 0 ? `Free models (promotions) still work: ${free.join(', ')}` : null;
277
+ }
232
278
  /** Catalog entries become discovered models; hidden (enable:false) keys stay routable. */
233
279
  export function qoderModels(catalog) {
234
280
  return [...catalog.entries.values()].map((entry) => ({
@@ -9,6 +9,8 @@ export const QODER_INFERENCE_BASE = 'https://api2.qoder.sh';
9
9
  export const QODER_LOGIN_URL = 'https://qoder.com/account/integrations';
10
10
  export const QODER_JOB_TOKEN_EXCHANGE_URL = `${QODER_OPENAPI_BASE}/api/v1/jobToken/exchange`;
11
11
  export const QODER_USERINFO_URL = `${QODER_OPENAPI_BASE}/api/v1/userinfo`;
12
+ // Credits (Qoder's billing unit) live on openapi, not on the inference host.
13
+ export const QODER_CREDITS_URL = `${QODER_OPENAPI_BASE}/api/v2/quota/usage`;
12
14
  export const QODER_CHAT_SIG_PATH = '/api/v2/service/pro/sse/agent_chat_generation';
13
15
  export const QODER_CHAT_URL = `${QODER_INFERENCE_BASE}/algo${QODER_CHAT_SIG_PATH}?FetchKeys=llm_model_result&AgentId=agent_common&Encode=1`;
14
16
  export const QODER_MODEL_LIST_URL = `${QODER_INFERENCE_BASE}/algo/api/v2/model/list`;
@@ -102,6 +102,17 @@ export async function qoderCredentialsFor(accountRecordId, options = {}) {
102
102
  export async function qoderAttemptFailure(accountRecordId, error) {
103
103
  const status = error.status ?? 502;
104
104
  const billing = Boolean(error.billing);
105
+ // Billing is checked first because a quota refusal carries status 403 (code 112) and would
106
+ // otherwise be classified as a rejected credential: that mislabels the cause, force-re-exchanges
107
+ // a perfectly good PAT on every request, and hides the real answer (the account is out of quota).
108
+ if (billing) {
109
+ // Disabled, not merely degraded: a depleted account cannot serve paid models until its Credits
110
+ // refill, and leaving it eligible made the pool retry it on every request while the client saw
111
+ // "usage limited". The operator re-enables it (or adds Credits) from the admin UI; Credits
112
+ // refresh keeps the snapshot honest in the meantime.
113
+ setQoderAccountHealth(accountRecordId, 'down', 'out of Credits — re-enable after topping up', false);
114
+ throw new GatewayError('upstream_error', 'Qoder account is out of quota', { status: 502, code: 'qoder_billing_block' });
115
+ }
105
116
  if (status === 401 || status === 403) {
106
117
  // The PAT may still be valid — the job token may merely have expired mid-request. Force one
107
118
  // exchange before condemning the account; a second rejection is the PAT's fault.
@@ -112,10 +123,6 @@ export async function qoderAttemptFailure(accountRecordId, error) {
112
123
  }
113
124
  throw new GatewayError('upstream_auth_error', 'Qoder job token was rejected after a refresh', { status: 502, code: 'qoder_auth_failed' });
114
125
  }
115
- if (billing) {
116
- setQoderAccountHealth(accountRecordId, 'degraded', 'upstream reported a quota or billing block');
117
- throw new GatewayError('upstream_error', 'Qoder account is out of quota', { status: 502, code: 'qoder_billing_block' });
118
- }
119
126
  throw upstreamHttpError(status, '', error);
120
127
  }
121
128
  /** Probe/discover share this: a catalog is the only proof the PAT works end to end. */
@@ -0,0 +1,101 @@
1
+ // Qoder Credits: the account's real usage. Qoder bills in Credits, not tokens — its chat
2
+ // stream carries no usage block at all — so the quota API is the only upstream truth about
3
+ // consumption. `is_free` on a catalog entry is the separate signal that a model is covered
4
+ // by a promotion (e.g. Qwen3.8-Max) and therefore does not draw on Credits at all.
5
+ import { QODER_CREDITS_URL } from './constants.js';
6
+ import { qoderCredentialsFor } from './credentials.js';
7
+ import { readQoderCreditsUpdatedAt, saveQoderCredits } from '../../db/repositories/qoder-accounts.js';
8
+ const FETCH_TIMEOUT_MS = 10_000;
9
+ const numberOr = (value, fallback) => {
10
+ const parsed = Number(value);
11
+ return Number.isFinite(parsed) ? parsed : fallback;
12
+ };
13
+ function isoOrNull(value) {
14
+ const ms = typeof value === 'number' ? value : Number(value);
15
+ // 253402214400000 is the far-future sentinel Qoder sends for "no expiry".
16
+ if (!Number.isFinite(ms) || ms >= 253402214400000)
17
+ return null;
18
+ return new Date(ms).toISOString();
19
+ }
20
+ function bucket(label, raw) {
21
+ if (!raw || typeof raw !== 'object')
22
+ return null;
23
+ const value = raw;
24
+ const total = numberOr(value.total, 0);
25
+ if (total <= 0)
26
+ return null;
27
+ const used = numberOr(value.used, 0);
28
+ return {
29
+ label,
30
+ total,
31
+ used,
32
+ remaining: numberOr(value.remaining, Math.max(0, total - used)),
33
+ unit: typeof value.unit === 'string' && value.unit ? value.unit : 'credits',
34
+ };
35
+ }
36
+ export function parseQoderCredits(body, freeModels, fetchedAt) {
37
+ const raw = (body && typeof body === 'object' ? body : {});
38
+ const buckets = [bucket('Plan credits', raw.userQuota), bucket('Add-on credits', raw.addOnQuota), bucket('Org resource package', raw.orgResourcePackage)].filter((entry) => entry !== null);
39
+ return {
40
+ userType: typeof raw.userType === 'string' ? raw.userType : 'unknown',
41
+ freeModels,
42
+ buckets,
43
+ totalUsedPercent: numberOr(raw.totalUsagePercentage, 0),
44
+ exhausted: raw.isQuotaExceeded === true,
45
+ expiresAt: isoOrNull(raw.expiresAt),
46
+ upgradeUrl: typeof raw.upgradeUrl === 'string' && raw.upgradeUrl ? raw.upgradeUrl : null,
47
+ fetchedAt,
48
+ };
49
+ }
50
+ /**
51
+ * Live Credits snapshot. A failure is reported as `unavailable` rather than thrown: Credits are
52
+ * observability, and an account that cannot report them must keep serving requests.
53
+ */
54
+ export async function fetchQoderCredits(creds, freeModels, deps = {}) {
55
+ const fetchedAt = new Date().toISOString();
56
+ const controller = new AbortController();
57
+ const timer = setTimeout(() => controller.abort(), deps.timeoutMs ?? FETCH_TIMEOUT_MS);
58
+ try {
59
+ const response = await (deps.fetchImpl ?? fetch)(QODER_CREDITS_URL, {
60
+ method: 'GET',
61
+ headers: { authorization: `Bearer ${creds.jobToken}`, accept: 'application/json', 'user-agent': 'qodercli/1.0.0' },
62
+ signal: controller.signal,
63
+ });
64
+ if (!response.ok)
65
+ return { ...parseQoderCredits({}, freeModels, fetchedAt), unavailable: `HTTP ${response.status}` };
66
+ return parseQoderCredits(await response.json(), freeModels, fetchedAt);
67
+ }
68
+ catch (error) {
69
+ return { ...parseQoderCredits({}, freeModels, fetchedAt), unavailable: error instanceof Error ? error.message : 'Credits request failed' };
70
+ }
71
+ finally {
72
+ clearTimeout(timer);
73
+ }
74
+ }
75
+ /**
76
+ * Fetch and persist one account's Credits snapshot. Free models come from the account's cached
77
+ * catalog because `is_free` is a catalog property, not a quota one — it is why an account at zero
78
+ * Credits can still serve a promotional model.
79
+ *
80
+ * Lives here rather than in the admin route so the gateway can reuse it: Credits are the only
81
+ * upstream truth about consumption (Qoder's chat stream reports no usage at all), so the snapshot
82
+ * has to be kept fresh as accounts are actually used, not only when an admin opens the page.
83
+ */
84
+ export async function refreshStoredQoderCredits(accountRecordId) {
85
+ const { config } = await qoderCredentialsFor(accountRecordId);
86
+ const freeModels = config.catalog ? [...config.catalog.entries.values()].filter((entry) => entry.isFree).map((entry) => entry.key) : [];
87
+ const credits = await fetchQoderCredits({ jobToken: config.jobToken }, freeModels, { timeoutMs: FETCH_TIMEOUT_MS });
88
+ saveQoderCredits(accountRecordId, credits.unavailable ? null : JSON.stringify(credits), credits.unavailable ?? null, credits.fetchedAt);
89
+ return credits;
90
+ }
91
+ /** Same throttle as Codex usage: the Credits endpoint is a second upstream call per account. */
92
+ export const QODER_CREDITS_REFRESH_INTERVAL_MS = 5 * 60 * 1000;
93
+ export async function refreshQoderCreditsIfStale(accountRecordId, now = new Date()) {
94
+ const row = readQoderCreditsUpdatedAt(accountRecordId);
95
+ const last = row ? Date.parse(row) : NaN;
96
+ if (Number.isFinite(last) && now.getTime() - last < QODER_CREDITS_REFRESH_INTERVAL_MS)
97
+ return;
98
+ // Never throws, matching `refreshStoredCodexUsage`: this runs in the background after a response
99
+ // has already been delivered, so a dead credential or unreachable quota API must not surface.
100
+ await refreshStoredQoderCredits(accountRecordId).catch(() => { });
101
+ }
@@ -274,6 +274,12 @@ export async function registerCodexRoutes(app) {
274
274
  values.push(typeof value === 'boolean' ? (value ? 1 : 0) : value);
275
275
  }
276
276
  }
277
+ // Re-enabling must also clear a quota verdict: every selection path excludes `health_state='down'`,
278
+ // so flipping `enabled` alone would leave the account unreachable while the UI showed it as on.
279
+ if (body.enabled === true) {
280
+ fields.push('health_state=?');
281
+ values.push('unknown');
282
+ }
277
283
  fields.push('updated_at=?');
278
284
  values.push(new Date().toISOString(), id);
279
285
  raw.prepare(`UPDATE codex_accounts SET ${fields.join(',')} WHERE id=?`).run(...values);