ldrouter 1.17.2 → 1.17.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/README.md +26 -0
- package/dist/server/db/repositories/qoder-accounts.js +36 -5
- package/dist/server/gateway/runner.js +83 -2
- package/dist/server/providers/codex-autostart.js +21 -0
- package/dist/server/providers/pooled.js +4 -2
- package/dist/server/providers/qoder/catalog.js +8 -3
- package/dist/server/providers/qoder/client.js +46 -0
- package/dist/server/providers/qoder/constants.js +2 -0
- package/dist/server/providers/qoder/credentials.js +11 -4
- package/dist/server/providers/qoder/credits.js +101 -0
- package/dist/server/routes/admin/codex.js +6 -0
- package/dist/server/routes/admin/qoder.js +31 -2
- package/dist/server/routing/combo.js +7 -0
- package/dist/server/upstream/client.js +4 -1
- package/dist/web/assets/index-DQd2y4FN.css +1 -0
- package/dist/web/assets/{index-DAzlm3Y6.js → index-f8A3f3uE.js} +77 -77
- package/dist/web/index.html +2 -2
- package/migrations/0008_qoder_credits.sql +7 -0
- package/package.json +1 -1
- package/dist/web/assets/index-Xccks9E5.css +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,32 @@ All notable changes to this project are documented here. The format follows
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/) and the project adheres to
|
|
5
5
|
[Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [1.17.4] - 2026-09-18
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
|
|
11
|
+
- **An account that ran out of quota kept being retried instead of leaving the pool.** A quota refusal was classified `unknown`, which is not a routing decision: `isUpstreamHealthFailure` excludes it and `shouldFallback` answers false for it, so the pool selected the same spent account on every request and answered "usage limited". Live production data confirmed the shape — 49 quota refusals (`Codex upstream HTTP 429: The usage limit has been reached`, `Qoder account is out of quota`) recorded over 14 days, every one of them `failure_reason=unknown`, with the same `codex_account_id` / `qoder_account_id` retried each time. Quota is now its own failure class, checked before the error-type switch (a 429 reaches the classifier as `upstream_rate_limit`, which previously returned `http_status` before the quota test could run).
|
|
12
|
+
- **The exhausted account is now disabled, not just marked degraded.** Disabling (`enabled=0`) is the write that actually stops the churn: every selection path — `expandCodexAccountCandidates`, `expandQoderAccountCandidates`, `getCodexAccountForProvider`, `findEligibleQoderAccount` — filters on it, whereas `health_state` alone was overwritten to `healthy` by the next successful request. A Qoder billing block now disables the account too, instead of only degrading it.
|
|
13
|
+
- **A direct model never advanced to the next account.** The retry was gated on `comboPlan ? ... : false`, so the reported case — a direct `qoder/...` model, logged 33 times in production — never retried at all. A quota failure now always advances to the next candidate, combo or not; that is safe precisely because the exhausted account was just disabled, so the retry cannot land on it again.
|
|
14
|
+
- **The upstream 429 body was discarded, hiding the reason.** Quota exhaustion arrives as a plain 429 on OpenAI-compatible providers, with the wording ("you exceeded your current quota") only in the body — which `upstreamHttpError` dropped, leaving an account unclassifiable. The redacted body excerpt is now carried on a 429 too. A 429 without quota wording is still treated as a transient throttle: it retries the same account and does not disable it.
|
|
15
|
+
- **Re-enabling an account in the admin UI did not put it back in the pool.** The enable toggle flipped `enabled` but left `health_state=down`, which every selection filter excludes, so the account stayed unroutable while the panel showed it as enabled. Both the Codex and Qoder routes now clear the health verdict when re-enabling.
|
|
16
|
+
|
|
17
|
+
### Added
|
|
18
|
+
|
|
19
|
+
- **Usage and Credits are refreshed as the pool is used.** Both the Codex usage snapshot and the Qoder credits snapshot are re-read in the background after a successful request, throttled per account so a burst of traffic costs at most one extra upstream call, and never awaited — the response path is unaffected.
|
|
20
|
+
|
|
21
|
+
## [1.17.3] - 2026-09-17
|
|
22
|
+
|
|
23
|
+
### Added
|
|
24
|
+
|
|
25
|
+
- **Qoder accounts now show Credits — the account's real usage.** Qoder does not bill in tokens: its chat stream carries no usage block at all, so every Qoder request logged zero tokens and the panel had no usage figure to show. Credits are now read from Qoder's quota API (`GET /api/v2/quota/usage`) and shown per account: plan / add-on / org buckets, percentage used, expiry, and whether the quota is exhausted. A **Refresh credits** action re-reads them for every account.
|
|
26
|
+
- **Promotion-covered models are shown next to Credits.** Qoder keeps a model free by marking its catalog entry `is_free` — currently `qmodel_38max` (Qwen3.8-Max). Such a model spends no Credits, so an account at zero Credits still answers on it while every other model is refused. The panel now pairs **No plan credits** with **Free now: <model>** so a partly-working account reads as intended instead of looking like a router fault.
|
|
27
|
+
|
|
28
|
+
### Fixed
|
|
29
|
+
|
|
30
|
+
- **A Qoder quota refusal was reported as a token rejection.** The upstream returns HTTP 200 with a `403 code 112` envelope inside the streamed body for quota/billing refusals; the attempt classifier checked `401/403` before billing, so a depleted account was labelled "job token was rejected after a refresh" and the account was force-re-exchanged on every request. Billing-shaped refusals are no longer mistaken for credential failures.
|
|
31
|
+
- **The account Test button reported a blocked account as working.** It only probed the model catalog, which proves the PAT exchanges and nothing else — the refusal arrives as a `403` envelope inside an HTTP 200 body, so a check that never sends a message cannot see it. Test now sends one real request, preferring a promotion-covered model so testing never spends Credits, and names the free models that still work when the account is out of them.
|
|
32
|
+
|
|
7
33
|
## [1.17.2] - 2026-09-17
|
|
8
34
|
|
|
9
35
|
### Fixed
|
package/README.md
CHANGED
|
@@ -65,6 +65,8 @@ ZIP upload and automatic Codex CLI config-file generation/mutation are not inclu
|
|
|
65
65
|
|
|
66
66
|
The account panel shows the 5-hour and weekly quota windows with reset countdowns, per-account and bulk usage refresh, weekly reset credits, an opt-in 5-hour window auto-start, and a **Test** control that probes the upstream account without exposing tokens. Routing order is set by dragging rows; the saved order is the fallback order the router uses. **Delete** is permanent and erases the stored encrypted credentials — past request logs are kept but lose the account reference.
|
|
67
67
|
|
|
68
|
+
**A quota-exhausted account leaves the pool on its own.** When an account refuses for quota reasons — Codex `429 The usage limit has been reached`, or Qoder `out of credits` — it is marked `down` *and disabled*, so the very next request picks a different account in the pool rather than retrying the spent one. This applies to direct `codex/...` and `qoder/...` models too, not only combos. Re-enable the account in the panel once its window resets. Both Codex usage and Qoder credits are also refreshed in the background after a successful request (throttled, and never on the response path), so the panel figures track real usage without pressing **Refresh**.
|
|
69
|
+
|
|
68
70
|
### Qoder setup
|
|
69
71
|
|
|
70
72
|
Qoder providers use a personal access token (PAT) instead of an API key. Create one at `https://qoder.com/account/integrations`; it starts with `pt-`.
|
|
@@ -81,6 +83,30 @@ Because the gateway serves model discovery from that live catalog, a new account
|
|
|
81
83
|
|
|
82
84
|
The panel shows the masked user id, job-token expiry, when the catalog was last fetched, health, and an enabled toggle. Routing order is set by dragging rows, exactly as with Codex.
|
|
83
85
|
|
|
86
|
+
### Qoder usage, Credits and promotions
|
|
87
|
+
|
|
88
|
+
Qoder does not meter models in tokens: its chat stream carries no usage block at all, so request logs
|
|
89
|
+
show zero tokens for every Qoder model and the panel's **Credits** column is the account's real
|
|
90
|
+
usage. It is read from Qoder's quota API (`GET /api/v2/quota/usage` on `openapi.qoder.sh`) and shows
|
|
91
|
+
the plan / add-on / org buckets, the percentage used, and whether the account is exhausted.
|
|
92
|
+
|
|
93
|
+
**Credits and free models are different things, and the panel shows both.** Qoder keeps promoting a
|
|
94
|
+
model — currently `qmodel_38max`, the Qwen3.8-Max route — by marking its catalog entry `is_free`.
|
|
95
|
+
Such a model spends no Credits, so an account at **zero Credits** still answers on it while every
|
|
96
|
+
other model returns an envelope `403 code 112` with a `pricingUrl`. That pairing is what makes an
|
|
97
|
+
"out of credits" account look partly broken: the panel labels the column **No plan credits** next to
|
|
98
|
+
**Free now: qmodel_38max** so the contradiction is visible instead of looking like a router fault.
|
|
99
|
+
|
|
100
|
+
There is an upstream edge worth knowing when reading logs: a quota refusal arrives as **HTTP 200**
|
|
101
|
+
with a `403` envelope inside the streamed body, so it is not an HTTP error at the transport layer.
|
|
102
|
+
|
|
103
|
+
Use **Refresh credits** to re-read the snapshot for every account. It deliberately reuses the live
|
|
104
|
+
`job token` instead of exchanging the PAT again — a fresh exchange invalidates the token a
|
|
105
|
+
concurrent request may be using.
|
|
106
|
+
|
|
107
|
+
**Test** sends one real message, because a catalog probe cannot see a quota refusal. It picks a
|
|
108
|
+
promotion-covered model when the account has one, so testing never spends Credits.
|
|
109
|
+
|
|
84
110
|
## Environment variables
|
|
85
111
|
|
|
86
112
|
| Variable | Description | Default |
|
|
@@ -5,6 +5,18 @@ import { decryptSecret, encryptSecret } from '../../auth/crypto.js';
|
|
|
5
5
|
import { uuid } from '../../auth/ids.js';
|
|
6
6
|
import { createHash } from 'node:crypto';
|
|
7
7
|
import { getRawDb } from '../index.js';
|
|
8
|
+
/** A corrupted or absent snapshot reads as "no data", never as a thrown error on a list call. */
|
|
9
|
+
function parseStoredCredits(json) {
|
|
10
|
+
if (!json)
|
|
11
|
+
return null;
|
|
12
|
+
try {
|
|
13
|
+
const parsed = JSON.parse(json);
|
|
14
|
+
return parsed && typeof parsed === 'object' ? parsed : null;
|
|
15
|
+
}
|
|
16
|
+
catch {
|
|
17
|
+
return null;
|
|
18
|
+
}
|
|
19
|
+
}
|
|
8
20
|
export function maskQoderValue(value) {
|
|
9
21
|
if (!value)
|
|
10
22
|
return null;
|
|
@@ -18,7 +30,7 @@ export function identityFromQoderRecord(record) {
|
|
|
18
30
|
label: record.label,
|
|
19
31
|
};
|
|
20
32
|
}
|
|
21
|
-
const SUMMARY_COLUMNS = 'id,provider_id AS providerId,label,email,qoder_user_id AS qoderUserId,machine_id AS machineId,enabled,health_state AS healthState,priority,job_token_expires_at AS jobTokenExpiresAt,catalog_fetched_at AS catalogFetchedAt,last_error AS lastError,consecutive_failures AS consecutiveFailures,created_at AS createdAt,updated_at AS updatedAt';
|
|
33
|
+
const SUMMARY_COLUMNS = 'id,provider_id AS providerId,label,email,qoder_user_id AS qoderUserId,machine_id AS machineId,enabled,health_state AS healthState,priority,job_token_expires_at AS jobTokenExpiresAt,catalog_fetched_at AS catalogFetchedAt,credits_json AS creditsJson,credits_updated_at AS creditsUpdatedAt,credits_error AS creditsError,last_error AS lastError,consecutive_failures AS consecutiveFailures,created_at AS createdAt,updated_at AS updatedAt';
|
|
22
34
|
export function toQoderAccountSummary(row) {
|
|
23
35
|
return {
|
|
24
36
|
id: row.id,
|
|
@@ -31,6 +43,9 @@ export function toQoderAccountSummary(row) {
|
|
|
31
43
|
priority: row.priority,
|
|
32
44
|
jobTokenExpiresAt: row.jobTokenExpiresAt,
|
|
33
45
|
catalogFetchedAt: row.catalogFetchedAt,
|
|
46
|
+
credits: (row.credits ?? null) || parseStoredCredits(row.creditsJson ?? null),
|
|
47
|
+
creditsUpdatedAt: row.creditsUpdatedAt ?? null,
|
|
48
|
+
creditsError: row.creditsError ?? null,
|
|
34
49
|
lastError: row.lastError,
|
|
35
50
|
consecutiveFailures: row.consecutiveFailures,
|
|
36
51
|
createdAt: row.createdAt,
|
|
@@ -140,11 +155,16 @@ export function persistQoderJobToken(id, update) {
|
|
|
140
155
|
health_state='healthy', last_error=NULL, updated_at=? WHERE id=?`)
|
|
141
156
|
.run(jobToken.ciphertext, jobToken.nonce, jobToken.version, update.expiresAt, update.catalogJson ?? null, update.catalogFetchedAt ?? null, nextUpdatedAt(id), id);
|
|
142
157
|
}
|
|
143
|
-
|
|
158
|
+
/** `enabled` mirrors the Codex setter: pass `false` to take an exhausted account out of the pool. */
|
|
159
|
+
export function setQoderAccountHealth(id, healthState, lastError = null, enabled) {
|
|
144
160
|
const failures = healthState === 'healthy' ? 0 : null;
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
161
|
+
const fields = enabled === undefined
|
|
162
|
+
? 'health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), updated_at=?'
|
|
163
|
+
: 'health_state=?, last_error=?, consecutive_failures=COALESCE(?, consecutive_failures), enabled=?, updated_at=?';
|
|
164
|
+
const values = enabled === undefined
|
|
165
|
+
? [healthState, lastError, failures, nextUpdatedAt(id), id]
|
|
166
|
+
: [healthState, lastError, failures, enabled ? 1 : 0, nextUpdatedAt(id), id];
|
|
167
|
+
getRawDb().prepare(`UPDATE qoder_accounts SET ${fields} WHERE id=?`).run(...values);
|
|
148
168
|
}
|
|
149
169
|
export function readQoderCatalog(id) {
|
|
150
170
|
const row = getRawDb()
|
|
@@ -157,6 +177,17 @@ export function saveQoderCatalog(id, catalogJson, fetchedAt) {
|
|
|
157
177
|
.prepare('UPDATE qoder_accounts SET catalog_json=?, catalog_fetched_at=?, updated_at=? WHERE id=?')
|
|
158
178
|
.run(catalogJson, fetchedAt ?? new Date().toISOString(), nextUpdatedAt(id), id);
|
|
159
179
|
}
|
|
180
|
+
/** When the Credits snapshot was last written; null when it never has been. */
|
|
181
|
+
export function readQoderCreditsUpdatedAt(id) {
|
|
182
|
+
const row = getRawDb().prepare('SELECT credits_updated_at AS at FROM qoder_accounts WHERE id=?').get(id);
|
|
183
|
+
return row?.at ?? null;
|
|
184
|
+
}
|
|
185
|
+
/** Credits are observability, not routing state: a failed fetch is recorded without touching health. */
|
|
186
|
+
export function saveQoderCredits(id, creditsJson, error, fetchedAt) {
|
|
187
|
+
getRawDb()
|
|
188
|
+
.prepare('UPDATE qoder_accounts SET credits_json=COALESCE(?, credits_json), credits_updated_at=?, credits_error=?, updated_at=? WHERE id=?')
|
|
189
|
+
.run(creditsJson, fetchedAt, error, nextUpdatedAt(id), id);
|
|
190
|
+
}
|
|
160
191
|
// Keeps `updated_at` strictly increasing even within the same millisecond, so a caller can
|
|
161
192
|
// detect a write it just made. Copied from the Codex repository for consistency.
|
|
162
193
|
function nextUpdatedAt(id) {
|
|
@@ -8,7 +8,9 @@ import { loadCombo, selectCandidates, orderCandidates, shouldFallback, expandCod
|
|
|
8
8
|
import { listCodexAccountsForProvider, setCodexAccountHealth } from '../db/repositories/codex-accounts.js';
|
|
9
9
|
import { listQoderAccountsForProvider, setQoderAccountHealth } from '../db/repositories/qoder-accounts.js';
|
|
10
10
|
import { qoderAttemptFailure, qoderCredentialsFor } from '../providers/qoder/credentials.js';
|
|
11
|
+
import { refreshQoderCreditsIfStale } from '../providers/qoder/credits.js';
|
|
11
12
|
import { callQoderNonStreaming, callQoderStreaming } from '../providers/qoder/client.js';
|
|
13
|
+
import { refreshCodexUsageIfStale } from '../providers/codex-autostart.js';
|
|
12
14
|
import { getEffectiveState, isOpen, recordSuccess, recordFailure, halfOpenProbeAllowed } from '../routing/circuit.js';
|
|
13
15
|
import { checkRpm, checkTpm, acquireConcurrent, releaseConcurrent } from '../routing/ratelimit.js';
|
|
14
16
|
import { checkDailyMonthly, consumeUsage } from '../routing/quota.js';
|
|
@@ -279,6 +281,13 @@ export class GatewayRunner {
|
|
|
279
281
|
setCodexAccountHealth(candidate.codexAccountId, 'healthy');
|
|
280
282
|
if (candidate.qoderAccountId)
|
|
281
283
|
setQoderAccountHealth(candidate.qoderAccountId, 'healthy');
|
|
284
|
+
// Refresh the quota snapshot in the background so the dashboard reflects real usage.
|
|
285
|
+
// Never awaited: it is a second upstream call, and the response is already complete.
|
|
286
|
+
// Throttled inside each helper, so a burst of requests costs at most one refresh.
|
|
287
|
+
if (candidate.codexAccountId)
|
|
288
|
+
void refreshCodexUsageIfStale(candidate.codexAccountId).catch(() => { });
|
|
289
|
+
if (candidate.qoderAccountId)
|
|
290
|
+
void refreshQoderCreditsIfStale(candidate.qoderAccountId).catch(() => { });
|
|
282
291
|
getDb().update(schema.providers).set({ healthState: 'healthy', updatedAt: new Date().toISOString() }).where(eq(schema.providers.id, provider.id)).run();
|
|
283
292
|
attempts.push(attempt);
|
|
284
293
|
sentToClient = out.streamStarted ?? false;
|
|
@@ -286,15 +295,19 @@ export class GatewayRunner {
|
|
|
286
295
|
}
|
|
287
296
|
catch (e) {
|
|
288
297
|
const err = e instanceof GatewayError ? e : new GatewayError('upstream_error', e.message, { cause: e });
|
|
298
|
+
const quotaFailure = isQuotaFailure(err);
|
|
289
299
|
debugUpstream(ctx.requestId, 'ATTEMPT ERROR', [
|
|
290
300
|
`attempt=${i + 1}`,
|
|
291
301
|
`provider=${provider.name}`,
|
|
292
302
|
`model=${candidate.publicModelId}`,
|
|
293
303
|
`type=${err.type}`,
|
|
294
304
|
`status=${err.status}`,
|
|
295
|
-
`willFallback=${
|
|
305
|
+
`willFallback=${shouldRetryAttempt(comboPlan, err)}`,
|
|
296
306
|
]);
|
|
297
|
-
|
|
307
|
+
// Quota failures always advance (see shouldRetryAttempt): the next candidate for a pool
|
|
308
|
+
// provider is the next account, and the exhausted one was just disabled so the retry cannot
|
|
309
|
+
// land on it again.
|
|
310
|
+
const shouldRetry = shouldRetryAttempt(comboPlan, err);
|
|
298
311
|
attempt.statusCode = err.status;
|
|
299
312
|
attempt.success = false;
|
|
300
313
|
attempt.latencyMs = Date.now() - attemptStart;
|
|
@@ -312,6 +325,10 @@ export class GatewayRunner {
|
|
|
312
325
|
}
|
|
313
326
|
attempts.push(attempt);
|
|
314
327
|
lastError = err;
|
|
328
|
+
// An exhausted account leaves the pool outright: without this the combo retried the same
|
|
329
|
+
// dead account on every request for hours (verified live) and answered "usage limited".
|
|
330
|
+
if (quotaFailure)
|
|
331
|
+
markQuotaExhausted(candidate, err.message);
|
|
315
332
|
if (isUpstreamHealthFailure(err)) {
|
|
316
333
|
recordFailure(provider.id, provider.cbFailureThreshold, provider.cbCooldownSeconds);
|
|
317
334
|
if (candidate.codexAccountId)
|
|
@@ -971,7 +988,71 @@ export class GatewayRunner {
|
|
|
971
988
|
export function isUpstreamHealthFailure(err) {
|
|
972
989
|
return ['connection_error', 'connect_timeout', 'first_token_timeout', 'http_status', 'upstream_rate_limit'].includes(classifyFailure(err));
|
|
973
990
|
}
|
|
991
|
+
/**
|
|
992
|
+
* An exhausted quota is a durable, per-account verdict — the account cannot serve anything again
|
|
993
|
+
* until its window resets, so it must leave the pool instead of being retried.
|
|
994
|
+
*
|
|
995
|
+
* Deliberately NOT part of `isUpstreamHealthFailure`: that path opens the provider's circuit
|
|
996
|
+
* breaker, which here would take down every sibling account of a healthy pool because one member
|
|
997
|
+
* ran dry. The exhaustion is applied account-by-account by `markQuotaExhausted` instead.
|
|
998
|
+
*
|
|
999
|
+
* Detection needs both signals because the two upstreams refuse differently, and neither reports a
|
|
1000
|
+
* status the router can trust alone:
|
|
1001
|
+
* - Codex answers 429 but the credential layer re-wraps it (status 502, cause 429), and a plain
|
|
1002
|
+
* rate limit is also a 429 — the "usage limit"/"quota" wording is what distinguishes "come back
|
|
1003
|
+
* in an hour" from "this account is spent".
|
|
1004
|
+
* - Qoder refuses with a billing envelope carrying status 403/code 112, surfaced as the
|
|
1005
|
+
* `qoder_billing_block` code.
|
|
1006
|
+
* Wording is only consulted when the status is quota-shaped, so a 400 that merely mentions the word
|
|
1007
|
+
* "quota" cannot disable a working account.
|
|
1008
|
+
*/
|
|
1009
|
+
export function isQuotaFailure(err) {
|
|
1010
|
+
if (err.code === 'qoder_billing_block')
|
|
1011
|
+
return true;
|
|
1012
|
+
const cause = err.cause;
|
|
1013
|
+
const upstreamStatus = typeof cause?.status === 'number' ? cause.status : err.status;
|
|
1014
|
+
if (upstreamStatus !== 429 && upstreamStatus !== 403)
|
|
1015
|
+
return false;
|
|
1016
|
+
return /usage[_ -]?limit|quota|out of credits|insufficient[_ -]?(credit|balance)|exceeded your current quota|billing/i.test(err.message);
|
|
1017
|
+
}
|
|
1018
|
+
/**
|
|
1019
|
+
* Take an exhausted account out of the pool: mark it down and disable it.
|
|
1020
|
+
*
|
|
1021
|
+
* `enabled=0` (not just `health_state='down'`) is what the operator asked for and what actually
|
|
1022
|
+
* stops the churn: every selection path — `expandCodexAccountCandidates`,
|
|
1023
|
+
* `expandQoderAccountCandidates`, `getCodexAccountForProvider`, `findEligibleQoderAccount` — filters
|
|
1024
|
+
* on `enabled`, so disabling is the one write that makes the next request pick a different account.
|
|
1025
|
+
* The account stays visible in the admin UI, where re-enabling it after the window resets is a
|
|
1026
|
+
* one-click action.
|
|
1027
|
+
*/
|
|
1028
|
+
export function markQuotaExhausted(candidate, message) {
|
|
1029
|
+
const reason = redactString(message);
|
|
1030
|
+
if (candidate.codexAccountId)
|
|
1031
|
+
setCodexAccountHealth(candidate.codexAccountId, 'down', reason, false);
|
|
1032
|
+
if (candidate.qoderAccountId)
|
|
1033
|
+
setQoderAccountHealth(candidate.qoderAccountId, 'down', reason, false);
|
|
1034
|
+
}
|
|
1035
|
+
/**
|
|
1036
|
+
* Whether a failed attempt should advance to the next candidate.
|
|
1037
|
+
*
|
|
1038
|
+
* A quota failure always advances, with or without a combo plan. For a pool provider the next
|
|
1039
|
+
* candidate is the next *account*, so requiring a combo plan left a direct `codex/…` or `qoder/…`
|
|
1040
|
+
* request pinned to the exhausted account — the reported bug ("does not switch to a new account,
|
|
1041
|
+
* just answers usage limited"). Safe because the exhausted account was just disabled: the retry
|
|
1042
|
+
* cannot land on it again. Everything else keeps the combo's configured triggers, where no combo
|
|
1043
|
+
* plan means no fallback.
|
|
1044
|
+
*/
|
|
1045
|
+
export function shouldRetryAttempt(comboPlan, err) {
|
|
1046
|
+
if (isQuotaFailure(err))
|
|
1047
|
+
return true;
|
|
1048
|
+
return comboPlan ? shouldFallback(comboPlan, { type: classifyFailure(err), status: err.status }) : false;
|
|
1049
|
+
}
|
|
974
1050
|
export function classifyFailure(err) {
|
|
1051
|
+
// Checked first, before the type switch: a quota refusal arrives as `upstream_rate_limit` (a
|
|
1052
|
+
// 429) or `upstream_error` (Codex rewraps it as a 502 with a 429 cause; Qoder uses code 112) and
|
|
1053
|
+
// must not fall through to `http_status`/`unknown`, neither of which is a routing decision.
|
|
1054
|
+
if (isQuotaFailure(err))
|
|
1055
|
+
return 'quota';
|
|
975
1056
|
switch (err.type) {
|
|
976
1057
|
case 'timeout_error':
|
|
977
1058
|
return err.message.includes('first token') ? 'first_token_timeout' : 'connect_timeout';
|
|
@@ -16,6 +16,27 @@ function providerFor(providerId) {
|
|
|
16
16
|
function accountFor(accountId) {
|
|
17
17
|
return getRawDb().prepare('SELECT id,chatgpt_account_id FROM codex_accounts WHERE id=?').get(accountId) ?? null;
|
|
18
18
|
}
|
|
19
|
+
/**
|
|
20
|
+
* Keep the quota snapshot honest while the account is in use.
|
|
21
|
+
*
|
|
22
|
+
* The usage API is a second upstream call, so this is throttled rather than run per request: a
|
|
23
|
+
* snapshot younger than the interval is left alone. Before this, usage only refreshed when an admin
|
|
24
|
+
* pressed the button or on the 10-minute autostart tick for opted-in accounts, so a dashboard could
|
|
25
|
+
* show hours-old consumption for an account actively serving traffic.
|
|
26
|
+
*/
|
|
27
|
+
export const CODEX_USAGE_REFRESH_INTERVAL_MS = 5 * 60 * 1000;
|
|
28
|
+
export async function refreshCodexUsageIfStale(accountId, now = new Date()) {
|
|
29
|
+
const row = getRawDb().prepare('SELECT provider_id AS providerId,chatgpt_account_id AS chatgptAccountId,codex_usage_updated_at AS at FROM codex_accounts WHERE id=?').get(accountId);
|
|
30
|
+
if (!row)
|
|
31
|
+
return;
|
|
32
|
+
const provider = providerFor(row.providerId);
|
|
33
|
+
if (!provider)
|
|
34
|
+
return;
|
|
35
|
+
const last = row.at ? Date.parse(row.at) : NaN;
|
|
36
|
+
if (Number.isFinite(last) && now.getTime() - last < CODEX_USAGE_REFRESH_INTERVAL_MS)
|
|
37
|
+
return;
|
|
38
|
+
await refreshStoredCodexUsage(accountId, provider, { id: accountId, chatgpt_account_id: row.chatgptAccountId });
|
|
39
|
+
}
|
|
19
40
|
/** Reads fresh usage; stores the snapshot or a sanitized reason on failure. */
|
|
20
41
|
export async function refreshStoredCodexUsage(accountId, provider, account) {
|
|
21
42
|
try {
|
|
@@ -6,7 +6,7 @@ import { listCodexAccountSummaries, chatgptAccountIdOf } from '../db/repositorie
|
|
|
6
6
|
import { listQoderAccountSummaries } from '../db/repositories/qoder-accounts.js';
|
|
7
7
|
import { probeCodex, codexModels, CODEX_BASE_URL } from './codex.js';
|
|
8
8
|
import { withCodexCredentials, codexCredentialError } from './codex-refresh.js';
|
|
9
|
-
import {
|
|
9
|
+
import { probeQoderInference, qoderModels } from './qoder/client.js';
|
|
10
10
|
import { withQoderCredentials } from './qoder/credentials.js';
|
|
11
11
|
import { QODER_INFERENCE_BASE } from './qoder/constants.js';
|
|
12
12
|
export const POOLED_PROVIDER_TYPES = ['codex', 'qoder'];
|
|
@@ -49,7 +49,9 @@ const qoderStrategy = {
|
|
|
49
49
|
// through the credential seam so a stale job token is refreshed before the upstream is asked.
|
|
50
50
|
probe: async (provider) => {
|
|
51
51
|
const account = firstEligible(qoderStrategy.listAccounts(provider.id));
|
|
52
|
-
|
|
52
|
+
// Inference, not just the catalog — see probeQoderInference: a catalog probe cannot see the
|
|
53
|
+
// billing envelope that Qoder returns with HTTP 200 when an account is out of Credits.
|
|
54
|
+
return withQoderCredentials(account.id, (config) => probeQoderInference({ ...config, accountRecordId: account.id, totalTimeoutMs: Math.min(provider.totalTimeoutMs, 30_000) }))
|
|
53
55
|
.catch((error) => { throw qoderCredentialError(error); });
|
|
54
56
|
},
|
|
55
57
|
discover: async (provider) => {
|
|
@@ -43,6 +43,7 @@ export function parseCatalog(body, fetchedAt) {
|
|
|
43
43
|
enabled: raw.enable !== false,
|
|
44
44
|
isReasoning: raw.is_reasoning === true,
|
|
45
45
|
isVl: raw.is_vl === true,
|
|
46
|
+
isFree: raw.is_free === true,
|
|
46
47
|
maxInputTokens: numberOr(raw.max_input_tokens, 0),
|
|
47
48
|
maxOutputTokens: numberOr(raw.max_output_tokens, 0),
|
|
48
49
|
raw,
|
|
@@ -65,9 +66,13 @@ export function deserializeCatalog(raw) {
|
|
|
65
66
|
const parsed = JSON.parse(raw);
|
|
66
67
|
const entries = new Map();
|
|
67
68
|
if (Array.isArray(parsed.entries)) {
|
|
68
|
-
for (const entry of parsed.entries)
|
|
69
|
-
if (entry?.key)
|
|
70
|
-
|
|
69
|
+
for (const entry of parsed.entries) {
|
|
70
|
+
if (!entry?.key)
|
|
71
|
+
continue;
|
|
72
|
+
// Catalogs cached before `isFree` existed carry only `raw.is_free`, so re-derive it
|
|
73
|
+
// rather than showing an empty free-model list until the next catalog refresh.
|
|
74
|
+
entries.set(entry.key, entry.isFree === undefined ? { ...entry, isFree: entry.raw?.is_free === true } : entry);
|
|
75
|
+
}
|
|
71
76
|
}
|
|
72
77
|
return { entries, fetchedAt: typeof parsed.fetchedAt === 'string' ? parsed.fetchedAt : '' };
|
|
73
78
|
}
|
|
@@ -229,6 +229,52 @@ export async function probeQoder(cfg) {
|
|
|
229
229
|
}
|
|
230
230
|
return { ok: true, detail: 'Connected (catalog loaded)', latencyMs: Date.now() - started, modelCount: catalogue.entries.size };
|
|
231
231
|
}
|
|
232
|
+
/**
|
|
233
|
+
* A catalog probe proves the PAT exchanges, nothing more. Qoder refuses inference with an
|
|
234
|
+
* envelope-level 403 (code 112, billing) on an HTTP 200 response, so an account can pass the
|
|
235
|
+
* catalog probe while every chat it serves is refused — which reads as "the API key is fine"
|
|
236
|
+
* right up until a real request fails. This probe sends one real message to settle it.
|
|
237
|
+
*
|
|
238
|
+
* It prefers a catalog entry marked `is_free` (a promotion spends no Credits) so testing an
|
|
239
|
+
* account never consumes paid quota.
|
|
240
|
+
*/
|
|
241
|
+
export async function probeQoderInference(cfg) {
|
|
242
|
+
const started = Date.now();
|
|
243
|
+
const catalogue = cfg.catalog;
|
|
244
|
+
if (!catalogue || catalogue.entries.size === 0) {
|
|
245
|
+
return { ok: false, detail: 'Model catalog is empty — check the personal access token', latencyMs: Date.now() - started };
|
|
246
|
+
}
|
|
247
|
+
const entries = [...catalogue.entries.values()];
|
|
248
|
+
const model = entries.find((entry) => entry.isFree) ?? entries[0];
|
|
249
|
+
const probe = {
|
|
250
|
+
model: model.key,
|
|
251
|
+
messages: [{ role: 'user', content: [{ type: 'text', text: 'ping' }] }],
|
|
252
|
+
maxOutputTokens: 16,
|
|
253
|
+
stream: false,
|
|
254
|
+
};
|
|
255
|
+
try {
|
|
256
|
+
await callQoderNonStreaming({ ...cfg, totalTimeoutMs: Math.min(cfg.totalTimeoutMs, 30_000) }, probe);
|
|
257
|
+
return { ok: true, detail: `Inference OK (${model.key}${model.isFree ? ', free' : ''})`, latencyMs: Date.now() - started, modelCount: entries.length };
|
|
258
|
+
}
|
|
259
|
+
catch (error) {
|
|
260
|
+
if (error instanceof QoderUpstreamError && error.billing) {
|
|
261
|
+
const free = freeModelHint(entries);
|
|
262
|
+
return {
|
|
263
|
+
ok: false,
|
|
264
|
+
detail: `Account is out of Credits — ${model.key} was refused as billing. ${free ?? 'No promotion-covered model is available on this account, so nothing it serves will work until Credits are added.'}`,
|
|
265
|
+
latencyMs: Date.now() - started,
|
|
266
|
+
modelCount: entries.length,
|
|
267
|
+
};
|
|
268
|
+
}
|
|
269
|
+
const detail = error instanceof QoderUpstreamError ? `HTTP ${error.status}: ${error.message}` : error instanceof Error ? error.message : 'inference probe failed';
|
|
270
|
+
return { ok: false, detail: redactString(detail), latencyMs: Date.now() - started, modelCount: entries.length };
|
|
271
|
+
}
|
|
272
|
+
}
|
|
273
|
+
/** Null when the account has no promotion-covered model — the caller words that case itself. */
|
|
274
|
+
function freeModelHint(entries) {
|
|
275
|
+
const free = entries.filter((entry) => entry.isFree).map((entry) => entry.key);
|
|
276
|
+
return free.length > 0 ? `Free models (promotions) still work: ${free.join(', ')}` : null;
|
|
277
|
+
}
|
|
232
278
|
/** Catalog entries become discovered models; hidden (enable:false) keys stay routable. */
|
|
233
279
|
export function qoderModels(catalog) {
|
|
234
280
|
return [...catalog.entries.values()].map((entry) => ({
|
|
@@ -9,6 +9,8 @@ export const QODER_INFERENCE_BASE = 'https://api2.qoder.sh';
|
|
|
9
9
|
export const QODER_LOGIN_URL = 'https://qoder.com/account/integrations';
|
|
10
10
|
export const QODER_JOB_TOKEN_EXCHANGE_URL = `${QODER_OPENAPI_BASE}/api/v1/jobToken/exchange`;
|
|
11
11
|
export const QODER_USERINFO_URL = `${QODER_OPENAPI_BASE}/api/v1/userinfo`;
|
|
12
|
+
// Credits (Qoder's billing unit) live on openapi, not on the inference host.
|
|
13
|
+
export const QODER_CREDITS_URL = `${QODER_OPENAPI_BASE}/api/v2/quota/usage`;
|
|
12
14
|
export const QODER_CHAT_SIG_PATH = '/api/v2/service/pro/sse/agent_chat_generation';
|
|
13
15
|
export const QODER_CHAT_URL = `${QODER_INFERENCE_BASE}/algo${QODER_CHAT_SIG_PATH}?FetchKeys=llm_model_result&AgentId=agent_common&Encode=1`;
|
|
14
16
|
export const QODER_MODEL_LIST_URL = `${QODER_INFERENCE_BASE}/algo/api/v2/model/list`;
|
|
@@ -102,6 +102,17 @@ export async function qoderCredentialsFor(accountRecordId, options = {}) {
|
|
|
102
102
|
export async function qoderAttemptFailure(accountRecordId, error) {
|
|
103
103
|
const status = error.status ?? 502;
|
|
104
104
|
const billing = Boolean(error.billing);
|
|
105
|
+
// Billing is checked first because a quota refusal carries status 403 (code 112) and would
|
|
106
|
+
// otherwise be classified as a rejected credential: that mislabels the cause, force-re-exchanges
|
|
107
|
+
// a perfectly good PAT on every request, and hides the real answer (the account is out of quota).
|
|
108
|
+
if (billing) {
|
|
109
|
+
// Disabled, not merely degraded: a depleted account cannot serve paid models until its Credits
|
|
110
|
+
// refill, and leaving it eligible made the pool retry it on every request while the client saw
|
|
111
|
+
// "usage limited". The operator re-enables it (or adds Credits) from the admin UI; Credits
|
|
112
|
+
// refresh keeps the snapshot honest in the meantime.
|
|
113
|
+
setQoderAccountHealth(accountRecordId, 'down', 'out of Credits — re-enable after topping up', false);
|
|
114
|
+
throw new GatewayError('upstream_error', 'Qoder account is out of quota', { status: 502, code: 'qoder_billing_block' });
|
|
115
|
+
}
|
|
105
116
|
if (status === 401 || status === 403) {
|
|
106
117
|
// The PAT may still be valid — the job token may merely have expired mid-request. Force one
|
|
107
118
|
// exchange before condemning the account; a second rejection is the PAT's fault.
|
|
@@ -112,10 +123,6 @@ export async function qoderAttemptFailure(accountRecordId, error) {
|
|
|
112
123
|
}
|
|
113
124
|
throw new GatewayError('upstream_auth_error', 'Qoder job token was rejected after a refresh', { status: 502, code: 'qoder_auth_failed' });
|
|
114
125
|
}
|
|
115
|
-
if (billing) {
|
|
116
|
-
setQoderAccountHealth(accountRecordId, 'degraded', 'upstream reported a quota or billing block');
|
|
117
|
-
throw new GatewayError('upstream_error', 'Qoder account is out of quota', { status: 502, code: 'qoder_billing_block' });
|
|
118
|
-
}
|
|
119
126
|
throw upstreamHttpError(status, '', error);
|
|
120
127
|
}
|
|
121
128
|
/** Probe/discover share this: a catalog is the only proof the PAT works end to end. */
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
// Qoder Credits: the account's real usage. Qoder bills in Credits, not tokens — its chat
|
|
2
|
+
// stream carries no usage block at all — so the quota API is the only upstream truth about
|
|
3
|
+
// consumption. `is_free` on a catalog entry is the separate signal that a model is covered
|
|
4
|
+
// by a promotion (e.g. Qwen3.8-Max) and therefore does not draw on Credits at all.
|
|
5
|
+
import { QODER_CREDITS_URL } from './constants.js';
|
|
6
|
+
import { qoderCredentialsFor } from './credentials.js';
|
|
7
|
+
import { readQoderCreditsUpdatedAt, saveQoderCredits } from '../../db/repositories/qoder-accounts.js';
|
|
8
|
+
const FETCH_TIMEOUT_MS = 10_000;
|
|
9
|
+
const numberOr = (value, fallback) => {
|
|
10
|
+
const parsed = Number(value);
|
|
11
|
+
return Number.isFinite(parsed) ? parsed : fallback;
|
|
12
|
+
};
|
|
13
|
+
function isoOrNull(value) {
|
|
14
|
+
const ms = typeof value === 'number' ? value : Number(value);
|
|
15
|
+
// 253402214400000 is the far-future sentinel Qoder sends for "no expiry".
|
|
16
|
+
if (!Number.isFinite(ms) || ms >= 253402214400000)
|
|
17
|
+
return null;
|
|
18
|
+
return new Date(ms).toISOString();
|
|
19
|
+
}
|
|
20
|
+
function bucket(label, raw) {
|
|
21
|
+
if (!raw || typeof raw !== 'object')
|
|
22
|
+
return null;
|
|
23
|
+
const value = raw;
|
|
24
|
+
const total = numberOr(value.total, 0);
|
|
25
|
+
if (total <= 0)
|
|
26
|
+
return null;
|
|
27
|
+
const used = numberOr(value.used, 0);
|
|
28
|
+
return {
|
|
29
|
+
label,
|
|
30
|
+
total,
|
|
31
|
+
used,
|
|
32
|
+
remaining: numberOr(value.remaining, Math.max(0, total - used)),
|
|
33
|
+
unit: typeof value.unit === 'string' && value.unit ? value.unit : 'credits',
|
|
34
|
+
};
|
|
35
|
+
}
|
|
36
|
+
export function parseQoderCredits(body, freeModels, fetchedAt) {
|
|
37
|
+
const raw = (body && typeof body === 'object' ? body : {});
|
|
38
|
+
const buckets = [bucket('Plan credits', raw.userQuota), bucket('Add-on credits', raw.addOnQuota), bucket('Org resource package', raw.orgResourcePackage)].filter((entry) => entry !== null);
|
|
39
|
+
return {
|
|
40
|
+
userType: typeof raw.userType === 'string' ? raw.userType : 'unknown',
|
|
41
|
+
freeModels,
|
|
42
|
+
buckets,
|
|
43
|
+
totalUsedPercent: numberOr(raw.totalUsagePercentage, 0),
|
|
44
|
+
exhausted: raw.isQuotaExceeded === true,
|
|
45
|
+
expiresAt: isoOrNull(raw.expiresAt),
|
|
46
|
+
upgradeUrl: typeof raw.upgradeUrl === 'string' && raw.upgradeUrl ? raw.upgradeUrl : null,
|
|
47
|
+
fetchedAt,
|
|
48
|
+
};
|
|
49
|
+
}
|
|
50
|
+
/**
|
|
51
|
+
* Live Credits snapshot. A failure is reported as `unavailable` rather than thrown: Credits are
|
|
52
|
+
* observability, and an account that cannot report them must keep serving requests.
|
|
53
|
+
*/
|
|
54
|
+
export async function fetchQoderCredits(creds, freeModels, deps = {}) {
|
|
55
|
+
const fetchedAt = new Date().toISOString();
|
|
56
|
+
const controller = new AbortController();
|
|
57
|
+
const timer = setTimeout(() => controller.abort(), deps.timeoutMs ?? FETCH_TIMEOUT_MS);
|
|
58
|
+
try {
|
|
59
|
+
const response = await (deps.fetchImpl ?? fetch)(QODER_CREDITS_URL, {
|
|
60
|
+
method: 'GET',
|
|
61
|
+
headers: { authorization: `Bearer ${creds.jobToken}`, accept: 'application/json', 'user-agent': 'qodercli/1.0.0' },
|
|
62
|
+
signal: controller.signal,
|
|
63
|
+
});
|
|
64
|
+
if (!response.ok)
|
|
65
|
+
return { ...parseQoderCredits({}, freeModels, fetchedAt), unavailable: `HTTP ${response.status}` };
|
|
66
|
+
return parseQoderCredits(await response.json(), freeModels, fetchedAt);
|
|
67
|
+
}
|
|
68
|
+
catch (error) {
|
|
69
|
+
return { ...parseQoderCredits({}, freeModels, fetchedAt), unavailable: error instanceof Error ? error.message : 'Credits request failed' };
|
|
70
|
+
}
|
|
71
|
+
finally {
|
|
72
|
+
clearTimeout(timer);
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
/**
|
|
76
|
+
* Fetch and persist one account's Credits snapshot. Free models come from the account's cached
|
|
77
|
+
* catalog because `is_free` is a catalog property, not a quota one — it is why an account at zero
|
|
78
|
+
* Credits can still serve a promotional model.
|
|
79
|
+
*
|
|
80
|
+
* Lives here rather than in the admin route so the gateway can reuse it: Credits are the only
|
|
81
|
+
* upstream truth about consumption (Qoder's chat stream reports no usage at all), so the snapshot
|
|
82
|
+
* has to be kept fresh as accounts are actually used, not only when an admin opens the page.
|
|
83
|
+
*/
|
|
84
|
+
export async function refreshStoredQoderCredits(accountRecordId) {
|
|
85
|
+
const { config } = await qoderCredentialsFor(accountRecordId);
|
|
86
|
+
const freeModels = config.catalog ? [...config.catalog.entries.values()].filter((entry) => entry.isFree).map((entry) => entry.key) : [];
|
|
87
|
+
const credits = await fetchQoderCredits({ jobToken: config.jobToken }, freeModels, { timeoutMs: FETCH_TIMEOUT_MS });
|
|
88
|
+
saveQoderCredits(accountRecordId, credits.unavailable ? null : JSON.stringify(credits), credits.unavailable ?? null, credits.fetchedAt);
|
|
89
|
+
return credits;
|
|
90
|
+
}
|
|
91
|
+
/** Same throttle as Codex usage: the Credits endpoint is a second upstream call per account. */
|
|
92
|
+
export const QODER_CREDITS_REFRESH_INTERVAL_MS = 5 * 60 * 1000;
|
|
93
|
+
export async function refreshQoderCreditsIfStale(accountRecordId, now = new Date()) {
|
|
94
|
+
const row = readQoderCreditsUpdatedAt(accountRecordId);
|
|
95
|
+
const last = row ? Date.parse(row) : NaN;
|
|
96
|
+
if (Number.isFinite(last) && now.getTime() - last < QODER_CREDITS_REFRESH_INTERVAL_MS)
|
|
97
|
+
return;
|
|
98
|
+
// Never throws, matching `refreshStoredCodexUsage`: this runs in the background after a response
|
|
99
|
+
// has already been delivered, so a dead credential or unreachable quota API must not surface.
|
|
100
|
+
await refreshStoredQoderCredits(accountRecordId).catch(() => { });
|
|
101
|
+
}
|
|
@@ -274,6 +274,12 @@ export async function registerCodexRoutes(app) {
|
|
|
274
274
|
values.push(typeof value === 'boolean' ? (value ? 1 : 0) : value);
|
|
275
275
|
}
|
|
276
276
|
}
|
|
277
|
+
// Re-enabling must also clear a quota verdict: every selection path excludes `health_state='down'`,
|
|
278
|
+
// so flipping `enabled` alone would leave the account unreachable while the UI showed it as on.
|
|
279
|
+
if (body.enabled === true) {
|
|
280
|
+
fields.push('health_state=?');
|
|
281
|
+
values.push('unknown');
|
|
282
|
+
}
|
|
277
283
|
fields.push('updated_at=?');
|
|
278
284
|
values.push(new Date().toISOString(), id);
|
|
279
285
|
raw.prepare(`UPDATE codex_accounts SET ${fields.join(',')} WHERE id=?`).run(...values);
|