@goodandready/dsh-key-rotation 0.8.63 → 0.8.65

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,34 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.8.65 - 2026-10-18
4
+
5
+ ### Features
6
+ - **Traffic Smoothing Micro-Queue (#496)**:
7
+ - Leaky bucket traffic smoother (`lib/traffic-smoother.js`) absorbs short micro-bursts ($\le 1$s) across multi-agent workflows without tripping upstream per-second or per-minute rate limits or triggering cooldown churning.
8
+ - Configurable `trafficSmoothingBurst` (default 5 or `ceil(rpmLimit/12)`) and `maxQueueWaitMs` (default 800ms).
9
+ - If scheduled micro-queue delay exceeds `maxQueueWaitMs`, the smoother flags burst overflow, immediately rotating to the next candidate key in the pool.
10
+ - Live metric `smoothedRequests` exposed on provider status, with `TRAFFIC_SMOOTHED` audit events.
11
+ - **Accurate Dynamic Provider Limits & Canonical Presets (#503)**:
12
+ - Header limits detector (`lib/provider-limits-detector.js`): extracts dynamic rate limit headers (`x-ratelimit-limit-*`, `x-ratelimit-remaining-*`, `x-ratelimit-reset-*`, `anthropic-ratelimit-*`, `retry-after`) from upstream provider HTTP responses into `pool.state.detectedLimits`.
13
+ - Canonical provider presets catalog for DeepSeek, Groq, OpenCode, Anthropic, OpenAI, SiliconFlow, Mistral, Gemini, and OpenRouter with standard RPM, TPM, burst capacity, and calendar reset window specifications.
14
+ - Endpoint `GET/POST /dsh-key-rotation/preset-limits` to query and auto-apply canonical limits to configured providers.
15
+ - WebUI button `⚡ Preset Limits` for 1-click rate limit configuration and live upstream limits indicator badge (`⚡ 60 RPM · 100k TPM`).
16
+
17
+ ## 0.8.64 - 2026-10-18
18
+
19
+ ### Features
20
+ - **Permanent Auth Pause & Granular Error Policies (#501)**:
21
+ - Differentiated error actions: upstream authentication failures (HTTP 401, 403, `invalid_api_key`, `unauthorized`) immediately place the failing key into a permanent auth pause state (`pausedRefs.add`, `pauseReasons.set('AUTH_FAILURE_PAUSED')`, `failedUntil: Infinity`, `KEY_AUTH_PAUSED`).
22
+ - Dead or unauthorized keys are stopped from waking up repeatedly every 60s cooldown, preventing repetitive broken dispatches.
23
+ - Granular soft error handling: transient infrastructure drops (HTTP 502, 503, 504, `TIMEOUT`, socket drops) receive a short flat cooldown (<=10s) without doubling exponential penalty counters or incrementing toward recoverable quarantine.
24
+ - Successful Test Matrix / manual re-test probe automatically unpauses the key.
25
+ - **Automatic Model Capabilities Discovery & Routing (#500)**:
26
+ - In `lib/sandbox.js` (`probeModels`), parse and return array of supported model IDs from upstream `/models` (`body.data.map(m => m.id)`).
27
+ - Discovered model capabilities are cached per key (`pool.state.modelCapabilities: Map<ref, Set<modelId>>`).
28
+ - In `lib/resolver.js` and `lib/rotate.js`, filter candidate keys for a requested model so prompts are never dispatched to a key that doesn't support that model (e.g., DeepSeek-R1 vs DeepSeek-Chat V3). Keys not yet probed are optimistically allowed.
29
+ - Atomic persistence across restarts: `pausedRefs`, `pauseReasons`, and `modelCapabilities` serialize and restore atomically via `lib/persistence.js`.
30
+ - Settings Card UI: Display `Auth Paused (401/403)` status badge and discovered model count badges (`🤖 Nm`) on configured keys.
31
+
3
32
  ## 0.8.63 - 2026-10-16
4
33
 
5
34
  ### Added
package/README.md CHANGED
@@ -67,6 +67,10 @@
67
67
  - **📡 Status extras**: per-provider `circuit` plus `meta.expectedClones` / `meta.notifyQueue`.
68
68
  - **🧪 Smoke harness**: scripted 429 → next-key → success path in `test/smoke-rotation-080.test.mjs`.
69
69
 
70
+ ### 🚀 What's New in v0.8.65
71
+ - **⚡ Leaky Bucket Traffic Smoothing (#496)**: Micro-queue smoothing absorbs rapid multi-agent burst spikes ($\le 1$s) without triggering unnecessary 429 errors or cooldown churning, with zero-delay burst capacity and early failover rotation on queue overflow.
72
+ - **📊 Dynamic Upstream Rate Limits & Presets (#503)**: Dynamic parsing of provider rate limit headers (`x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after`), live badge display on keys, and 1-click `⚡ Preset Limits` button applying canonical RPM/TPM presets for DeepSeek, Groq, Anthropic, OpenAI, Gemini, and more.
73
+
70
74
  ### 🛠️ What's New in v0.7.33 (Stability & Bugfix Release)
71
75
  - **🔍 Resolved Key Probing BaseURL**: Fixed `resolveBaseUrl` to map key credential refs to owning provider pools, restoring live `probeModels` testing.
72
76
  - **🛡️ Guarded Cascade Recursion**: Prevented call stack overflow in cross-provider failover when circular cascade chains occur.
package/README.ru.md CHANGED
@@ -36,6 +36,10 @@
36
36
 
37
37
  ---
38
38
 
39
+ ### 🚀 Что нового в v0.8.65
40
+ - **⚡ Сглаживание трафика (Traffic Smoothing Micro-Queue) (#496)**: Микро-очередь leaky bucket сглаживает кратковременные пики нагрузки (<=1с) от мультиагентных вызовов без лишних ошибок 429 и лишних кулдаунов. При превышении таймаута ожидания микро-очереди запрос автоматически переключается на следующий ключ в пуле.
41
+ - **📊 Динамическое определение лимитов провайдеров и пресеты (#503)**: Автоматический парсинг заголовков `x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after` с отображением актуальных лимитов прямо на карточке ключа, а также кнопка в 1 клик `⚡ Preset Limits` с каноническими пресетами для DeepSeek, Groq, Anthropic, OpenAI, Gemini и др.
42
+
39
43
  ### 🚀 Что нового в v0.8.11 (обновление в один клик и quality gate)
40
44
 
41
45
  - **Обновление плагина в карточке настроек**: текущая/последняя версия и кнопка установки без выхода из DSH (#307).
package/README.zh.md CHANGED
@@ -38,6 +38,10 @@
38
38
 
39
39
  ## ⚡ 概述与核心痛点
40
40
 
41
+ ### 🚀 v0.8.65 新特性
42
+ - **⚡ 流量平滑微队列 (Traffic Smoothing Micro-Queue) (#496)**: 漏桶微队列平滑多智能体调用中的毫秒级突发并发,避免触发不必要的429限流与冷却流转。队列等待超限时自动轮转至池中下一个候选密钥。
43
+ - **📊 动态上游限制检测与预设规则 (#503)**: 自动提取上游响应头中的限流数据 (`x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after`),并在前端展示实时配额状态;支持一键应用 DeepSeek、Groq、Anthropic、OpenAI、Gemini 等官方规范预设限制。
44
+
41
45
  ### 🚀 v0.8.11 新特性(一键更新与质量门禁)
42
46
 
43
47
  - **设置卡片中的插件更新器**:查看当前/最新版本并一键安装(#307)。
package/lib/client.js CHANGED
@@ -98,6 +98,12 @@ window.__ModuleLoader__.load({
98
98
  bulkImportSuccess: 'Imported {n} keys',
99
99
  keyPaused: 'Paused',
100
100
  keyRevoked: 'Revoked (401)',
101
+ keyAuthPaused: 'Auth Paused (401/403)',
102
+ presetLimitsBtn: '⚡ Preset Limits',
103
+ presetApplied: 'Applied {prov} preset limits ({rpm} RPM / {tpm} TPM)',
104
+ presetNone: 'No canonical preset for this provider',
105
+ detectedLimitsLabel: 'Upstream: {rpm} RPM · {tpm} TPM',
106
+ trafficSmoothingLabel: 'Traffic Smoothing (Burst)',
101
107
  pauseKey: 'Pause key from rotation',
102
108
  resumeKey: 'Resume key rotation',
103
109
  sparklineTraffic: 'Traffic Trend',
@@ -299,6 +305,12 @@ window.__ModuleLoader__.load({
299
305
  bulkImportSuccess: '已导入 {n} 个密钥',
300
306
  keyPaused: '已暂停',
301
307
  keyRevoked: '已撤销 (401)',
308
+ keyAuthPaused: '身份验证失败已暂停 (401/403)',
309
+ presetLimitsBtn: '⚡ 预设限制',
310
+ presetApplied: '已应用 {prov} 预设限制 ({rpm} RPM / {tpm} TPM)',
311
+ presetNone: '未找到该提供商预设规则',
312
+ detectedLimitsLabel: '上游限制:{rpm} RPM · {tpm} TPM',
313
+ trafficSmoothingLabel: '流量平滑 (突发并发)',
302
314
  pauseKey: '暂停此密钥轮换',
303
315
  resumeKey: '恢复此密钥轮换',
304
316
  sparklineTraffic: '流量趋势',
@@ -1554,6 +1566,31 @@ window.__ModuleLoader__.load({
1554
1566
  };
1555
1567
 
1556
1568
  const [validating, setValidating] = React.useState('');
1569
+ const [presetNotice, setPresetNotice] = React.useState('');
1570
+ const applyPreset = async (pIndex, provName) => {
1571
+ try {
1572
+ const r = await fetch('/dsh-key-rotation/preset-limits?provider=' + encodeURIComponent(provName));
1573
+ const data = await r.json();
1574
+ if (data && data.preset) {
1575
+ setField((cur) => {
1576
+ const providers = [...(cur.providers ?? [])];
1577
+ const ent = { ...providers[pIndex] };
1578
+ if (data.preset.rpmLimit) ent.rpmLimit = data.preset.rpmLimit;
1579
+ if (data.preset.tpmLimit) ent.tpmLimit = data.preset.tpmLimit;
1580
+ providers[pIndex] = ent;
1581
+ return { ...cur, providers };
1582
+ });
1583
+ setPresetNotice(t('presetApplied').replace('{prov}', provName).replace('{rpm}', String(data.preset.rpmLimit)).replace('{tpm}', String(data.preset.tpmLimit)));
1584
+ setTimeout(() => setPresetNotice(''), 4000);
1585
+ } else {
1586
+ setPresetNotice(t('presetNone'));
1587
+ setTimeout(() => setPresetNotice(''), 3000);
1588
+ }
1589
+ } catch (err) {
1590
+ setPresetNotice(String(err?.message || err));
1591
+ setTimeout(() => setPresetNotice(''), 3000);
1592
+ }
1593
+ };
1557
1594
  const [validationResult, setValidationResult] = React.useState({});
1558
1595
  const validateBeforeSave = (ref, value) => {
1559
1596
  setValidating(ref);
@@ -1952,6 +1989,7 @@ window.__ModuleLoader__.load({
1952
1989
  if (isPaused) return { color: 'var(--dsw-alias-state-warn-primary)', text: t('keyPaused'), ready: false, paused: true };
1953
1990
  const hit = keyInfo(providerId, ref);
1954
1991
  if (!hit) return null;
1992
+ if (hit.authPaused || hit.pauseReason === 'AUTH_FAILURE_PAUSED') return { color: 'var(--dsw-alias-state-error-primary)', text: t('keyAuthPaused'), ready: false, authPaused: true };
1955
1993
  if (hit.paused) return { color: 'var(--dsw-alias-state-warn-primary)', text: t('keyPaused'), ready: false, paused: true };
1956
1994
  if (hit.expired) return { color: 'var(--dsw-alias-state-error-primary)', text: t('keyExpired'), ready: false, expired: true };
1957
1995
  if (hit.broken || hit.revoked || hit.status === 'revoked') return { color: 'var(--dsw-alias-state-error-primary)', text: t('keyRevoked'), ready: false, revoked: true };
@@ -2124,6 +2162,28 @@ window.__ModuleLoader__.load({
2124
2162
  if (info && info.broken) {
2125
2163
  meta.push(h('button', { key: 'rt', className: 'krot-btn', onClick: () => retestBroken(key), title: t('retestBroken') }, t('retestBroken')));
2126
2164
  }
2165
+ if (info && info.detectedLimits) {
2166
+ const dl = info.detectedLimits;
2167
+ const parts = [];
2168
+ if (dl.rpmLimit) parts.push(dl.rpmLimit + ' RPM');
2169
+ if (dl.tpmLimit) parts.push(dl.tpmLimit + ' TPM');
2170
+ if (parts.length > 0) {
2171
+ meta.push(h('span', {
2172
+ key: 'dl',
2173
+ className: 'krot-tail',
2174
+ title: 'Live upstream limits:\n' + JSON.stringify(dl, null, 2),
2175
+ style: { fontSize: '11px', opacity: 0.8, color: 'var(--dsw-alias-brand-primary)' }
2176
+ }, '⚡ ' + parts.join(' · ')));
2177
+ }
2178
+ }
2179
+ if (info && info.modelCapabilities && info.modelCapabilities.length > 0) {
2180
+ meta.push(h('span', {
2181
+ key: 'caps',
2182
+ className: 'krot-tail',
2183
+ title: 'Discovered models (' + info.modelCapabilities.length + '):\n' + info.modelCapabilities.join('\n'),
2184
+ style: { fontSize: '11px', opacity: 0.85 }
2185
+ }, '🤖 ' + info.modelCapabilities.length + 'm'));
2186
+ }
2127
2187
  meta.push(h('span', { key: 'a', className: 'krot-acts' },
2128
2188
  btn('↑', () => moveKey(pIndex, kIndex, -1), { disabled: kIndex === 0, title: t('moveUp') }),
2129
2189
  btn('↓', () => moveKey(pIndex, kIndex, 1), { disabled: kIndex === keys.length - 1, title: t('moveDown') }),
@@ -2357,6 +2417,8 @@ window.__ModuleLoader__.load({
2357
2417
  });
2358
2418
  }, { title: t('sortUsage') }),
2359
2419
  h('select', { className: 'krot-in', value: entry.provider, onChange: (e) => setProvider(pIndex, e.target.value) }, options),
2420
+ btn(t('presetLimitsBtn') || '⚡ Preset Limits', () => applyPreset(pIndex, entry.provider), { title: t('presetLimitsBtn') || '⚡ Preset Limits', disabled: !entry.provider.trim() || !stateRef.current.writable }),
2421
+ presetNotice ? h('span', { className: 'krot-badge krot-badge-ok', style: { marginLeft: '4px' } }, presetNotice) : null,
2360
2422
  (() => {
2361
2423
  const ps = status[entry.provider];
2362
2424
  const score = ps && typeof ps.healthScore === 'number' ? ps.healthScore : null;
@@ -36,9 +36,11 @@ export function classifyFailure(failureOrPayload) {
36
36
  if (status === 429 || code === 'RATE_LIMIT' || code === 'RESOURCE_EXHAUSTED' || code === 'QUOTA') {
37
37
  return { action: 'switch', code: status ? '429' : (code || 'RATE_LIMIT'), soft: false, reason: 'quota' };
38
38
  }
39
- if (SWITCH_STATUS.has(status)) {
40
- const soft = SOFT_FAILURE_CODES.has(String(status));
41
- return { action: 'switch', code: String(status), soft, reason: 'http' };
39
+ const numCode = Number(code);
40
+ if (SWITCH_STATUS.has(status) || (numCode > 0 && SWITCH_STATUS.has(numCode)) || SOFT_FAILURE_CODES.has(code)) {
41
+ const effCode = status ? String(status) : (numCode > 0 ? String(numCode) : code);
42
+ const soft = SOFT_FAILURE_CODES.has(effCode);
43
+ return { action: 'switch', code: effCode, soft, reason: 'http' };
42
44
  }
43
45
  if (SURFACE_STATUS.has(status)) {
44
46
  return { action: 'surface', code: String(status), soft: false, reason: 'client' };
package/lib/heal.js CHANGED
@@ -90,6 +90,10 @@ export async function autoUnbreakBrokenKeys(pools, probeFn, now = Date.now(), {
90
90
  pool.state.failedUntil?.delete(ref);
91
91
  pool.state.failCounts?.delete(ref);
92
92
  pool.state.authFailCounts?.delete(ref);
93
+ if (Array.isArray(res.models) && res.models.length > 0) {
94
+ if (!pool.state.modelCapabilities) pool.state.modelCapabilities = new Map();
95
+ pool.state.modelCapabilities.set(ref, new Set(res.models));
96
+ }
93
97
  if (Array.isArray(pool.state.events)) {
94
98
  pool.state.events.push({ at: now, ref, reason: 'auto-unbreak', cooldownMs: 0, type: 'heal' });
95
99
  if (pool.state.events.length > 50) pool.state.events.shift();
package/lib/index.js CHANGED
@@ -33,6 +33,7 @@ import { registerOpsRoutes } from './routes-ops.js';
33
33
  import { registerPluginUpdater } from './plugin-updater.js';
34
34
  import { notifySwitch, notifyExhaustion, notifyEarlyWarning, pushEvent } from './notify-events.js';
35
35
  import { QUOTA_WINDOW_TYPES } from './quota-window.js';
36
+ import { TrafficSmoother } from './traffic-smoother.js';
36
37
  import { getLogger } from './logger.js';
37
38
  import { createSandboxService } from './sandbox-service.js';
38
39
  import { checkBudgetAndHealthAlerts } from './budget-monitor.js';
@@ -198,6 +199,7 @@ export function apply(ctx, config = {}) {
198
199
  const webhookSender = new WebhookSender({ fetchImpl: globalThis.fetch });
199
200
  const alertDebouncer = new AlertDebouncer({ sender: webhookSender, debounceMs: 2000 });
200
201
  const moduleNotifyQueue = new NotifyQueue({ send: (url, payload) => alertDebouncer.enqueue(url, payload) });
202
+ const trafficSmoother = new TrafficSmoother({ maxQueueWaitMs: 800 });
201
203
 
202
204
  const sandboxService = createSandboxService({ getRuntime: () => buildRuntime() });
203
205
  const { ensureSandboxRunner, probeRef, lastTestCache } = sandboxService;
@@ -355,7 +357,7 @@ export function apply(ctx, config = {}) {
355
357
  switchNotifyThrottleMs: cfg.switchNotifyThrottleMs ?? 60000,
356
358
  warnBelowHealthy: cfg.warnBelowHealthy ?? 0, latencySloMs: cfg.latencySloMs ?? 0,
357
359
  index, pools,
358
- breaker: moduleBreaker, notifyQueue: moduleNotifyQueue, concurrencyTracker, quotaStore, latencyHistogram,
360
+ breaker: moduleBreaker, notifyQueue: moduleNotifyQueue, concurrencyTracker, quotaStore, latencyHistogram, trafficSmoother,
359
361
  providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider,
360
362
  cloneIds, expectedClones,
361
363
  circuitBreakerEnabled: cfg.circuitBreakerEnabled ?? true,
@@ -377,6 +379,7 @@ export function apply(ctx, config = {}) {
377
379
  buildRuntime,
378
380
  currentPool: () => dispatchStorage.getStore()?.pool,
379
381
  currentSession: () => dispatchStorage.getStore()?.sessionId,
382
+ currentModel: () => dispatchStorage.getStore()?.model,
380
383
  onPicked: (pool, candidate, meta) => {
381
384
  const store = dispatchStorage.getStore();
382
385
  if (store && store.pool === pool) {
@@ -433,6 +436,7 @@ export function apply(ctx, config = {}) {
433
436
  MARKER,
434
437
  finishError,
435
438
  setRotateStartMs: (v) => { _rotateStartMs = v; },
439
+ trafficSmoother,
436
440
  quotaStore,
437
441
  circuitBreaker: moduleBreaker,
438
442
  now: nowMono,
package/lib/ops-keys.js CHANGED
@@ -22,7 +22,9 @@ import {
22
22
  DRAIN_PATH,
23
23
  DISCOVER_PATH,
24
24
  AUTO_IMPORT_PATH,
25
+ PRESET_LIMITS_PATH,
25
26
  } from './ops-paths.js';
27
+ import { lookupPreset, CANONICAL_PROVIDER_PRESETS } from './provider-limits-detector.js';
26
28
  import fs from 'node:fs';
27
29
  import os from 'node:os';
28
30
  import path from 'node:path';
@@ -539,4 +541,40 @@ ctx.effect(() => ctx.webServer.register({
539
541
  },
540
542
  }), 'dsh-key-rotation: auto-import route');
541
543
 
544
+ ctx.effect(() => ctx.webServer.register({
545
+ kind: 'exact',
546
+ path: PRESET_LIMITS_PATH,
547
+ handler: async (req, res) => {
548
+ if (!isTrustedBridgeRequest(req)) {
549
+ res.writeHead(403, { 'Content-Type': 'application/json' });
550
+ res.end(JSON.stringify({ ok: false, error: 'Forbidden' }));
551
+ return;
552
+ }
553
+ if (req.method === 'GET') {
554
+ const urlObj = new URL(req.url, 'http://127.0.0.1');
555
+ const prov = urlObj.searchParams.get('provider');
556
+ if (prov) {
557
+ const preset = lookupPreset(prov);
558
+ json(res, 200, { ok: true, provider: prov, preset });
559
+ return;
560
+ }
561
+ json(res, 200, { ok: true, presets: CANONICAL_PROVIDER_PRESETS });
562
+ return;
563
+ }
564
+ if (req.method === 'POST') {
565
+ const body = await readJson(req);
566
+ const prov = body?.provider;
567
+ const preset = lookupPreset(prov);
568
+ if (!preset) {
569
+ json(res, 404, { ok: false, error: `No canonical preset found for provider '${prov}'` });
570
+ return;
571
+ }
572
+ json(res, 200, { ok: true, provider: prov, preset });
573
+ return;
574
+ }
575
+ res.writeHead(405, { 'Content-Type': 'application/json' });
576
+ res.end(JSON.stringify({ ok: false, error: 'Method Not Allowed' }));
577
+ },
578
+ }), 'dsh-key-rotation: preset limits route');
579
+
542
580
  }
package/lib/ops-paths.js CHANGED
@@ -17,3 +17,4 @@ export const DRAIN_PATH = '/dsh-key-rotation/drain';
17
17
  export const TEST_MATRIX_PATH = '/dsh-key-rotation/test-matrix';
18
18
  export const DISCOVER_PATH = '/dsh-key-rotation/discover';
19
19
  export const AUTO_IMPORT_PATH = '/dsh-key-rotation/auto-import';
20
+ export const PRESET_LIMITS_PATH = '/dsh-key-rotation/preset-limits';
package/lib/ops-status.js CHANGED
@@ -120,7 +120,10 @@ ctx.effect(() => ctx.webServer.register({
120
120
  expired: pool.expiresAt?.[ref] !== undefined && now >= pool.expiresAt[ref],
121
121
  broken: pool.state.brokenUntil?.has(ref) ?? false,
122
122
  paused: isKeyPaused(pool, ref),
123
+ pauseReason: pool.state.pauseReasons?.get?.(ref) ?? null,
123
124
  revoked: isKeyRevoked(pool, ref),
125
+ modelCapabilities: pool.state.modelCapabilities?.get?.(ref) ? [...pool.state.modelCapabilities.get(ref)] : null,
126
+ detectedLimits: pool.state.detectedLimits?.get?.(ref) ?? null,
124
127
  modelQuota,
125
128
  modelPools: activeModelPools.map(mp => mp.base),
126
129
  });
@@ -170,6 +173,7 @@ ctx.effect(() => ctx.webServer.register({
170
173
  pauseOnBudget: (providerBudgets?.get ? providerBudgets.get(pool.base) : providerBudgets?.[pool.base])?.pauseOnBudget ?? false,
171
174
  routingStrategy: pool.routingStrategy ?? runtime.routingStrategy ?? 'round-robin',
172
175
  proactiveRateLimitGuard: pool.proactiveRateLimitGuard ?? runtime.proactiveRateLimitGuard ?? true,
176
+ smoothedRequests: pool.state.smoothedRequests ?? 0,
173
177
  });
174
178
  } catch (e) {
175
179
  (ctx?.logger ? ctx.logger('dsh-key-rotation') : null)?.warn?.(`[dsh-key-rotation] status: pool ${pool.base} failed: ${String(e?.message ?? e)} ${e?.stack ?? ''}`);
package/lib/ops-test.js CHANGED
@@ -73,16 +73,24 @@ ctx.effect(() => ctx.webServer.register({
73
73
  lastTestCache.set(ref, cached);
74
74
  if (cached.ok) {
75
75
  for (const st of poolState.values()) {
76
- if (st.failedUntil?.has(ref) || st.failCounts?.has(ref) || st.brokenUntil?.has(ref) || st.revokedRefs?.has(ref)) {
77
- st.failedUntil?.delete(ref);
78
- st.failCounts?.delete(ref);
79
- st.authFailCounts?.delete(ref);
80
- st.brokenUntil?.delete(ref);
81
- st.revokedRefs?.delete(ref);
76
+ st.failedUntil?.delete(ref);
77
+ st.failCounts?.delete(ref);
78
+ st.authFailCounts?.delete(ref);
79
+ st.brokenUntil?.delete(ref);
80
+ st.revokedRefs?.delete(ref);
81
+ if (st.pausedRefs instanceof Set) st.pausedRefs.delete(ref);
82
+ else if (Array.isArray(st.pausedRefs)) {
83
+ const pIdx = st.pausedRefs.indexOf(ref);
84
+ if (pIdx >= 0) st.pausedRefs.splice(pIdx, 1);
85
+ }
86
+ st.pauseReasons?.delete(ref);
87
+ if (Array.isArray(cached.models) && cached.models.length > 0) {
88
+ if (!st.modelCapabilities) st.modelCapabilities = new Map();
89
+ st.modelCapabilities.set(ref, new Set(cached.models));
82
90
  }
83
91
  }
84
92
  }
85
- json(res, 200, { ok: cached.ok, ref, tail, source, probe, code: cached.code, latencyMs: cached.latencyMs, modelsCount: cached.modelsCount });
93
+ json(res, 200, { ok: cached.ok, ref, tail, source, probe, code: cached.code, latencyMs: cached.latencyMs, modelsCount: cached.modelsCount, models: cached.models ?? [] });
86
94
  return;
87
95
  }
88
96
  json(res, 200, { ok: true, ref, tail, source });
@@ -179,6 +187,16 @@ ctx.effect(() => ctx.webServer.register({
179
187
  st.authFailCounts?.delete(ref);
180
188
  st.brokenUntil?.delete(ref);
181
189
  st.revokedRefs?.delete(ref);
190
+ if (st.pausedRefs instanceof Set) st.pausedRefs.delete(ref);
191
+ else if (Array.isArray(st.pausedRefs)) {
192
+ const pIdx = st.pausedRefs.indexOf(ref);
193
+ if (pIdx >= 0) st.pausedRefs.splice(pIdx, 1);
194
+ }
195
+ st.pauseReasons?.delete(ref);
196
+ if (Array.isArray(cached.models) && cached.models.length > 0) {
197
+ if (!st.modelCapabilities) st.modelCapabilities = new Map();
198
+ st.modelCapabilities.set(ref, new Set(cached.models));
199
+ }
182
200
  }
183
201
  }
184
202
  results.push({
@@ -189,6 +207,7 @@ ctx.effect(() => ctx.webServer.register({
189
207
  code: cached.code,
190
208
  latencyMs: cached.latencyMs,
191
209
  modelsCount: cached.modelsCount,
210
+ models: cached.models ?? [],
192
211
  });
193
212
  } catch (err) {
194
213
  results.push({
@@ -44,6 +44,32 @@ export class StatePersistence {
44
44
  revokedRefs.push(...st.revokedRefs);
45
45
  }
46
46
  }
47
+ const pausedRefs = [];
48
+ if (st && st.pausedRefs) {
49
+ if (typeof st.pausedRefs.forEach === 'function') {
50
+ st.pausedRefs.forEach((r) => pausedRefs.push(r));
51
+ } else if (Array.isArray(st.pausedRefs)) {
52
+ pausedRefs.push(...st.pausedRefs);
53
+ }
54
+ }
55
+ const pauseReasons = {};
56
+ if (st && st.pauseReasons && typeof st.pauseReasons.forEach === 'function') {
57
+ st.pauseReasons.forEach((reason, ref) => {
58
+ if (typeof reason === 'string') pauseReasons[ref] = reason;
59
+ });
60
+ }
61
+ const modelCapabilities = {};
62
+ if (st && st.modelCapabilities && typeof st.modelCapabilities.forEach === 'function') {
63
+ st.modelCapabilities.forEach((caps, ref) => {
64
+ if (caps && typeof caps.forEach === 'function') {
65
+ const arr = [];
66
+ caps.forEach((m) => { if (typeof m === 'string') arr.push(m); });
67
+ if (arr.length > 0) modelCapabilities[ref] = arr;
68
+ } else if (Array.isArray(caps) && caps.length > 0) {
69
+ modelCapabilities[ref] = caps.filter((m) => typeof m === 'string');
70
+ }
71
+ });
72
+ }
47
73
  const tokenUsage = {};
48
74
  if (st && st.tokenUsage && typeof st.tokenUsage.forEach === 'function') {
49
75
  st.tokenUsage.forEach((entry, ref) => {
@@ -109,6 +135,9 @@ export class StatePersistence {
109
135
  lastUsed: typeof st?.lastUsed === 'string' ? st.lastUsed : null,
110
136
  };
111
137
  if (revokedRefs.length > 0) outPool.revokedRefs = revokedRefs;
138
+ if (pausedRefs.length > 0) outPool.pausedRefs = pausedRefs;
139
+ if (Object.keys(pauseReasons).length > 0) outPool.pauseReasons = pauseReasons;
140
+ if (Object.keys(modelCapabilities).length > 0) outPool.modelCapabilities = modelCapabilities;
112
141
  if (Object.keys(tokenUsage).length > 0) outPool.tokenUsage = tokenUsage;
113
142
  if (Object.keys(costDays).length > 0) outPool.costDays = costDays;
114
143
  if (Object.keys(costPerKey).length > 0) outPool.costPerKey = costPerKey;
@@ -200,6 +229,26 @@ export class StatePersistence {
200
229
  if (typeof r === 'string') state.revokedRefs.add(r);
201
230
  }
202
231
  }
232
+ if (Array.isArray(saved.pausedRefs)) {
233
+ if (!state.pausedRefs) state.pausedRefs = new Set();
234
+ for (const r of saved.pausedRefs) {
235
+ if (typeof r === 'string') state.pausedRefs.add(r);
236
+ }
237
+ }
238
+ if (saved.pauseReasons && typeof saved.pauseReasons === 'object' && !Array.isArray(saved.pauseReasons)) {
239
+ if (!state.pauseReasons) state.pauseReasons = new Map();
240
+ for (const [ref, reason] of Object.entries(saved.pauseReasons)) {
241
+ if (typeof reason === 'string') state.pauseReasons.set(ref, reason);
242
+ }
243
+ }
244
+ if (saved.modelCapabilities && typeof saved.modelCapabilities === 'object' && !Array.isArray(saved.modelCapabilities)) {
245
+ if (!state.modelCapabilities) state.modelCapabilities = new Map();
246
+ for (const [ref, list] of Object.entries(saved.modelCapabilities)) {
247
+ if (Array.isArray(list)) {
248
+ state.modelCapabilities.set(ref, new Set(list.filter((m) => typeof m === 'string')));
249
+ }
250
+ }
251
+ }
203
252
  if (saved.tokenUsage && typeof saved.tokenUsage === 'object' && !Array.isArray(saved.tokenUsage)) {
204
253
  for (const [ref, entry] of Object.entries(saved.tokenUsage)) {
205
254
  if (!entry || typeof entry !== 'object' || Array.isArray(entry)) continue;
@@ -37,6 +37,8 @@ export function buildPoolItem({
37
37
  labels = null,
38
38
  pinnedRef = null,
39
39
  useKeyRef = null,
40
+ trafficSmoothingBurst = null,
41
+ maxQueueWaitMs = null,
40
42
  }) {
41
43
  const seen = new Set();
42
44
  const indexed = (Array.isArray(keys) ? keys : []).map((key, index) => ({ ref: typeof key === 'string' ? key.trim() : '', index }))
@@ -135,6 +137,12 @@ const weightsMap = {};
135
137
  proactiveRateLimitGuard: poolGuard,
136
138
  labels: labelsMap,
137
139
  pinnedRef: cleanPinned,
140
+ trafficSmoothingBurst: typeof trafficSmoothingBurst === 'number' && Number.isFinite(trafficSmoothingBurst) && trafficSmoothingBurst > 0
141
+ ? Math.floor(trafficSmoothingBurst)
142
+ : (typeof rpmLimit === 'number' && rpmLimit > 0 ? Math.max(2, Math.ceil(rpmLimit / 12)) : 5),
143
+ maxQueueWaitMs: typeof maxQueueWaitMs === 'number' && Number.isFinite(maxQueueWaitMs) && maxQueueWaitMs >= 0
144
+ ? Math.floor(maxQueueWaitMs)
145
+ : 800,
138
146
  };
139
147
  }
140
148
 
@@ -175,6 +183,8 @@ export function buildPools({ cfg = {}, poolState = new Map(), defaultState = ini
175
183
  model: null,
176
184
  labels: p.labels,
177
185
  pinnedRef: p.pinnedRef ?? p.useKeyRef,
186
+ trafficSmoothingBurst: p.trafficSmoothingBurst ?? cfg.trafficSmoothingBurst,
187
+ maxQueueWaitMs: p.maxQueueWaitMs ?? cfg.maxQueueWaitMs,
178
188
  }, provRpm, provTpm, provConcurrency);
179
189
  if (pool) {
180
190
  for (const ref of pool.refs) {
@@ -203,6 +213,8 @@ export function buildPools({ cfg = {}, poolState = new Map(), defaultState = ini
203
213
  quotas: mp.quotas,
204
214
  labels: mp.labels ?? p.labels,
205
215
  pinnedRef: mp.pinnedRef ?? mp.useKeyRef ?? p.pinnedRef ?? p.useKeyRef,
216
+ trafficSmoothingBurst: mp.trafficSmoothingBurst ?? p.trafficSmoothingBurst ?? cfg.trafficSmoothingBurst,
217
+ maxQueueWaitMs: mp.maxQueueWaitMs ?? p.maxQueueWaitMs ?? cfg.maxQueueWaitMs,
206
218
  }, modelRpm, modelTpm, modelConcurrency);
207
219
  if (!mpool) continue;
208
220
  mpool.basePool = pool;
package/lib/pool-state.js CHANGED
@@ -3,13 +3,18 @@
3
3
  const MAP_FIELDS = [
4
4
  'failedUntil', 'failCounts', 'authFailCounts', 'brokenUntil', 'costPerKey',
5
5
  'lastUsedAt', 'usageCounts', 'byModel', 'usageDays', 'quotaWindows', 'costDays',
6
- 'tokenUsage', 'rpmWindows', 'tpmWindows', 'sessionSticky',
6
+ 'tokenUsage', 'rpmWindows', 'tpmWindows', 'sessionSticky', 'modelCapabilities', 'pauseReasons', 'detectedLimits',
7
7
  ];
8
8
 
9
+ const SET_FIELDS = ['pausedRefs', 'revokedRefs'];
10
+
9
11
  export function initializePoolState(state = {}) {
10
12
  for (const key of MAP_FIELDS) {
11
13
  if (!(state[key] instanceof Map)) state[key] = new Map();
12
14
  }
15
+ for (const key of SET_FIELDS) {
16
+ if (!(state[key] instanceof Set)) state[key] = new Set(Array.isArray(state[key]) ? state[key] : []);
17
+ }
13
18
  if (!Array.isArray(state.events)) state.events = [];
14
19
  if (!Number.isSafeInteger(state.pointer) || state.pointer < 0) state.pointer = 0;
15
20
  if (typeof state.lastUsed !== 'string') state.lastUsed = null;
@@ -21,5 +26,7 @@ export function initializePoolState(state = {}) {
21
26
  export function hasPoolActivity(state) {
22
27
  return state.pointer !== 0 || state.lastUsed !== null || state.events.length > 0 ||
23
28
  MAP_FIELDS.some((key) => state[key].size > 0) ||
24
- (state.lastSuccessAt?.size ?? 0) > 0;
29
+ (state.lastSuccessAt?.size ?? 0) > 0 ||
30
+ (state.pausedRefs?.size ?? 0) > 0 ||
31
+ (state.revokedRefs?.size ?? 0) > 0;
25
32
  }
@@ -0,0 +1,173 @@
1
+ // lib/provider-limits-detector.js — accurate dynamic provider limits and presets (#503).
2
+
3
+ export const CANONICAL_PROVIDER_PRESETS = {
4
+ deepseek: {
5
+ rpmLimit: 60,
6
+ tpmLimit: 100000,
7
+ trafficSmoothingBurst: 5,
8
+ quotaResetWindow: 'midnight_utc',
9
+ displayName: 'DeepSeek Official',
10
+ },
11
+ groq: {
12
+ rpmLimit: 30,
13
+ tpmLimit: 30000,
14
+ trafficSmoothingBurst: 5,
15
+ quotaResetWindow: 'per-minute',
16
+ displayName: 'Groq Cloud',
17
+ },
18
+ opencode: {
19
+ rpmLimit: 120,
20
+ tpmLimit: 200000,
21
+ trafficSmoothingBurst: 10,
22
+ quotaResetWindow: 'midnight_utc',
23
+ displayName: 'OpenCode Zen',
24
+ },
25
+ anthropic: {
26
+ rpmLimit: 50,
27
+ tpmLimit: 100000,
28
+ trafficSmoothingBurst: 5,
29
+ quotaResetWindow: 'per-minute',
30
+ displayName: 'Anthropic Claude',
31
+ },
32
+ openai: {
33
+ rpmLimit: 60,
34
+ tpmLimit: 150000,
35
+ trafficSmoothingBurst: 10,
36
+ quotaResetWindow: 'per-minute',
37
+ displayName: 'OpenAI API',
38
+ },
39
+ siliconflow: {
40
+ rpmLimit: 60,
41
+ tpmLimit: 100000,
42
+ trafficSmoothingBurst: 5,
43
+ quotaResetWindow: 'per-minute',
44
+ displayName: 'SiliconFlow',
45
+ },
46
+ mistral: {
47
+ rpmLimit: 60,
48
+ tpmLimit: 100000,
49
+ trafficSmoothingBurst: 5,
50
+ quotaResetWindow: 'per-minute',
51
+ displayName: 'Mistral AI',
52
+ },
53
+ gemini: {
54
+ rpmLimit: 60,
55
+ tpmLimit: 1000000,
56
+ trafficSmoothingBurst: 10,
57
+ quotaResetWindow: 'midnight_pst',
58
+ displayName: 'Google Gemini',
59
+ },
60
+ openrouter: {
61
+ rpmLimit: 200,
62
+ tpmLimit: 500000,
63
+ trafficSmoothingBurst: 15,
64
+ quotaResetWindow: 'per-minute',
65
+ displayName: 'OpenRouter',
66
+ },
67
+ };
68
+
69
+ /**
70
+ * Extract rate limit and quota information from HTTP response headers.
71
+ * @param {Headers|Record<string, string>} rawHeaders
72
+ * @returns {object|null}
73
+ */
74
+ export function extractProviderLimits(rawHeaders) {
75
+ if (!rawHeaders) return null;
76
+ const headers = {};
77
+ if (typeof rawHeaders.forEach === 'function') {
78
+ rawHeaders.forEach((val, key) => { headers[key.toLowerCase()] = String(val); });
79
+ } else if (typeof rawHeaders === 'object') {
80
+ for (const [k, v] of Object.entries(rawHeaders)) {
81
+ if (typeof k === 'string') headers[k.toLowerCase()] = String(v ?? '');
82
+ }
83
+ }
84
+
85
+ const getNum = (...names) => {
86
+ for (const name of names) {
87
+ const val = headers[name];
88
+ if (val !== undefined && val !== null && val !== '') {
89
+ const n = Number(val);
90
+ if (Number.isFinite(n) && n >= 0) return n;
91
+ }
92
+ }
93
+ return undefined;
94
+ };
95
+
96
+ const getSeconds = (...names) => {
97
+ for (const name of names) {
98
+ const val = headers[name];
99
+ if (!val) continue;
100
+ const str = String(val).trim();
101
+ // Duration string format "6m0s", "1s", "250ms"
102
+ const match = /^(\d+(?:\.\d+)?)(s|ms|m|h)?$/i.exec(str);
103
+ if (match) {
104
+ const num = parseFloat(match[1]);
105
+ const unit = (match[2] || 's').toLowerCase();
106
+ if (unit === 'ms') return Math.max(0.001, num / 1000);
107
+ if (unit === 's') return num;
108
+ if (unit === 'm') return num * 60;
109
+ if (unit === 'h') return num * 3600;
110
+ }
111
+ // Check for ISO Date string (e.g., Anthropic reset timestamp)
112
+ const parsedDate = Date.parse(str);
113
+ if (Number.isFinite(parsedDate)) {
114
+ const diffSec = (parsedDate - Date.now()) / 1000;
115
+ return Math.max(0, diffSec);
116
+ }
117
+ const n = Number(str);
118
+ if (Number.isFinite(n) && n >= 0) return n;
119
+ }
120
+ return undefined;
121
+ };
122
+
123
+ const rpmLimit = getNum('x-ratelimit-limit-requests', 'x-ratelimit-limit-rpm', 'ratelimit-limit', 'anthropic-ratelimit-requests-limit');
124
+ const tpmLimit = getNum('x-ratelimit-limit-tokens', 'x-ratelimit-limit-tpm', 'anthropic-ratelimit-tokens-limit');
125
+ const remainingRequests = getNum('x-ratelimit-remaining-requests', 'x-ratelimit-remaining', 'anthropic-ratelimit-requests-remaining');
126
+ const remainingTokens = getNum('x-ratelimit-remaining-tokens', 'anthropic-ratelimit-tokens-remaining');
127
+ const resetRequests = getSeconds('x-ratelimit-reset-requests', 'x-ratelimit-reset', 'anthropic-ratelimit-requests-reset');
128
+ const resetTokens = getSeconds('x-ratelimit-reset-tokens', 'anthropic-ratelimit-tokens-reset');
129
+ const retryAfter = getSeconds('retry-after');
130
+
131
+ if (rpmLimit === undefined && tpmLimit === undefined && remainingRequests === undefined && remainingTokens === undefined && resetRequests === undefined && retryAfter === undefined) {
132
+ return null;
133
+ }
134
+
135
+ const out = {};
136
+ if (rpmLimit !== undefined) out.rpmLimit = Math.floor(rpmLimit);
137
+ if (tpmLimit !== undefined) out.tpmLimit = Math.floor(tpmLimit);
138
+ if (remainingRequests !== undefined) {
139
+ out.remainingRequests = Math.floor(remainingRequests);
140
+ out.rpmRemaining = out.remainingRequests;
141
+ }
142
+ if (remainingTokens !== undefined) {
143
+ out.remainingTokens = Math.floor(remainingTokens);
144
+ out.tpmRemaining = out.remainingTokens;
145
+ }
146
+ if (resetRequests !== undefined) {
147
+ out.resetRequests = resetRequests;
148
+ out.rpmResetMs = Math.round(resetRequests * 1000);
149
+ }
150
+ if (resetTokens !== undefined) {
151
+ out.resetTokens = resetTokens;
152
+ out.tpmResetMs = Math.round(resetTokens * 1000);
153
+ }
154
+ if (retryAfter !== undefined) {
155
+ out.retryAfterMs = Math.round(retryAfter * 1000);
156
+ }
157
+
158
+ return out;
159
+ }
160
+
161
+ /**
162
+ * Match a configured provider name to canonical preset.
163
+ * @param {string} providerName
164
+ * @returns {object|null}
165
+ */
166
+ export function lookupPreset(providerName) {
167
+ if (!providerName || typeof providerName !== 'string') return null;
168
+ const p = providerName.toLowerCase().replace(/[^a-z0-9]/g, '');
169
+ for (const [key, val] of Object.entries(CANONICAL_PROVIDER_PRESETS)) {
170
+ if (p.includes(key)) return { ...val, id: key };
171
+ }
172
+ return null;
173
+ }
package/lib/resolver.js CHANGED
@@ -77,6 +77,7 @@ export function createResolver({
77
77
  buildRuntime,
78
78
  currentPool,
79
79
  currentSession,
80
+ currentModel,
80
81
  onPicked,
81
82
  latencyHistogram,
82
83
  concurrencyTracker,
@@ -88,6 +89,7 @@ export function createResolver({
88
89
  // pool that rotate() actually dispatched with decides ownership. A bare
89
90
  // resolve outside rotation falls back to the global lookup.
90
91
  const requested = currentPool ? currentPool() ?? null : null;
92
+ const model = resolveOpts?.model ?? currentModel?.() ?? requested?.model ?? null;
91
93
  const pool = resolveRefPool(runtime.index, ref, requested);
92
94
  if (!pool) {
93
95
  const res = await original(ref);
@@ -168,6 +170,11 @@ export function createResolver({
168
170
  if (until !== undefined && until > at) continue;
169
171
  const candidateExp = pool.expiresAt?.[candidate] ?? pool.basePool?.expiresAt?.[candidate];
170
172
  if (candidateExp !== undefined && at >= candidateExp) continue;
173
+ // Issue #500: skip candidate if modelCapabilities is known and does not include requested model
174
+ if (model && pool.state.modelCapabilities) {
175
+ const caps = pool.state.modelCapabilities.get(candidate);
176
+ if (caps && caps.size > 0 && !caps.has(model)) continue;
177
+ }
171
178
  // Local model token budget: a spent credential is simply not a candidate.
172
179
  if (quotaManaged && !isModelQuotaAvailable(pool, candidate, at)) continue;
173
180
  const microThrottling = pool.microThrottling ?? runtime.microThrottling ?? false;
package/lib/rotate.js CHANGED
@@ -35,6 +35,7 @@ import {
35
35
  } from './model-quota.js';
36
36
 
37
37
  import { LOCAL_POOL_EXHAUSTED_CODE } from './resolver.js';
38
+ import { extractProviderLimits } from './provider-limits-detector.js';
38
39
 
39
40
  export const LOCAL_QUOTA_CODE = 'LOCAL_MODEL_QUOTA_EXHAUSTED';
40
41
 
@@ -79,6 +80,7 @@ export function createRotate(deps) {
79
80
  circuitBreaker,
80
81
  logger,
81
82
  schedulePersist,
83
+ trafficSmoother,
82
84
  now = nowMono,
83
85
  } = deps;
84
86
 
@@ -108,12 +110,19 @@ export function createRotate(deps) {
108
110
  const breaker = runtime0.circuitBreakerEnabled === false ? null : circuitBreaker;
109
111
  let activePermit = null;
110
112
  const sessionId = options?.sessionId ?? options?.session ?? options?.metadata?.sessionId ?? options?.chatId ?? options?.conversationId;
111
- const reqStore = { pool, pickedRef: undefined, concurrencyRef: undefined, startMs: now(), sessionId };
113
+ const reqStore = { pool, pickedRef: undefined, concurrencyRef: undefined, startMs: now(), sessionId, model: options.model };
112
114
  setRotateStartMs(reqStore.startMs);
113
115
  const wallNow = Date.now();
114
116
  const quotaManaged = hasModelQuotaConfig(pool);
115
117
  let attemptList = (pool.weightedRefs ?? pool.refs).slice();
116
118
  attemptList = attemptList.filter((r) => !isKeyPaused(pool, r) && !isKeyRevoked(pool, r) && !isKeyBroken(pool, r, wallNow) && !isKeyDraining(r));
119
+ // Issue #500: filter candidate keys by confirmed model capabilities
120
+ if (options.model && pool.state.modelCapabilities) {
121
+ attemptList = attemptList.filter((ref) => {
122
+ const caps = pool.state.modelCapabilities.get(ref);
123
+ return !caps || caps.size === 0 || caps.has(options.model);
124
+ });
125
+ }
117
126
  if (pool.pinnedRef && attemptList.includes(pool.pinnedRef)) {
118
127
  const pIdx = attemptList.indexOf(pool.pinnedRef);
119
128
  if (pIdx > 0) {
@@ -196,7 +205,7 @@ export function createRotate(deps) {
196
205
  const _base = pool.cooldownMs ?? cooldownMs;
197
206
  const _max = pool.maxCooldownMs ?? maxCooldownMs;
198
207
  const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
199
- const cls = classifyFailure({ code: errCode, message: errMsg });
208
+ const cls = classifyFailure({ status: Number(errCode) || 0, code: errCode, message: errMsg });
200
209
  const _b = recordFailure(pool, targetRef, now(), _effBase, _max, cls.soft, true);
201
210
 
202
211
  // #337 & #486 (GitHub #24): connect quota-window calendar reset for quota exhaustion
@@ -245,16 +254,26 @@ export function createRotate(deps) {
245
254
  }
246
255
  if (!pool.state.authFailCounts) pool.state.authFailCounts = new Map();
247
256
  if (!pool.state.brokenUntil) pool.state.brokenUntil = new Map();
248
- const _cStr = String(errCode ?? '');
249
- if (_cStr === 'AUTH' || /auth|unauthorized|invalid[_\s-]?api[_\s-]?key|unauthenticated/i.test(errMsg)) {
257
+ const _cStr = String(errCode ?? '').toUpperCase();
258
+ const _isAuth = cls.reason === 'auth' || cls.code === '401' || cls.code === '403' ||
259
+ _cStr === 'AUTH' || _cStr === '401' || _cStr === '403' ||
260
+ _cStr === 'PERMISSION_DENIED' || _cStr === 'UNAUTHENTICATED' ||
261
+ /auth|unauthorized|invalid[_\s-]?api[_\s-]?key|unauthenticated|permission[_\s-]?denied/i.test(errMsg);
262
+
263
+ if (_isAuth) {
250
264
  const _c2 = (pool.state.authFailCounts.get(targetRef) ?? 0) + 1;
251
265
  pool.state.authFailCounts.set(targetRef, _c2);
252
266
  if (!pool.state.revokedRefs) pool.state.revokedRefs = new Set();
253
267
  pool.state.revokedRefs.add(targetRef);
268
+ if (!pool.state.pausedRefs) pool.state.pausedRefs = new Set();
269
+ if (pool.state.pausedRefs instanceof Set) pool.state.pausedRefs.add(targetRef);
270
+ else if (Array.isArray(pool.state.pausedRefs) && !pool.state.pausedRefs.includes(targetRef)) pool.state.pausedRefs.push(targetRef);
271
+ if (!pool.state.pauseReasons) pool.state.pauseReasons = new Map();
272
+ pool.state.pauseReasons.set(targetRef, 'AUTH_FAILURE_PAUSED');
254
273
  pool.state.failedUntil.set(targetRef, Infinity);
255
- pushEvent(pool, targetRef, 'KEY_REVOKED', 0);
256
- logWarn(`[dsh-key-rotation] ${options.provider}: key ${targetRef} permanently REVOKED on auth failure`);
257
- // Fan-out revocation across all pools sharing this physical ref (#408):
274
+ pushEvent(pool, targetRef, 'KEY_AUTH_PAUSED', 0);
275
+ logWarn(`[dsh-key-rotation] ${options.provider}: key ${targetRef} permanently PAUSED on auth failure (${errCode})`);
276
+ // Fan-out permanent pause across all pools sharing this physical ref (#408):
258
277
  const rAll = buildRuntime();
259
278
  const candidatePools = [];
260
279
  if (Array.isArray(rAll.pools)) candidatePools.push(...rAll.pools);
@@ -275,8 +294,13 @@ export function createRotate(deps) {
275
294
  if (hasRef && other.state) {
276
295
  if (!other.state.revokedRefs) other.state.revokedRefs = new Set();
277
296
  other.state.revokedRefs.add(targetRef);
297
+ if (!other.state.pausedRefs) other.state.pausedRefs = new Set();
298
+ if (other.state.pausedRefs instanceof Set) other.state.pausedRefs.add(targetRef);
299
+ else if (Array.isArray(other.state.pausedRefs) && !other.state.pausedRefs.includes(targetRef)) other.state.pausedRefs.push(targetRef);
300
+ if (!other.state.pauseReasons) other.state.pauseReasons = new Map();
301
+ other.state.pauseReasons.set(targetRef, 'AUTH_FAILURE_PAUSED');
278
302
  if (other.state.failedUntil) other.state.failedUntil.set(targetRef, Infinity);
279
- pushEvent(other, targetRef, 'KEY_REVOKED', 0);
303
+ pushEvent(other, targetRef, 'KEY_AUTH_PAUSED', 0);
280
304
  }
281
305
  }
282
306
  } else {
@@ -296,6 +320,25 @@ export function createRotate(deps) {
296
320
  };
297
321
 
298
322
  for (let attempt = 0; attempt < attemptList.length; attempt++) {
323
+ const candidateRef = attemptList[attempt];
324
+ // #496: Traffic smoothing micro-queue
325
+ const smoother = pool.trafficSmoother ?? runtime0.trafficSmoother ?? trafficSmoother;
326
+ if (smoother && candidateRef) {
327
+ const smoothRes = await smoother.acquire(candidateRef, {
328
+ rpmLimit: pool.rpmLimit,
329
+ burstLimit: pool.trafficSmoothingBurst,
330
+ maxQueueWaitMs: pool.maxQueueWaitMs,
331
+ signal: options.signal,
332
+ });
333
+ if (smoothRes.overflow) {
334
+ logWarn(`[dsh-key-rotation] ${options.provider}: key ${candidateRef} micro-queue burst overflow (${smoothRes.delayMs}ms) — rotating to next candidate`);
335
+ continue;
336
+ }
337
+ if (smoothRes.delayed) {
338
+ pool.state.smoothedRequests = (pool.state.smoothedRequests ?? 0) + 1;
339
+ pushEvent(pool, candidateRef, 'TRAFFIC_SMOOTHED', smoothRes.delayMs);
340
+ }
341
+ }
299
342
  // #464: Check deadline & attempt budget before dispatch
300
343
  if (!budget.canAttempt()) {
301
344
  const isTimeout = budget.isExpired();
@@ -484,6 +527,15 @@ export function createRotate(deps) {
484
527
  // Proactive rate-limit (#115, #303): if response headers say this key is near
485
528
  // its quota or has retry-after, cool it down so the NEXT request starts on a different key.
486
529
  // We do NOT re-run this (already successful) request — that would double-send.
530
+ // #503: Extract provider limits and reset headers
531
+ const rawHdrs = chunk?.metadata?.headers ?? chunk?.headers;
532
+ if (rawHdrs && activeRef) {
533
+ const detected = extractProviderLimits(rawHdrs);
534
+ if (detected) {
535
+ if (!pool.state.detectedLimits) pool.state.detectedLimits = new Map();
536
+ pool.state.detectedLimits.set(activeRef, { ...detected, at: now() });
537
+ }
538
+ }
487
539
  const guardEnabled = pool.proactiveRateLimitGuard ?? runtime0.proactiveRateLimitGuard ?? true;
488
540
  const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
489
541
  if (guardEnabled && rate && activeRef) {
package/lib/sandbox.js CHANGED
@@ -107,11 +107,15 @@ export class SandboxRunner {
107
107
  const status = res.status;
108
108
  if (status >= 200 && status < 300) {
109
109
  let modelsCount = 0;
110
+ let models = [];
110
111
  try {
111
112
  const body = await res.json();
112
- modelsCount = Array.isArray(body && body.data) ? body.data.length : 0;
113
+ if (Array.isArray(body && body.data)) {
114
+ modelsCount = body.data.length;
115
+ models = body.data.map((m) => (typeof m === 'string' ? m : (m && m.id))).filter(Boolean);
116
+ }
113
117
  } catch (_) { /* not json */ }
114
- return { ok: true, code: 'ok', latencyMs: Date.now() - started, modelsCount };
118
+ return { ok: true, code: 'ok', latencyMs: Date.now() - started, modelsCount, models };
115
119
  }
116
120
  return { ok: false, code: classifyStatus(status), latencyMs: Date.now() - started };
117
121
  } finally {
@@ -0,0 +1,96 @@
1
+ // lib/traffic-smoother.js — leaky bucket micro-queue traffic smoother (#496).
2
+ // Absorbs momentary request bursts without tripping upstream per-second/per-minute rate limits.
3
+
4
+ export class TrafficSmoother {
5
+ constructor({ maxQueueWaitMs = 800, now = Date.now, sleep = null } = {}) {
6
+ this.maxQueueWaitMs = maxQueueWaitMs;
7
+ this.now = now;
8
+ this.sleep = sleep || ((ms, signal) => new Promise((resolve, reject) => {
9
+ if (signal?.aborted) return reject(signal.reason ?? new Error('aborted'));
10
+ const timer = setTimeout(resolve, ms);
11
+ if (signal) {
12
+ signal.addEventListener('abort', () => {
13
+ clearTimeout(timer);
14
+ reject(signal.reason ?? new Error('aborted'));
15
+ }, { once: true });
16
+ }
17
+ }));
18
+ // Map<ref, { lastTime: number, tokens: number, lastUpdate: number }>
19
+ this._buckets = new Map();
20
+ }
21
+
22
+ /**
23
+ * Acquire a slot for `ref` given effective RPM or burst limits.
24
+ * @param {string} ref
25
+ * @param {object} opts { rpmLimit, burstLimit, maxQueueWaitMs, signal }
26
+ * @returns {Promise<{ allowed: boolean, delayed: boolean, delayMs: number, overflow: boolean }>}
27
+ */
28
+ async acquire(ref, { rpmLimit, burstLimit, maxQueueWaitMs, signal } = {}) {
29
+ if (!ref) {
30
+ return { allowed: true, delayed: false, delayMs: 0, overflow: false };
31
+ }
32
+ const maxWait = typeof maxQueueWaitMs === 'number' && maxQueueWaitMs >= 0 ? maxQueueWaitMs : this.maxQueueWaitMs;
33
+ const effectiveRpm = Number(rpmLimit) > 0 ? Number(rpmLimit) : 0;
34
+ if (effectiveRpm <= 0) {
35
+ return { allowed: true, delayed: false, delayMs: 0, overflow: false };
36
+ }
37
+
38
+ const capacity = Number(burstLimit) > 0
39
+ ? Math.floor(burstLimit)
40
+ : Math.max(2, Math.ceil(effectiveRpm / 12));
41
+
42
+ const ratePerSec = effectiveRpm / 60; // tokens per second
43
+ const intervalMs = Math.max(1, 1000 / ratePerSec); // ms between tokens
44
+ const curNow = this.now();
45
+
46
+ let bucket = this._buckets.get(ref);
47
+ if (!bucket) {
48
+ bucket = { tokens: capacity, lastUpdate: curNow, nextSlot: curNow };
49
+ this._buckets.set(ref, bucket);
50
+ }
51
+
52
+ // Replenish tokens based on elapsed time
53
+ const elapsed = Math.max(0, curNow - bucket.lastUpdate);
54
+ const addedTokens = elapsed * (ratePerSec / 1000);
55
+ bucket.tokens = Math.min(capacity, bucket.tokens + addedTokens);
56
+ bucket.lastUpdate = curNow;
57
+
58
+ if (bucket.tokens >= 1) {
59
+ bucket.tokens -= 1;
60
+ bucket.nextSlot = Math.max(curNow, bucket.nextSlot);
61
+ return { allowed: true, delayed: false, delayMs: 0, overflow: false };
62
+ }
63
+
64
+ // Burst is exhausted: calculate delay required for next available slot
65
+ const targetSlot = Math.max(curNow, bucket.nextSlot) + intervalMs;
66
+ const delayMs = Math.max(0, targetSlot - curNow);
67
+
68
+ if (delayMs > maxWait) {
69
+ // Exceeds max micro-queue wait: signal overflow so caller rotates to next candidate
70
+ return { allowed: false, delayed: false, delayMs, overflow: true };
71
+ }
72
+
73
+ bucket.nextSlot = targetSlot;
74
+
75
+ // Wait for the scheduled slot
76
+ await this.sleep(delayMs, signal);
77
+
78
+ return { allowed: true, delayed: true, delayMs, overflow: false };
79
+ }
80
+
81
+ getQueueDepth(ref) {
82
+ if (!ref) return this._buckets.size;
83
+ const b = this._buckets.get(ref);
84
+ if (!b) return 0;
85
+ const curNow = this.now();
86
+ const wait = Math.max(0, b.nextSlot - curNow);
87
+ return wait > 0 ? Math.ceil(wait / 100) : 0;
88
+ }
89
+
90
+ clear(ref) {
91
+ if (ref) this._buckets.delete(ref);
92
+ else this._buckets.clear();
93
+ }
94
+ }
95
+
96
+ export default TrafficSmoother;
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@goodandready/dsh-key-rotation",
3
- "version": "0.8.63",
3
+ "version": "0.8.65",
4
4
  "packageManager": "pnpm@10.33.2",
5
5
  "description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
6
6
  "keywords": [