@goodandready/dsh-key-rotation 0.8.64 → 0.8.65
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/README.md +4 -0
- package/README.ru.md +4 -0
- package/README.zh.md +4 -0
- package/lib/client.js +51 -0
- package/lib/index.js +4 -1
- package/lib/ops-keys.js +38 -0
- package/lib/ops-paths.js +1 -0
- package/lib/ops-status.js +2 -0
- package/lib/pool-builder.js +12 -0
- package/lib/pool-state.js +1 -1
- package/lib/provider-limits-detector.js +173 -0
- package/lib/rotate.js +30 -0
- package/lib/traffic-smoother.js +96 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.8.65 - 2026-10-18
|
|
4
|
+
|
|
5
|
+
### Features
|
|
6
|
+
- **Traffic Smoothing Micro-Queue (#496)**:
|
|
7
|
+
- Leaky bucket traffic smoother (`lib/traffic-smoother.js`) absorbs short micro-bursts ($\le 1$s) across multi-agent workflows without tripping upstream per-second or per-minute rate limits or triggering cooldown churning.
|
|
8
|
+
- Configurable `trafficSmoothingBurst` (default 5 or `ceil(rpmLimit/12)`) and `maxQueueWaitMs` (default 800ms).
|
|
9
|
+
- If scheduled micro-queue delay exceeds `maxQueueWaitMs`, the smoother flags burst overflow, immediately rotating to the next candidate key in the pool.
|
|
10
|
+
- Live metric `smoothedRequests` exposed on provider status, with `TRAFFIC_SMOOTHED` audit events.
|
|
11
|
+
- **Accurate Dynamic Provider Limits & Canonical Presets (#503)**:
|
|
12
|
+
- Header limits detector (`lib/provider-limits-detector.js`): extracts dynamic rate limit headers (`x-ratelimit-limit-*`, `x-ratelimit-remaining-*`, `x-ratelimit-reset-*`, `anthropic-ratelimit-*`, `retry-after`) from upstream provider HTTP responses into `pool.state.detectedLimits`.
|
|
13
|
+
- Canonical provider presets catalog for DeepSeek, Groq, OpenCode, Anthropic, OpenAI, SiliconFlow, Mistral, Gemini, and OpenRouter with standard RPM, TPM, burst capacity, and calendar reset window specifications.
|
|
14
|
+
- Endpoint `GET/POST /dsh-key-rotation/preset-limits` to query and auto-apply canonical limits to configured providers.
|
|
15
|
+
- WebUI button `⚡ Preset Limits` for 1-click rate limit configuration and live upstream limits indicator badge (`⚡ 60 RPM · 100k TPM`).
|
|
16
|
+
|
|
3
17
|
## 0.8.64 - 2026-10-18
|
|
4
18
|
|
|
5
19
|
### Features
|
package/README.md
CHANGED
|
@@ -67,6 +67,10 @@
|
|
|
67
67
|
- **📡 Status extras**: per-provider `circuit` plus `meta.expectedClones` / `meta.notifyQueue`.
|
|
68
68
|
- **🧪 Smoke harness**: scripted 429 → next-key → success path in `test/smoke-rotation-080.test.mjs`.
|
|
69
69
|
|
|
70
|
+
### 🚀 What's New in v0.8.65
|
|
71
|
+
- **⚡ Leaky Bucket Traffic Smoothing (#496)**: Micro-queue smoothing absorbs rapid multi-agent burst spikes ($\le 1$s) without triggering unnecessary 429 errors or cooldown churning, with zero-delay burst capacity and early failover rotation on queue overflow.
|
|
72
|
+
- **📊 Dynamic Upstream Rate Limits & Presets (#503)**: Dynamic parsing of provider rate limit headers (`x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after`), live badge display on keys, and 1-click `⚡ Preset Limits` button applying canonical RPM/TPM presets for DeepSeek, Groq, Anthropic, OpenAI, Gemini, and more.
|
|
73
|
+
|
|
70
74
|
### 🛠️ What's New in v0.7.33 (Stability & Bugfix Release)
|
|
71
75
|
- **🔍 Resolved Key Probing BaseURL**: Fixed `resolveBaseUrl` to map key credential refs to owning provider pools, restoring live `probeModels` testing.
|
|
72
76
|
- **🛡️ Guarded Cascade Recursion**: Prevented call stack overflow in cross-provider failover when circular cascade chains occur.
|
package/README.ru.md
CHANGED
|
@@ -36,6 +36,10 @@
|
|
|
36
36
|
|
|
37
37
|
---
|
|
38
38
|
|
|
39
|
+
### 🚀 Что нового в v0.8.65
|
|
40
|
+
- **⚡ Сглаживание трафика (Traffic Smoothing Micro-Queue) (#496)**: Микро-очередь leaky bucket сглаживает кратковременные пики нагрузки (<=1с) от мультиагентных вызовов без лишних ошибок 429 и лишних кулдаунов. При превышении таймаута ожидания микро-очереди запрос автоматически переключается на следующий ключ в пуле.
|
|
41
|
+
- **📊 Динамическое определение лимитов провайдеров и пресеты (#503)**: Автоматический парсинг заголовков `x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after` с отображением актуальных лимитов прямо на карточке ключа, а также кнопка в 1 клик `⚡ Preset Limits` с каноническими пресетами для DeepSeek, Groq, Anthropic, OpenAI, Gemini и др.
|
|
42
|
+
|
|
39
43
|
### 🚀 Что нового в v0.8.11 (обновление в один клик и quality gate)
|
|
40
44
|
|
|
41
45
|
- **Обновление плагина в карточке настроек**: текущая/последняя версия и кнопка установки без выхода из DSH (#307).
|
package/README.zh.md
CHANGED
|
@@ -38,6 +38,10 @@
|
|
|
38
38
|
|
|
39
39
|
## ⚡ 概述与核心痛点
|
|
40
40
|
|
|
41
|
+
### 🚀 v0.8.65 新特性
|
|
42
|
+
- **⚡ 流量平滑微队列 (Traffic Smoothing Micro-Queue) (#496)**: 漏桶微队列平滑多智能体调用中的毫秒级突发并发,避免触发不必要的429限流与冷却流转。队列等待超限时自动轮转至池中下一个候选密钥。
|
|
43
|
+
- **📊 动态上游限制检测与预设规则 (#503)**: 自动提取上游响应头中的限流数据 (`x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after`),并在前端展示实时配额状态;支持一键应用 DeepSeek、Groq、Anthropic、OpenAI、Gemini 等官方规范预设限制。
|
|
44
|
+
|
|
41
45
|
### 🚀 v0.8.11 新特性(一键更新与质量门禁)
|
|
42
46
|
|
|
43
47
|
- **设置卡片中的插件更新器**:查看当前/最新版本并一键安装(#307)。
|
package/lib/client.js
CHANGED
|
@@ -99,6 +99,11 @@ window.__ModuleLoader__.load({
|
|
|
99
99
|
keyPaused: 'Paused',
|
|
100
100
|
keyRevoked: 'Revoked (401)',
|
|
101
101
|
keyAuthPaused: 'Auth Paused (401/403)',
|
|
102
|
+
presetLimitsBtn: '⚡ Preset Limits',
|
|
103
|
+
presetApplied: 'Applied {prov} preset limits ({rpm} RPM / {tpm} TPM)',
|
|
104
|
+
presetNone: 'No canonical preset for this provider',
|
|
105
|
+
detectedLimitsLabel: 'Upstream: {rpm} RPM · {tpm} TPM',
|
|
106
|
+
trafficSmoothingLabel: 'Traffic Smoothing (Burst)',
|
|
102
107
|
pauseKey: 'Pause key from rotation',
|
|
103
108
|
resumeKey: 'Resume key rotation',
|
|
104
109
|
sparklineTraffic: 'Traffic Trend',
|
|
@@ -301,6 +306,11 @@ window.__ModuleLoader__.load({
|
|
|
301
306
|
keyPaused: '已暂停',
|
|
302
307
|
keyRevoked: '已撤销 (401)',
|
|
303
308
|
keyAuthPaused: '身份验证失败已暂停 (401/403)',
|
|
309
|
+
presetLimitsBtn: '⚡ 预设限制',
|
|
310
|
+
presetApplied: '已应用 {prov} 预设限制 ({rpm} RPM / {tpm} TPM)',
|
|
311
|
+
presetNone: '未找到该提供商预设规则',
|
|
312
|
+
detectedLimitsLabel: '上游限制:{rpm} RPM · {tpm} TPM',
|
|
313
|
+
trafficSmoothingLabel: '流量平滑 (突发并发)',
|
|
304
314
|
pauseKey: '暂停此密钥轮换',
|
|
305
315
|
resumeKey: '恢复此密钥轮换',
|
|
306
316
|
sparklineTraffic: '流量趋势',
|
|
@@ -1556,6 +1566,31 @@ window.__ModuleLoader__.load({
|
|
|
1556
1566
|
};
|
|
1557
1567
|
|
|
1558
1568
|
const [validating, setValidating] = React.useState('');
|
|
1569
|
+
const [presetNotice, setPresetNotice] = React.useState('');
|
|
1570
|
+
const applyPreset = async (pIndex, provName) => {
|
|
1571
|
+
try {
|
|
1572
|
+
const r = await fetch('/dsh-key-rotation/preset-limits?provider=' + encodeURIComponent(provName));
|
|
1573
|
+
const data = await r.json();
|
|
1574
|
+
if (data && data.preset) {
|
|
1575
|
+
setField((cur) => {
|
|
1576
|
+
const providers = [...(cur.providers ?? [])];
|
|
1577
|
+
const ent = { ...providers[pIndex] };
|
|
1578
|
+
if (data.preset.rpmLimit) ent.rpmLimit = data.preset.rpmLimit;
|
|
1579
|
+
if (data.preset.tpmLimit) ent.tpmLimit = data.preset.tpmLimit;
|
|
1580
|
+
providers[pIndex] = ent;
|
|
1581
|
+
return { ...cur, providers };
|
|
1582
|
+
});
|
|
1583
|
+
setPresetNotice(t('presetApplied').replace('{prov}', provName).replace('{rpm}', String(data.preset.rpmLimit)).replace('{tpm}', String(data.preset.tpmLimit)));
|
|
1584
|
+
setTimeout(() => setPresetNotice(''), 4000);
|
|
1585
|
+
} else {
|
|
1586
|
+
setPresetNotice(t('presetNone'));
|
|
1587
|
+
setTimeout(() => setPresetNotice(''), 3000);
|
|
1588
|
+
}
|
|
1589
|
+
} catch (err) {
|
|
1590
|
+
setPresetNotice(String(err?.message || err));
|
|
1591
|
+
setTimeout(() => setPresetNotice(''), 3000);
|
|
1592
|
+
}
|
|
1593
|
+
};
|
|
1559
1594
|
const [validationResult, setValidationResult] = React.useState({});
|
|
1560
1595
|
const validateBeforeSave = (ref, value) => {
|
|
1561
1596
|
setValidating(ref);
|
|
@@ -2127,6 +2162,20 @@ window.__ModuleLoader__.load({
|
|
|
2127
2162
|
if (info && info.broken) {
|
|
2128
2163
|
meta.push(h('button', { key: 'rt', className: 'krot-btn', onClick: () => retestBroken(key), title: t('retestBroken') }, t('retestBroken')));
|
|
2129
2164
|
}
|
|
2165
|
+
if (info && info.detectedLimits) {
|
|
2166
|
+
const dl = info.detectedLimits;
|
|
2167
|
+
const parts = [];
|
|
2168
|
+
if (dl.rpmLimit) parts.push(dl.rpmLimit + ' RPM');
|
|
2169
|
+
if (dl.tpmLimit) parts.push(dl.tpmLimit + ' TPM');
|
|
2170
|
+
if (parts.length > 0) {
|
|
2171
|
+
meta.push(h('span', {
|
|
2172
|
+
key: 'dl',
|
|
2173
|
+
className: 'krot-tail',
|
|
2174
|
+
title: 'Live upstream limits:\n' + JSON.stringify(dl, null, 2),
|
|
2175
|
+
style: { fontSize: '11px', opacity: 0.8, color: 'var(--dsw-alias-brand-primary)' }
|
|
2176
|
+
}, '⚡ ' + parts.join(' · ')));
|
|
2177
|
+
}
|
|
2178
|
+
}
|
|
2130
2179
|
if (info && info.modelCapabilities && info.modelCapabilities.length > 0) {
|
|
2131
2180
|
meta.push(h('span', {
|
|
2132
2181
|
key: 'caps',
|
|
@@ -2368,6 +2417,8 @@ window.__ModuleLoader__.load({
|
|
|
2368
2417
|
});
|
|
2369
2418
|
}, { title: t('sortUsage') }),
|
|
2370
2419
|
h('select', { className: 'krot-in', value: entry.provider, onChange: (e) => setProvider(pIndex, e.target.value) }, options),
|
|
2420
|
+
btn(t('presetLimitsBtn') || '⚡ Preset Limits', () => applyPreset(pIndex, entry.provider), { title: t('presetLimitsBtn') || '⚡ Preset Limits', disabled: !entry.provider.trim() || !stateRef.current.writable }),
|
|
2421
|
+
presetNotice ? h('span', { className: 'krot-badge krot-badge-ok', style: { marginLeft: '4px' } }, presetNotice) : null,
|
|
2371
2422
|
(() => {
|
|
2372
2423
|
const ps = status[entry.provider];
|
|
2373
2424
|
const score = ps && typeof ps.healthScore === 'number' ? ps.healthScore : null;
|
package/lib/index.js
CHANGED
|
@@ -33,6 +33,7 @@ import { registerOpsRoutes } from './routes-ops.js';
|
|
|
33
33
|
import { registerPluginUpdater } from './plugin-updater.js';
|
|
34
34
|
import { notifySwitch, notifyExhaustion, notifyEarlyWarning, pushEvent } from './notify-events.js';
|
|
35
35
|
import { QUOTA_WINDOW_TYPES } from './quota-window.js';
|
|
36
|
+
import { TrafficSmoother } from './traffic-smoother.js';
|
|
36
37
|
import { getLogger } from './logger.js';
|
|
37
38
|
import { createSandboxService } from './sandbox-service.js';
|
|
38
39
|
import { checkBudgetAndHealthAlerts } from './budget-monitor.js';
|
|
@@ -198,6 +199,7 @@ export function apply(ctx, config = {}) {
|
|
|
198
199
|
const webhookSender = new WebhookSender({ fetchImpl: globalThis.fetch });
|
|
199
200
|
const alertDebouncer = new AlertDebouncer({ sender: webhookSender, debounceMs: 2000 });
|
|
200
201
|
const moduleNotifyQueue = new NotifyQueue({ send: (url, payload) => alertDebouncer.enqueue(url, payload) });
|
|
202
|
+
const trafficSmoother = new TrafficSmoother({ maxQueueWaitMs: 800 });
|
|
201
203
|
|
|
202
204
|
const sandboxService = createSandboxService({ getRuntime: () => buildRuntime() });
|
|
203
205
|
const { ensureSandboxRunner, probeRef, lastTestCache } = sandboxService;
|
|
@@ -355,7 +357,7 @@ export function apply(ctx, config = {}) {
|
|
|
355
357
|
switchNotifyThrottleMs: cfg.switchNotifyThrottleMs ?? 60000,
|
|
356
358
|
warnBelowHealthy: cfg.warnBelowHealthy ?? 0, latencySloMs: cfg.latencySloMs ?? 0,
|
|
357
359
|
index, pools,
|
|
358
|
-
breaker: moduleBreaker, notifyQueue: moduleNotifyQueue, concurrencyTracker, quotaStore, latencyHistogram,
|
|
360
|
+
breaker: moduleBreaker, notifyQueue: moduleNotifyQueue, concurrencyTracker, quotaStore, latencyHistogram, trafficSmoother,
|
|
359
361
|
providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider,
|
|
360
362
|
cloneIds, expectedClones,
|
|
361
363
|
circuitBreakerEnabled: cfg.circuitBreakerEnabled ?? true,
|
|
@@ -434,6 +436,7 @@ export function apply(ctx, config = {}) {
|
|
|
434
436
|
MARKER,
|
|
435
437
|
finishError,
|
|
436
438
|
setRotateStartMs: (v) => { _rotateStartMs = v; },
|
|
439
|
+
trafficSmoother,
|
|
437
440
|
quotaStore,
|
|
438
441
|
circuitBreaker: moduleBreaker,
|
|
439
442
|
now: nowMono,
|
package/lib/ops-keys.js
CHANGED
|
@@ -22,7 +22,9 @@ import {
|
|
|
22
22
|
DRAIN_PATH,
|
|
23
23
|
DISCOVER_PATH,
|
|
24
24
|
AUTO_IMPORT_PATH,
|
|
25
|
+
PRESET_LIMITS_PATH,
|
|
25
26
|
} from './ops-paths.js';
|
|
27
|
+
import { lookupPreset, CANONICAL_PROVIDER_PRESETS } from './provider-limits-detector.js';
|
|
26
28
|
import fs from 'node:fs';
|
|
27
29
|
import os from 'node:os';
|
|
28
30
|
import path from 'node:path';
|
|
@@ -539,4 +541,40 @@ ctx.effect(() => ctx.webServer.register({
|
|
|
539
541
|
},
|
|
540
542
|
}), 'dsh-key-rotation: auto-import route');
|
|
541
543
|
|
|
544
|
+
ctx.effect(() => ctx.webServer.register({
|
|
545
|
+
kind: 'exact',
|
|
546
|
+
path: PRESET_LIMITS_PATH,
|
|
547
|
+
handler: async (req, res) => {
|
|
548
|
+
if (!isTrustedBridgeRequest(req)) {
|
|
549
|
+
res.writeHead(403, { 'Content-Type': 'application/json' });
|
|
550
|
+
res.end(JSON.stringify({ ok: false, error: 'Forbidden' }));
|
|
551
|
+
return;
|
|
552
|
+
}
|
|
553
|
+
if (req.method === 'GET') {
|
|
554
|
+
const urlObj = new URL(req.url, 'http://127.0.0.1');
|
|
555
|
+
const prov = urlObj.searchParams.get('provider');
|
|
556
|
+
if (prov) {
|
|
557
|
+
const preset = lookupPreset(prov);
|
|
558
|
+
json(res, 200, { ok: true, provider: prov, preset });
|
|
559
|
+
return;
|
|
560
|
+
}
|
|
561
|
+
json(res, 200, { ok: true, presets: CANONICAL_PROVIDER_PRESETS });
|
|
562
|
+
return;
|
|
563
|
+
}
|
|
564
|
+
if (req.method === 'POST') {
|
|
565
|
+
const body = await readJson(req);
|
|
566
|
+
const prov = body?.provider;
|
|
567
|
+
const preset = lookupPreset(prov);
|
|
568
|
+
if (!preset) {
|
|
569
|
+
json(res, 404, { ok: false, error: `No canonical preset found for provider '${prov}'` });
|
|
570
|
+
return;
|
|
571
|
+
}
|
|
572
|
+
json(res, 200, { ok: true, provider: prov, preset });
|
|
573
|
+
return;
|
|
574
|
+
}
|
|
575
|
+
res.writeHead(405, { 'Content-Type': 'application/json' });
|
|
576
|
+
res.end(JSON.stringify({ ok: false, error: 'Method Not Allowed' }));
|
|
577
|
+
},
|
|
578
|
+
}), 'dsh-key-rotation: preset limits route');
|
|
579
|
+
|
|
542
580
|
}
|
package/lib/ops-paths.js
CHANGED
|
@@ -17,3 +17,4 @@ export const DRAIN_PATH = '/dsh-key-rotation/drain';
|
|
|
17
17
|
export const TEST_MATRIX_PATH = '/dsh-key-rotation/test-matrix';
|
|
18
18
|
export const DISCOVER_PATH = '/dsh-key-rotation/discover';
|
|
19
19
|
export const AUTO_IMPORT_PATH = '/dsh-key-rotation/auto-import';
|
|
20
|
+
export const PRESET_LIMITS_PATH = '/dsh-key-rotation/preset-limits';
|
package/lib/ops-status.js
CHANGED
|
@@ -123,6 +123,7 @@ ctx.effect(() => ctx.webServer.register({
|
|
|
123
123
|
pauseReason: pool.state.pauseReasons?.get?.(ref) ?? null,
|
|
124
124
|
revoked: isKeyRevoked(pool, ref),
|
|
125
125
|
modelCapabilities: pool.state.modelCapabilities?.get?.(ref) ? [...pool.state.modelCapabilities.get(ref)] : null,
|
|
126
|
+
detectedLimits: pool.state.detectedLimits?.get?.(ref) ?? null,
|
|
126
127
|
modelQuota,
|
|
127
128
|
modelPools: activeModelPools.map(mp => mp.base),
|
|
128
129
|
});
|
|
@@ -172,6 +173,7 @@ ctx.effect(() => ctx.webServer.register({
|
|
|
172
173
|
pauseOnBudget: (providerBudgets?.get ? providerBudgets.get(pool.base) : providerBudgets?.[pool.base])?.pauseOnBudget ?? false,
|
|
173
174
|
routingStrategy: pool.routingStrategy ?? runtime.routingStrategy ?? 'round-robin',
|
|
174
175
|
proactiveRateLimitGuard: pool.proactiveRateLimitGuard ?? runtime.proactiveRateLimitGuard ?? true,
|
|
176
|
+
smoothedRequests: pool.state.smoothedRequests ?? 0,
|
|
175
177
|
});
|
|
176
178
|
} catch (e) {
|
|
177
179
|
(ctx?.logger ? ctx.logger('dsh-key-rotation') : null)?.warn?.(`[dsh-key-rotation] status: pool ${pool.base} failed: ${String(e?.message ?? e)} ${e?.stack ?? ''}`);
|
package/lib/pool-builder.js
CHANGED
|
@@ -37,6 +37,8 @@ export function buildPoolItem({
|
|
|
37
37
|
labels = null,
|
|
38
38
|
pinnedRef = null,
|
|
39
39
|
useKeyRef = null,
|
|
40
|
+
trafficSmoothingBurst = null,
|
|
41
|
+
maxQueueWaitMs = null,
|
|
40
42
|
}) {
|
|
41
43
|
const seen = new Set();
|
|
42
44
|
const indexed = (Array.isArray(keys) ? keys : []).map((key, index) => ({ ref: typeof key === 'string' ? key.trim() : '', index }))
|
|
@@ -135,6 +137,12 @@ const weightsMap = {};
|
|
|
135
137
|
proactiveRateLimitGuard: poolGuard,
|
|
136
138
|
labels: labelsMap,
|
|
137
139
|
pinnedRef: cleanPinned,
|
|
140
|
+
trafficSmoothingBurst: typeof trafficSmoothingBurst === 'number' && Number.isFinite(trafficSmoothingBurst) && trafficSmoothingBurst > 0
|
|
141
|
+
? Math.floor(trafficSmoothingBurst)
|
|
142
|
+
: (typeof rpmLimit === 'number' && rpmLimit > 0 ? Math.max(2, Math.ceil(rpmLimit / 12)) : 5),
|
|
143
|
+
maxQueueWaitMs: typeof maxQueueWaitMs === 'number' && Number.isFinite(maxQueueWaitMs) && maxQueueWaitMs >= 0
|
|
144
|
+
? Math.floor(maxQueueWaitMs)
|
|
145
|
+
: 800,
|
|
138
146
|
};
|
|
139
147
|
}
|
|
140
148
|
|
|
@@ -175,6 +183,8 @@ export function buildPools({ cfg = {}, poolState = new Map(), defaultState = ini
|
|
|
175
183
|
model: null,
|
|
176
184
|
labels: p.labels,
|
|
177
185
|
pinnedRef: p.pinnedRef ?? p.useKeyRef,
|
|
186
|
+
trafficSmoothingBurst: p.trafficSmoothingBurst ?? cfg.trafficSmoothingBurst,
|
|
187
|
+
maxQueueWaitMs: p.maxQueueWaitMs ?? cfg.maxQueueWaitMs,
|
|
178
188
|
}, provRpm, provTpm, provConcurrency);
|
|
179
189
|
if (pool) {
|
|
180
190
|
for (const ref of pool.refs) {
|
|
@@ -203,6 +213,8 @@ export function buildPools({ cfg = {}, poolState = new Map(), defaultState = ini
|
|
|
203
213
|
quotas: mp.quotas,
|
|
204
214
|
labels: mp.labels ?? p.labels,
|
|
205
215
|
pinnedRef: mp.pinnedRef ?? mp.useKeyRef ?? p.pinnedRef ?? p.useKeyRef,
|
|
216
|
+
trafficSmoothingBurst: mp.trafficSmoothingBurst ?? p.trafficSmoothingBurst ?? cfg.trafficSmoothingBurst,
|
|
217
|
+
maxQueueWaitMs: mp.maxQueueWaitMs ?? p.maxQueueWaitMs ?? cfg.maxQueueWaitMs,
|
|
206
218
|
}, modelRpm, modelTpm, modelConcurrency);
|
|
207
219
|
if (!mpool) continue;
|
|
208
220
|
mpool.basePool = pool;
|
package/lib/pool-state.js
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
const MAP_FIELDS = [
|
|
4
4
|
'failedUntil', 'failCounts', 'authFailCounts', 'brokenUntil', 'costPerKey',
|
|
5
5
|
'lastUsedAt', 'usageCounts', 'byModel', 'usageDays', 'quotaWindows', 'costDays',
|
|
6
|
-
'tokenUsage', 'rpmWindows', 'tpmWindows', 'sessionSticky', 'modelCapabilities', 'pauseReasons',
|
|
6
|
+
'tokenUsage', 'rpmWindows', 'tpmWindows', 'sessionSticky', 'modelCapabilities', 'pauseReasons', 'detectedLimits',
|
|
7
7
|
];
|
|
8
8
|
|
|
9
9
|
const SET_FIELDS = ['pausedRefs', 'revokedRefs'];
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
// lib/provider-limits-detector.js — accurate dynamic provider limits and presets (#503).
|
|
2
|
+
|
|
3
|
+
export const CANONICAL_PROVIDER_PRESETS = {
|
|
4
|
+
deepseek: {
|
|
5
|
+
rpmLimit: 60,
|
|
6
|
+
tpmLimit: 100000,
|
|
7
|
+
trafficSmoothingBurst: 5,
|
|
8
|
+
quotaResetWindow: 'midnight_utc',
|
|
9
|
+
displayName: 'DeepSeek Official',
|
|
10
|
+
},
|
|
11
|
+
groq: {
|
|
12
|
+
rpmLimit: 30,
|
|
13
|
+
tpmLimit: 30000,
|
|
14
|
+
trafficSmoothingBurst: 5,
|
|
15
|
+
quotaResetWindow: 'per-minute',
|
|
16
|
+
displayName: 'Groq Cloud',
|
|
17
|
+
},
|
|
18
|
+
opencode: {
|
|
19
|
+
rpmLimit: 120,
|
|
20
|
+
tpmLimit: 200000,
|
|
21
|
+
trafficSmoothingBurst: 10,
|
|
22
|
+
quotaResetWindow: 'midnight_utc',
|
|
23
|
+
displayName: 'OpenCode Zen',
|
|
24
|
+
},
|
|
25
|
+
anthropic: {
|
|
26
|
+
rpmLimit: 50,
|
|
27
|
+
tpmLimit: 100000,
|
|
28
|
+
trafficSmoothingBurst: 5,
|
|
29
|
+
quotaResetWindow: 'per-minute',
|
|
30
|
+
displayName: 'Anthropic Claude',
|
|
31
|
+
},
|
|
32
|
+
openai: {
|
|
33
|
+
rpmLimit: 60,
|
|
34
|
+
tpmLimit: 150000,
|
|
35
|
+
trafficSmoothingBurst: 10,
|
|
36
|
+
quotaResetWindow: 'per-minute',
|
|
37
|
+
displayName: 'OpenAI API',
|
|
38
|
+
},
|
|
39
|
+
siliconflow: {
|
|
40
|
+
rpmLimit: 60,
|
|
41
|
+
tpmLimit: 100000,
|
|
42
|
+
trafficSmoothingBurst: 5,
|
|
43
|
+
quotaResetWindow: 'per-minute',
|
|
44
|
+
displayName: 'SiliconFlow',
|
|
45
|
+
},
|
|
46
|
+
mistral: {
|
|
47
|
+
rpmLimit: 60,
|
|
48
|
+
tpmLimit: 100000,
|
|
49
|
+
trafficSmoothingBurst: 5,
|
|
50
|
+
quotaResetWindow: 'per-minute',
|
|
51
|
+
displayName: 'Mistral AI',
|
|
52
|
+
},
|
|
53
|
+
gemini: {
|
|
54
|
+
rpmLimit: 60,
|
|
55
|
+
tpmLimit: 1000000,
|
|
56
|
+
trafficSmoothingBurst: 10,
|
|
57
|
+
quotaResetWindow: 'midnight_pst',
|
|
58
|
+
displayName: 'Google Gemini',
|
|
59
|
+
},
|
|
60
|
+
openrouter: {
|
|
61
|
+
rpmLimit: 200,
|
|
62
|
+
tpmLimit: 500000,
|
|
63
|
+
trafficSmoothingBurst: 15,
|
|
64
|
+
quotaResetWindow: 'per-minute',
|
|
65
|
+
displayName: 'OpenRouter',
|
|
66
|
+
},
|
|
67
|
+
};
|
|
68
|
+
|
|
69
|
+
/**
|
|
70
|
+
* Extract rate limit and quota information from HTTP response headers.
|
|
71
|
+
* @param {Headers|Record<string, string>} rawHeaders
|
|
72
|
+
* @returns {object|null}
|
|
73
|
+
*/
|
|
74
|
+
export function extractProviderLimits(rawHeaders) {
|
|
75
|
+
if (!rawHeaders) return null;
|
|
76
|
+
const headers = {};
|
|
77
|
+
if (typeof rawHeaders.forEach === 'function') {
|
|
78
|
+
rawHeaders.forEach((val, key) => { headers[key.toLowerCase()] = String(val); });
|
|
79
|
+
} else if (typeof rawHeaders === 'object') {
|
|
80
|
+
for (const [k, v] of Object.entries(rawHeaders)) {
|
|
81
|
+
if (typeof k === 'string') headers[k.toLowerCase()] = String(v ?? '');
|
|
82
|
+
}
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
const getNum = (...names) => {
|
|
86
|
+
for (const name of names) {
|
|
87
|
+
const val = headers[name];
|
|
88
|
+
if (val !== undefined && val !== null && val !== '') {
|
|
89
|
+
const n = Number(val);
|
|
90
|
+
if (Number.isFinite(n) && n >= 0) return n;
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
return undefined;
|
|
94
|
+
};
|
|
95
|
+
|
|
96
|
+
const getSeconds = (...names) => {
|
|
97
|
+
for (const name of names) {
|
|
98
|
+
const val = headers[name];
|
|
99
|
+
if (!val) continue;
|
|
100
|
+
const str = String(val).trim();
|
|
101
|
+
// Duration string format "6m0s", "1s", "250ms"
|
|
102
|
+
const match = /^(\d+(?:\.\d+)?)(s|ms|m|h)?$/i.exec(str);
|
|
103
|
+
if (match) {
|
|
104
|
+
const num = parseFloat(match[1]);
|
|
105
|
+
const unit = (match[2] || 's').toLowerCase();
|
|
106
|
+
if (unit === 'ms') return Math.max(0.001, num / 1000);
|
|
107
|
+
if (unit === 's') return num;
|
|
108
|
+
if (unit === 'm') return num * 60;
|
|
109
|
+
if (unit === 'h') return num * 3600;
|
|
110
|
+
}
|
|
111
|
+
// Check for ISO Date string (e.g., Anthropic reset timestamp)
|
|
112
|
+
const parsedDate = Date.parse(str);
|
|
113
|
+
if (Number.isFinite(parsedDate)) {
|
|
114
|
+
const diffSec = (parsedDate - Date.now()) / 1000;
|
|
115
|
+
return Math.max(0, diffSec);
|
|
116
|
+
}
|
|
117
|
+
const n = Number(str);
|
|
118
|
+
if (Number.isFinite(n) && n >= 0) return n;
|
|
119
|
+
}
|
|
120
|
+
return undefined;
|
|
121
|
+
};
|
|
122
|
+
|
|
123
|
+
const rpmLimit = getNum('x-ratelimit-limit-requests', 'x-ratelimit-limit-rpm', 'ratelimit-limit', 'anthropic-ratelimit-requests-limit');
|
|
124
|
+
const tpmLimit = getNum('x-ratelimit-limit-tokens', 'x-ratelimit-limit-tpm', 'anthropic-ratelimit-tokens-limit');
|
|
125
|
+
const remainingRequests = getNum('x-ratelimit-remaining-requests', 'x-ratelimit-remaining', 'anthropic-ratelimit-requests-remaining');
|
|
126
|
+
const remainingTokens = getNum('x-ratelimit-remaining-tokens', 'anthropic-ratelimit-tokens-remaining');
|
|
127
|
+
const resetRequests = getSeconds('x-ratelimit-reset-requests', 'x-ratelimit-reset', 'anthropic-ratelimit-requests-reset');
|
|
128
|
+
const resetTokens = getSeconds('x-ratelimit-reset-tokens', 'anthropic-ratelimit-tokens-reset');
|
|
129
|
+
const retryAfter = getSeconds('retry-after');
|
|
130
|
+
|
|
131
|
+
if (rpmLimit === undefined && tpmLimit === undefined && remainingRequests === undefined && remainingTokens === undefined && resetRequests === undefined && retryAfter === undefined) {
|
|
132
|
+
return null;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
const out = {};
|
|
136
|
+
if (rpmLimit !== undefined) out.rpmLimit = Math.floor(rpmLimit);
|
|
137
|
+
if (tpmLimit !== undefined) out.tpmLimit = Math.floor(tpmLimit);
|
|
138
|
+
if (remainingRequests !== undefined) {
|
|
139
|
+
out.remainingRequests = Math.floor(remainingRequests);
|
|
140
|
+
out.rpmRemaining = out.remainingRequests;
|
|
141
|
+
}
|
|
142
|
+
if (remainingTokens !== undefined) {
|
|
143
|
+
out.remainingTokens = Math.floor(remainingTokens);
|
|
144
|
+
out.tpmRemaining = out.remainingTokens;
|
|
145
|
+
}
|
|
146
|
+
if (resetRequests !== undefined) {
|
|
147
|
+
out.resetRequests = resetRequests;
|
|
148
|
+
out.rpmResetMs = Math.round(resetRequests * 1000);
|
|
149
|
+
}
|
|
150
|
+
if (resetTokens !== undefined) {
|
|
151
|
+
out.resetTokens = resetTokens;
|
|
152
|
+
out.tpmResetMs = Math.round(resetTokens * 1000);
|
|
153
|
+
}
|
|
154
|
+
if (retryAfter !== undefined) {
|
|
155
|
+
out.retryAfterMs = Math.round(retryAfter * 1000);
|
|
156
|
+
}
|
|
157
|
+
|
|
158
|
+
return out;
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/**
|
|
162
|
+
* Match a configured provider name to canonical preset.
|
|
163
|
+
* @param {string} providerName
|
|
164
|
+
* @returns {object|null}
|
|
165
|
+
*/
|
|
166
|
+
export function lookupPreset(providerName) {
|
|
167
|
+
if (!providerName || typeof providerName !== 'string') return null;
|
|
168
|
+
const p = providerName.toLowerCase().replace(/[^a-z0-9]/g, '');
|
|
169
|
+
for (const [key, val] of Object.entries(CANONICAL_PROVIDER_PRESETS)) {
|
|
170
|
+
if (p.includes(key)) return { ...val, id: key };
|
|
171
|
+
}
|
|
172
|
+
return null;
|
|
173
|
+
}
|
package/lib/rotate.js
CHANGED
|
@@ -35,6 +35,7 @@ import {
|
|
|
35
35
|
} from './model-quota.js';
|
|
36
36
|
|
|
37
37
|
import { LOCAL_POOL_EXHAUSTED_CODE } from './resolver.js';
|
|
38
|
+
import { extractProviderLimits } from './provider-limits-detector.js';
|
|
38
39
|
|
|
39
40
|
export const LOCAL_QUOTA_CODE = 'LOCAL_MODEL_QUOTA_EXHAUSTED';
|
|
40
41
|
|
|
@@ -79,6 +80,7 @@ export function createRotate(deps) {
|
|
|
79
80
|
circuitBreaker,
|
|
80
81
|
logger,
|
|
81
82
|
schedulePersist,
|
|
83
|
+
trafficSmoother,
|
|
82
84
|
now = nowMono,
|
|
83
85
|
} = deps;
|
|
84
86
|
|
|
@@ -318,6 +320,25 @@ export function createRotate(deps) {
|
|
|
318
320
|
};
|
|
319
321
|
|
|
320
322
|
for (let attempt = 0; attempt < attemptList.length; attempt++) {
|
|
323
|
+
const candidateRef = attemptList[attempt];
|
|
324
|
+
// #496: Traffic smoothing micro-queue
|
|
325
|
+
const smoother = pool.trafficSmoother ?? runtime0.trafficSmoother ?? trafficSmoother;
|
|
326
|
+
if (smoother && candidateRef) {
|
|
327
|
+
const smoothRes = await smoother.acquire(candidateRef, {
|
|
328
|
+
rpmLimit: pool.rpmLimit,
|
|
329
|
+
burstLimit: pool.trafficSmoothingBurst,
|
|
330
|
+
maxQueueWaitMs: pool.maxQueueWaitMs,
|
|
331
|
+
signal: options.signal,
|
|
332
|
+
});
|
|
333
|
+
if (smoothRes.overflow) {
|
|
334
|
+
logWarn(`[dsh-key-rotation] ${options.provider}: key ${candidateRef} micro-queue burst overflow (${smoothRes.delayMs}ms) — rotating to next candidate`);
|
|
335
|
+
continue;
|
|
336
|
+
}
|
|
337
|
+
if (smoothRes.delayed) {
|
|
338
|
+
pool.state.smoothedRequests = (pool.state.smoothedRequests ?? 0) + 1;
|
|
339
|
+
pushEvent(pool, candidateRef, 'TRAFFIC_SMOOTHED', smoothRes.delayMs);
|
|
340
|
+
}
|
|
341
|
+
}
|
|
321
342
|
// #464: Check deadline & attempt budget before dispatch
|
|
322
343
|
if (!budget.canAttempt()) {
|
|
323
344
|
const isTimeout = budget.isExpired();
|
|
@@ -506,6 +527,15 @@ export function createRotate(deps) {
|
|
|
506
527
|
// Proactive rate-limit (#115, #303): if response headers say this key is near
|
|
507
528
|
// its quota or has retry-after, cool it down so the NEXT request starts on a different key.
|
|
508
529
|
// We do NOT re-run this (already successful) request — that would double-send.
|
|
530
|
+
// #503: Extract provider limits and reset headers
|
|
531
|
+
const rawHdrs = chunk?.metadata?.headers ?? chunk?.headers;
|
|
532
|
+
if (rawHdrs && activeRef) {
|
|
533
|
+
const detected = extractProviderLimits(rawHdrs);
|
|
534
|
+
if (detected) {
|
|
535
|
+
if (!pool.state.detectedLimits) pool.state.detectedLimits = new Map();
|
|
536
|
+
pool.state.detectedLimits.set(activeRef, { ...detected, at: now() });
|
|
537
|
+
}
|
|
538
|
+
}
|
|
509
539
|
const guardEnabled = pool.proactiveRateLimitGuard ?? runtime0.proactiveRateLimitGuard ?? true;
|
|
510
540
|
const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
|
|
511
541
|
if (guardEnabled && rate && activeRef) {
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
// lib/traffic-smoother.js — leaky bucket micro-queue traffic smoother (#496).
|
|
2
|
+
// Absorbs momentary request bursts without tripping upstream per-second/per-minute rate limits.
|
|
3
|
+
|
|
4
|
+
export class TrafficSmoother {
|
|
5
|
+
constructor({ maxQueueWaitMs = 800, now = Date.now, sleep = null } = {}) {
|
|
6
|
+
this.maxQueueWaitMs = maxQueueWaitMs;
|
|
7
|
+
this.now = now;
|
|
8
|
+
this.sleep = sleep || ((ms, signal) => new Promise((resolve, reject) => {
|
|
9
|
+
if (signal?.aborted) return reject(signal.reason ?? new Error('aborted'));
|
|
10
|
+
const timer = setTimeout(resolve, ms);
|
|
11
|
+
if (signal) {
|
|
12
|
+
signal.addEventListener('abort', () => {
|
|
13
|
+
clearTimeout(timer);
|
|
14
|
+
reject(signal.reason ?? new Error('aborted'));
|
|
15
|
+
}, { once: true });
|
|
16
|
+
}
|
|
17
|
+
}));
|
|
18
|
+
// Map<ref, { lastTime: number, tokens: number, lastUpdate: number }>
|
|
19
|
+
this._buckets = new Map();
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
/**
|
|
23
|
+
* Acquire a slot for `ref` given effective RPM or burst limits.
|
|
24
|
+
* @param {string} ref
|
|
25
|
+
* @param {object} opts { rpmLimit, burstLimit, maxQueueWaitMs, signal }
|
|
26
|
+
* @returns {Promise<{ allowed: boolean, delayed: boolean, delayMs: number, overflow: boolean }>}
|
|
27
|
+
*/
|
|
28
|
+
async acquire(ref, { rpmLimit, burstLimit, maxQueueWaitMs, signal } = {}) {
|
|
29
|
+
if (!ref) {
|
|
30
|
+
return { allowed: true, delayed: false, delayMs: 0, overflow: false };
|
|
31
|
+
}
|
|
32
|
+
const maxWait = typeof maxQueueWaitMs === 'number' && maxQueueWaitMs >= 0 ? maxQueueWaitMs : this.maxQueueWaitMs;
|
|
33
|
+
const effectiveRpm = Number(rpmLimit) > 0 ? Number(rpmLimit) : 0;
|
|
34
|
+
if (effectiveRpm <= 0) {
|
|
35
|
+
return { allowed: true, delayed: false, delayMs: 0, overflow: false };
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
const capacity = Number(burstLimit) > 0
|
|
39
|
+
? Math.floor(burstLimit)
|
|
40
|
+
: Math.max(2, Math.ceil(effectiveRpm / 12));
|
|
41
|
+
|
|
42
|
+
const ratePerSec = effectiveRpm / 60; // tokens per second
|
|
43
|
+
const intervalMs = Math.max(1, 1000 / ratePerSec); // ms between tokens
|
|
44
|
+
const curNow = this.now();
|
|
45
|
+
|
|
46
|
+
let bucket = this._buckets.get(ref);
|
|
47
|
+
if (!bucket) {
|
|
48
|
+
bucket = { tokens: capacity, lastUpdate: curNow, nextSlot: curNow };
|
|
49
|
+
this._buckets.set(ref, bucket);
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
// Replenish tokens based on elapsed time
|
|
53
|
+
const elapsed = Math.max(0, curNow - bucket.lastUpdate);
|
|
54
|
+
const addedTokens = elapsed * (ratePerSec / 1000);
|
|
55
|
+
bucket.tokens = Math.min(capacity, bucket.tokens + addedTokens);
|
|
56
|
+
bucket.lastUpdate = curNow;
|
|
57
|
+
|
|
58
|
+
if (bucket.tokens >= 1) {
|
|
59
|
+
bucket.tokens -= 1;
|
|
60
|
+
bucket.nextSlot = Math.max(curNow, bucket.nextSlot);
|
|
61
|
+
return { allowed: true, delayed: false, delayMs: 0, overflow: false };
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
// Burst is exhausted: calculate delay required for next available slot
|
|
65
|
+
const targetSlot = Math.max(curNow, bucket.nextSlot) + intervalMs;
|
|
66
|
+
const delayMs = Math.max(0, targetSlot - curNow);
|
|
67
|
+
|
|
68
|
+
if (delayMs > maxWait) {
|
|
69
|
+
// Exceeds max micro-queue wait: signal overflow so caller rotates to next candidate
|
|
70
|
+
return { allowed: false, delayed: false, delayMs, overflow: true };
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
bucket.nextSlot = targetSlot;
|
|
74
|
+
|
|
75
|
+
// Wait for the scheduled slot
|
|
76
|
+
await this.sleep(delayMs, signal);
|
|
77
|
+
|
|
78
|
+
return { allowed: true, delayed: true, delayMs, overflow: false };
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
getQueueDepth(ref) {
|
|
82
|
+
if (!ref) return this._buckets.size;
|
|
83
|
+
const b = this._buckets.get(ref);
|
|
84
|
+
if (!b) return 0;
|
|
85
|
+
const curNow = this.now();
|
|
86
|
+
const wait = Math.max(0, b.nextSlot - curNow);
|
|
87
|
+
return wait > 0 ? Math.ceil(wait / 100) : 0;
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
clear(ref) {
|
|
91
|
+
if (ref) this._buckets.delete(ref);
|
|
92
|
+
else this._buckets.clear();
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
export default TrafficSmoother;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@goodandready/dsh-key-rotation",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.65",
|
|
4
4
|
"packageManager": "pnpm@10.33.2",
|
|
5
5
|
"description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
|
|
6
6
|
"keywords": [
|