@goodandready/dsh-key-rotation 0.8.65 → 0.8.66

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.8.66 - 2026-10-19
+
+### Features
+- **Dynamic Reset Scheduler Based on Rate-Limit Headers (#498)**:
+  - Exact cooldown alignment: parses upstream `retry-after`, `x-ratelimit-reset-*`, and `anthropic-ratelimit-*-reset` headers or message duration patterns upon rate-limit failures, setting precise cooldown `failedUntil = now + dynamicCooldownMs` (+250ms safety jitter, bounded by `maxCooldownMs`) with `resetReason = 'header'`.
+  - Immediate unref recovery timer: schedules an exact unref `setTimeout` wake-up for the penalised key right when the upstream window rolls over, rather than waiting up to 30s for the periodic sweep.
+  - Exposes `resetReason` and `headerResetMs` in `/dsh-key-rotation/status` and WebUI countdown badge (`⏳ reset in Xs (header)`).
+- **Upstream Balance Inspector & Credit Tracking (#495)**:
+  - Multi-provider balance checker (`lib/balance-inspector.js`): queries live account balances and remaining credits for DeepSeek (`/user/balance`), SiliconFlow (`/v1/user/info`), OpenRouter (`/api/v1/auth/key`), and OpenAI (`/v1/dashboard/billing/credit_grants`).
+  - Balance ops route `GET/POST /dsh-key-rotation/balance` (`lib/ops-balance.js`): queries cached balances or triggers immediate live balance inspection across provider keys.
+  - Pre-emptive empty balance protection: keys with confirmed zero balance (`isZero: true`) enter a soft pause (`EMPTY_BALANCE_PAUSED`), avoiding upstream dispatch failures, and auto-resume immediately when refreshed with positive balance.
+  - Persistent balance state: balances are cached in `pool.state.balances` and persisted across harness restarts via `lib/persistence.js`.
+  - WebUI integration: interactive `Check Balances` action button in provider footer and real-time balance badge pills (`💰 15.50 CNY`, `💰 $4.20`) on key cards.
+
 ## 0.8.65 - 2026-10-18
 
 ### Features

package/README.md

@@ -67,6 +67,10 @@
 - **📡 Status extras**: per-provider `circuit` plus `meta.expectedClones` / `meta.notifyQueue`.
 - **🧪 Smoke harness**: scripted 429 → next-key → success path in `test/smoke-rotation-080.test.mjs`.
 
+### 🚀 What's New in v0.8.66
+- **⏳ Dynamic Reset Scheduler (#498)**: Reads upstream `retry-after` and `x-ratelimit-reset-*` headers to set exact cooldowns (+250ms safety jitter) and schedules immediate unref wake-up timers when the upstream reset window rolls over.
+- **💰 Upstream Balance Inspector & Credit Tracking (#495)**: Live balance checks for DeepSeek, SiliconFlow, OpenRouter, and OpenAI via `/dsh-key-rotation/balance`, pre-emptive empty balance protection (`EMPTY_BALANCE_PAUSED`), persistent balance cache, and WebUI `💰 Balance` pills.
+
 ### 🚀 What's New in v0.8.65
 - **⚡ Leaky Bucket Traffic Smoothing (#496)**: Micro-queue smoothing absorbs rapid multi-agent burst spikes ($\le 1$s) without triggering unnecessary 429 errors or cooldown churning, with zero-delay burst capacity and early failover rotation on queue overflow.
 - **📊 Dynamic Upstream Rate Limits & Presets (#503)**: Dynamic parsing of provider rate limit headers (`x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after`), live badge display on keys, and 1-click `⚡ Preset Limits` button applying canonical RPM/TPM presets for DeepSeek, Groq, Anthropic, OpenAI, Gemini, and more.

package/README.ru.md

@@ -36,6 +36,10 @@
 
 ---
 
+### 🚀 Что нового в v0.8.66
+- **⏳ Динамический планировщик сброса кулдауна (#498)**: Точное вычисление времени восстановления ключей по заголовкам `retry-after` и `x-ratelimit-reset-*` (+250мс джиттер) с запуском точного unref-таймера восстановления сразу по истечении окна, не дожидаясь 30-секундного периодического свипа.
+- **💰 Инспектор баланса и отслеживание кредитов (#495)**: Запрос реального баланса провайдеров (DeepSeek, SiliconFlow, OpenRouter, OpenAI) через маршрут `/dsh-key-rotation/balance`, автоматическая мягкая пауза при нулевом балансе (`EMPTY_BALANCE_PAUSED`), персистентное сохранение и плашки баланса в WebUI карточках ключей.
+
 ### 🚀 Что нового в v0.8.65
 - **⚡ Сглаживание трафика (Traffic Smoothing Micro-Queue) (#496)**: Микро-очередь leaky bucket сглаживает кратковременные пики нагрузки (<=1с) от мультиагентных вызовов без лишних ошибок 429 и лишних кулдаунов. При превышении таймаута ожидания микро-очереди запрос автоматически переключается на следующий ключ в пуле.
 - **📊 Динамическое определение лимитов провайдеров и пресеты (#503)**: Автоматический парсинг заголовков `x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after` с отображением актуальных лимитов прямо на карточке ключа, а также кнопка в 1 клик `⚡ Preset Limits` с каноническими пресетами для DeepSeek, Groq, Anthropic, OpenAI, Gemini и др.

package/README.zh.md

@@ -38,6 +38,10 @@
 
 ## ⚡ 概述与核心痛点
 
+### 🚀 v0.8.66 新特性
+- **⏳ 基于限流响应头的动态重置调度器 (#498)**: 解析上游 `retry-after` 与 `x-ratelimit-reset-*` 响应头，设定精确冷却时间（+250ms抖动保护），并通过独立 unref 定时器在窗口重置瞬间立刻唤醒可用密钥，无需等待30秒周期性轮询。
+- **💰 上游余额与信用额度巡检 (#495)**: 支持 DeepSeek、SiliconFlow、OpenRouter、OpenAI 实时余额查询 (`/dsh-key-rotation/balance`)，零余额前置软暂停 (`EMPTY_BALANCE_PAUSED`)，持久化余额状态以及 WebUI 密钥卡片实时余额标签 (`💰 15.50 CNY`)。
+
 ### 🚀 v0.8.65 新特性
 - **⚡ 流量平滑微队列 (Traffic Smoothing Micro-Queue) (#496)**: 漏桶微队列平滑多智能体调用中的毫秒级突发并发，避免触发不必要的429限流与冷却流转。队列等待超限时自动轮转至池中下一个候选密钥。
 - **📊 动态上游限制检测与预设规则 (#503)**: 自动提取上游响应头中的限流数据 (`x-ratelimit-*`, `anthropic-ratelimit-*`, `retry-after`)，并在前端展示实时配额状态；支持一键应用 DeepSeek、Groq、Anthropic、OpenAI、Gemini 等官方规范预设限制。

package/lib/balance-inspector.js

@@ -0,0 +1,176 @@
+// lib/balance-inspector.js — upstream balance inspector and credit tracking (#495).
+
+import { decryptSecret } from './crypto-storage.js';
+
+/**
+ * Inspect upstream balance and credits for a provider API key.
+ * @param {string} provider Provider identifier (e.g., 'deepseek', 'siliconflow', 'openrouter', 'openai')
+ * @param {string} apiKey Raw API key
+ * @param {object} [options]
+ * @returns {Promise<{ ok: boolean, total?: number, currency?: string, isAvailable?: boolean, error?: string }>}
+ */
+export async function checkProviderBalance(provider, apiKey, options = {}) {
+  const fetchImpl = options.fetchImpl ?? globalThis.fetch;
+  const timeoutMs = typeof options.timeoutMs === 'number' && options.timeoutMs > 0 ? options.timeoutMs : 6000;
+  const signal = options.signal;
+  const prov = String(provider || '').toLowerCase().trim();
+
+  if (!apiKey || typeof apiKey !== 'string' || !apiKey.trim()) {
+    return { ok: false, error: 'missing api key' };
+  }
+
+  // 1. DeepSeek Official
+  if (prov.includes('deepseek')) {
+    try {
+      const url = 'https://api.deepseek.com/user/balance';
+      const ac = new AbortController();
+      const timer = setTimeout(() => ac.abort(), timeoutMs);
+      const res = await fetchImpl(url, {
+        method: 'GET',
+        headers: {
+          'Authorization': `Bearer ${apiKey.trim()}`,
+          'Accept': 'application/json',
+        },
+        signal: signal ?? ac.signal,
+      });
+      clearTimeout(timer);
+      if (!res.ok) {
+        return { ok: false, status: res.status, error: `HTTP ${res.status}` };
+      }
+      const data = await res.json();
+      const info = data?.balance_infos?.[0];
+      const total = info ? parseFloat(info.total_balance) : 0;
+      const currency = info?.currency || 'CNY';
+      const isAvailable = Boolean(data?.is_available && (Number.isNaN(total) || total > 0));
+      return {
+        ok: true,
+        provider: 'deepseek',
+        total: Number.isFinite(total) ? total : 0,
+        currency,
+        isAvailable,
+        granted: parseFloat(info?.granted_balance || 0),
+        toppedUp: parseFloat(info?.topped_up_balance || 0),
+      };
+    } catch (err) {
+      return { ok: false, error: String(err?.message || err) };
+    }
+  }
+
+  // 2. SiliconFlow
+  if (prov.includes('siliconflow')) {
+    try {
+      const url = 'https://api.siliconflow.cn/v1/user/info';
+      const ac = new AbortController();
+      const timer = setTimeout(() => ac.abort(), timeoutMs);
+      const res = await fetchImpl(url, {
+        method: 'GET',
+        headers: {
+          'Authorization': `Bearer ${apiKey.trim()}`,
+          'Accept': 'application/json',
+        },
+        signal: signal ?? ac.signal,
+      });
+      clearTimeout(timer);
+      if (!res.ok) {
+        return { ok: false, status: res.status, error: `HTTP ${res.status}` };
+      }
+      const json = await res.json();
+      const total = json?.data?.totalBalance !== undefined
+        ? parseFloat(json.data.totalBalance)
+        : (json?.data?.balance !== undefined ? parseFloat(json.data.balance) : 0);
+      const isAvailable = Number.isFinite(total) ? total > 0 : true;
+      return {
+        ok: true,
+        provider: 'siliconflow',
+        total: Number.isFinite(total) ? total : 0,
+        currency: 'CNY',
+        isAvailable,
+      };
+    } catch (err) {
+      return { ok: false, error: String(err?.message || err) };
+    }
+  }
+
+  // 3. OpenRouter
+  if (prov.includes('openrouter')) {
+    try {
+      const url = 'https://openrouter.ai/api/v1/auth/key';
+      const ac = new AbortController();
+      const timer = setTimeout(() => ac.abort(), timeoutMs);
+      const res = await fetchImpl(url, {
+        method: 'GET',
+        headers: {
+          'Authorization': `Bearer ${apiKey.trim()}`,
+          'Accept': 'application/json',
+        },
+        signal: signal ?? ac.signal,
+      });
+      clearTimeout(timer);
+      if (!res.ok) {
+        return { ok: false, status: res.status, error: `HTTP ${res.status}` };
+      }
+      const json = await res.json();
+      const limit = json?.data?.limit;
+      const usage = json?.data?.usage ?? 0;
+      const total = limit != null ? Math.max(0, limit - usage) : null;
+      return {
+        ok: true,
+        provider: 'openrouter',
+        total: total != null ? total : 999,
+        currency: 'USD',
+        isAvailable: total == null || total > 0,
+      };
+    } catch (err) {
+      return { ok: false, error: String(err?.message || err) };
+    }
+  }
+
+  // 4. OpenAI Credit Grants
+  if (prov.includes('openai')) {
+    try {
+      const url = 'https://api.openai.com/v1/dashboard/billing/credit_grants';
+      const ac = new AbortController();
+      const timer = setTimeout(() => ac.abort(), timeoutMs);
+      const res = await fetchImpl(url, {
+        method: 'GET',
+        headers: {
+          'Authorization': `Bearer ${apiKey.trim()}`,
+          'Accept': 'application/json',
+        },
+        signal: signal ?? ac.signal,
+      });
+      clearTimeout(timer);
+      if (!res.ok) {
+        return { ok: false, status: res.status, error: `HTTP ${res.status}` };
+      }
+      const data = await res.json();
+      const total = parseFloat(data?.total_available ?? 0);
+      return {
+        ok: true,
+        provider: 'openai',
+        total: Number.isFinite(total) ? total : 0,
+        currency: 'USD',
+        isAvailable: total > 0,
+      };
+    } catch (err) {
+      return { ok: false, error: String(err?.message || err) };
+    }
+  }
+
+  return { ok: false, error: `Provider '${provider}' does not support balance inspection` };
+}
+
+/**
+ * Resolve credential and inspect balance for a given ref.
+ */
+export async function inspectBalanceForRef(ctx, ref, provider, { testValue, fetchImpl, timeoutMs } = {}) {
+  const base = ctx.get('credentials');
+  let hit = await (base?.__dshKeyRotationOriginalResolve ?? base?.resolve)?.call(base, ref);
+  let rawValue = testValue || hit?.value;
+  if (!rawValue && process.env[ref]) rawValue = process.env[ref];
+  if (typeof rawValue === 'string' && rawValue.startsWith('enc:v1:')) {
+    rawValue = decryptSecret(rawValue);
+  }
+  if (!rawValue) return { ok: false, error: 'no credential' };
+  return checkProviderBalance(provider, rawValue, { fetchImpl, timeoutMs });
+}

package/lib/client.js

@@ -104,6 +104,11 @@ window.__ModuleLoader__.load({
       presetNone: 'No canonical preset for this provider',
       detectedLimitsLabel: 'Upstream: {rpm} RPM · {tpm} TPM',
       trafficSmoothingLabel: 'Traffic Smoothing (Burst)',
+      balanceLabel: 'Balance: {amt} {curr}',
+      balanceEmpty: 'Balance Empty (Paused)',
+      balanceCheckBtn: '💰 Check Balance',
+      checkingBalance: 'Checking…',
+      upstreamResetIn: '⚡ Reset in {time} (header)',
       pauseKey: 'Pause key from rotation',
       resumeKey: 'Resume key rotation',
       sparklineTraffic: 'Traffic Trend',
@@ -311,6 +316,11 @@ window.__ModuleLoader__.load({
       presetNone: '未找到该提供商预设规则',
       detectedLimitsLabel: '上游限制：{rpm} RPM · {tpm} TPM',
       trafficSmoothingLabel: '流量平滑 (突发并发)',
+      balanceLabel: '余额：{amt} {curr}',
+      balanceEmpty: '余额不足已暂停',
+      balanceCheckBtn: '💰 检查余额',
+      checkingBalance: '查询中…',
+      upstreamResetIn: '⚡ {time}后重置 (上游头信息)',
       pauseKey: '暂停此密钥轮换',
       resumeKey: '恢复此密钥轮换',
       sparklineTraffic: '流量趋势',
@@ -1567,6 +1577,26 @@ window.__ModuleLoader__.load({
 
       const [validating, setValidating] = React.useState('');
       const [presetNotice, setPresetNotice] = React.useState('');
+      const [checkingBalance, setCheckingBalance] = React.useState('');
+      const checkBalances = async (pIndex, provName) => {
+        setCheckingBalance(provName);
+        try {
+          const r = await fetch('/dsh-key-rotation/balance', {
+            method: 'POST',
+            headers: { 'Content-Type': 'application/json' },
+            body: JSON.stringify({ provider: provName }),
+            signal: (typeof AbortSignal !== 'undefined' && typeof AbortSignal.timeout === 'function') ? AbortSignal.timeout(15000) : undefined,
+          });
+          const data = await r.json();
+          if (data && data.ok) {
+            load();
+          }
+        } catch (err) {
+          setSecretError(String(err?.message || err));
+        } finally {
+          setCheckingBalance('');
+        }
+      };
       const applyPreset = async (pIndex, provName) => {
         try {
           const r = await fetch('/dsh-key-rotation/preset-limits?provider=' + encodeURIComponent(provName));
@@ -2162,6 +2192,18 @@ window.__ModuleLoader__.load({
           if (info && info.broken) {
             meta.push(h('button', { key: 'rt', className: 'krot-btn', onClick: () => retestBroken(key), title: t('retestBroken') }, t('retestBroken')));
           }
+          if (info && info.balance) {
+            const b = info.balance;
+            const isZero = b.isAvailable === false || (typeof b.total === 'number' && b.total <= 0);
+            const cur = b.currency || '';
+            const amtStr = typeof b.total === 'number' ? b.total.toFixed(2) : String(b.total || 0);
+            meta.push(h('span', {
+              key: 'bal',
+              className: 'krot-tail' + (isZero ? ' krot-badge-bad' : ''),
+              title: isZero ? t('balanceEmpty') : ('Upstream balance: ' + amtStr + ' ' + cur),
+              style: isZero ? { color: 'var(--dsw-alias-state-error-primary)', fontWeight: 700 } : undefined,
+            }, '💰 ' + amtStr + (cur ? (' ' + cur) : '')));
+          }
           if (info && info.detectedLimits) {
             const dl = info.detectedLimits;
             const parts = [];
@@ -2509,6 +2551,10 @@ window.__ModuleLoader__.load({
               danger: false,
               onConfirm: () => doReset(entry.provider),
             }), { disabled: !(providerStatus && providerStatus.switches > 0) || resetting === entry.provider, title: t('resetCooldown') }),
+            btn(checkingBalance === entry.provider ? t('checkingBalance') : ('💰 ' + t('balanceCheckBtn')), () => checkBalances(pIndex, entry.provider), {
+              disabled: Boolean(checkingBalance) || !entry.provider.trim(),
+              title: t('balanceCheckBtn'),
+            }),
             btn(testAllProvider === entry.provider ? (testingProgress[entry.provider] ? t('testingAllProgress').replace('{n}', testingProgress[entry.provider].split('/')[0]).replace('{total}', testingProgress[entry.provider].split('/')[1]) : t('testing')) : t('testAll'), () => doTestAll(entry.provider), { disabled: Boolean(testing || testAllProvider), title: t('testAll') }),
             exportCsv,
           ),

package/lib/ops-balance.js

@@ -0,0 +1,147 @@
+// lib/ops-balance.js — operational HTTP routes for balance inspection (#495).
+
+import {
+  json,
+  readJson,
+} from './http-bridge.js';
+import {
+  isTrustedBridgeRequest,
+  isValidRef,
+} from './pool.js';
+import { BALANCE_PATH } from './ops-paths.js';
+import { inspectBalanceForRef } from './balance-inspector.js';
+import { pushEvent } from './notify-events.js';
+
+export function registerBalanceRoutes(ctx, deps) {
+  const {
+    poolState,
+    buildRuntime,
+  } = deps;
+
+  ctx.effect(() => ctx.webServer.register({
+    kind: 'exact',
+    path: BALANCE_PATH,
+    handler: async (req, res) => {
+      if (!isTrustedBridgeRequest(req)) {
+        json(res, 403, { ok: false, error: 'Forbidden' });
+        return;
+      }
+
+      const urlObj = new URL(req.url, 'http://127.0.0.1');
+
+      // GET: Query single key or provider cached balances
+      if (req.method === 'GET') {
+        const ref = urlObj.searchParams.get('ref');
+        const provider = urlObj.searchParams.get('provider');
+
+        if (ref) {
+          const runtime = buildRuntime();
+          const targetPool = provider ? runtime.pools?.find((p) => p.provider === provider) : null;
+          const pool = targetPool ?? runtime.poolByRef?.get(ref);
+          const cached = pool?.state?.balances?.get(ref) ?? null;
+
+          if (urlObj.searchParams.get('refresh') === 'true' && provider) {
+            const inspectRes = await inspectBalanceForRef(ctx, ref, provider);
+            if (inspectRes.ok) {
+              if (pool && pool.state) {
+                if (!pool.state.balances) pool.state.balances = new Map();
+                pool.state.balances.set(ref, { ...inspectRes, at: Date.now() });
+
+                // Check soft pause
+                applyBalanceState(pool, ref, inspectRes);
+              }
+              json(res, 200, { ok: true, ref, balance: inspectRes });
+              return;
+            }
+            json(res, 200, { ok: false, ref, error: inspectRes.error, cached });
+            return;
+          }
+
+          json(res, 200, { ok: true, ref, balance: cached });
+          return;
+        }
+
+        // Return all cached balances
+        const out = {};
+        if (poolState && typeof poolState.forEach === 'function') {
+          poolState.forEach((st, provName) => {
+            if (st && st.balances instanceof Map) {
+              for (const [k, v] of st.balances) {
+                out[k] = v;
+              }
+            }
+          });
+        }
+        json(res, 200, { ok: true, balances: out });
+        return;
+      }
+
+      // POST: Trigger refresh of balances for a provider or key list
+      if (req.method === 'POST') {
+        const body = await readJson(req);
+        const provider = body?.provider;
+        const ref = body?.ref;
+        const testValue = body?.value;
+
+        if (ref && provider) {
+          const inspectRes = await inspectBalanceForRef(ctx, ref, provider, { testValue });
+          const runtime = buildRuntime();
+          const pool = runtime.pools?.find((p) => p.provider === provider) ?? runtime.poolByRef?.get(ref);
+
+          if (inspectRes.ok && pool && pool.state) {
+            if (!pool.state.balances) pool.state.balances = new Map();
+            pool.state.balances.set(ref, { ...inspectRes, at: Date.now() });
+            applyBalanceState(pool, ref, inspectRes);
+          }
+
+          json(res, inspectRes.ok ? 200 : 400, { ok: inspectRes.ok, ref, balance: inspectRes });
+          return;
+        }
+
+        if (provider) {
+          const runtime = buildRuntime();
+          const pool = runtime.pools?.find((p) => p.provider === provider);
+          if (!pool || !Array.isArray(pool.refs) || pool.refs.length === 0) {
+            json(res, 404, { ok: false, error: `Provider '${provider}' not found or has no keys` });
+            return;
+          }
+
+          const results = {};
+          for (const kRef of pool.refs) {
+            const r = await inspectBalanceForRef(ctx, kRef, provider);
+            results[kRef] = r;
+            if (r.ok && pool.state) {
+              if (!pool.state.balances) pool.state.balances = new Map();
+              pool.state.balances.set(kRef, { ...r, at: Date.now() });
+              applyBalanceState(pool, kRef, r);
+            }
+          }
+          json(res, 200, { ok: true, provider, balances: results });
+          return;
+        }
+
+        json(res, 400, { ok: false, error: 'ref or provider required' });
+        return;
+      }
+
+      json(res, 405, { ok: false, error: 'Method Not Allowed' });
+    },
+  }), 'dsh-key-rotation: balance routes');
+}
+
+function applyBalanceState(pool, ref, balanceRes) {
+  if (!pool || !pool.state) return;
+  const isZero = balanceRes.isAvailable === false || (typeof balanceRes.total === 'number' && balanceRes.total <= 0);
+
+  if (isZero) {
+    if (!pool.state.pausedRefs) pool.state.pausedRefs = new Set();
+    if (pool.state.pausedRefs instanceof Set) pool.state.pausedRefs.add(ref);
+    if (!pool.state.pauseReasons) pool.state.pauseReasons = new Map();
+    pool.state.pauseReasons.set(ref, 'EMPTY_BALANCE_PAUSED');
+    pushEvent(pool, ref, 'BALANCE_EMPTY', 0);
+  } else if (pool.state.pauseReasons?.get?.(ref) === 'EMPTY_BALANCE_PAUSED') {
+    if (pool.state.pausedRefs instanceof Set) pool.state.pausedRefs.delete(ref);
+    pool.state.pauseReasons.delete(ref);
+    pushEvent(pool, ref, 'BALANCE_RESTORED', 0);
+  }
+}

package/lib/ops-paths.js

@@ -18,3 +18,4 @@ export const TEST_MATRIX_PATH = '/dsh-key-rotation/test-matrix';
 export const DISCOVER_PATH = '/dsh-key-rotation/discover';
 export const AUTO_IMPORT_PATH = '/dsh-key-rotation/auto-import';
 export const PRESET_LIMITS_PATH = '/dsh-key-rotation/preset-limits';
+export const BALANCE_PATH = '/dsh-key-rotation/balance';

package/lib/ops-status.js

@@ -124,6 +124,9 @@ ctx.effect(() => ctx.webServer.register({
           revoked: isKeyRevoked(pool, ref),
           modelCapabilities: pool.state.modelCapabilities?.get?.(ref) ? [...pool.state.modelCapabilities.get(ref)] : null,
           detectedLimits: pool.state.detectedLimits?.get?.(ref) ?? null,
+          balance: pool.state.balances?.get?.(ref) ?? null,
+          resetReason: pool.state.resetReasons?.get?.(ref) ?? null,
+          headerResetMs: pool.state.headerResetTimes?.get?.(ref) ?? null,
           modelQuota,
           modelPools: activeModelPools.map(mp => mp.base),
         });

package/lib/persistence.js

@@ -138,6 +138,13 @@ export class StatePersistence {
         if (pausedRefs.length > 0) outPool.pausedRefs = pausedRefs;
         if (Object.keys(pauseReasons).length > 0) outPool.pauseReasons = pauseReasons;
         if (Object.keys(modelCapabilities).length > 0) outPool.modelCapabilities = modelCapabilities;
+        const balances = {};
+        if (st && st.balances && typeof st.balances.forEach === 'function') {
+          st.balances.forEach((bal, ref) => {
+            if (bal && typeof bal === 'object') balances[ref] = bal;
+          });
+        }
+        if (Object.keys(balances).length > 0) outPool.balances = balances;
         if (Object.keys(tokenUsage).length > 0) outPool.tokenUsage = tokenUsage;
         if (Object.keys(costDays).length > 0) outPool.costDays = costDays;
         if (Object.keys(costPerKey).length > 0) outPool.costPerKey = costPerKey;
@@ -241,6 +248,12 @@ export class StatePersistence {
             if (typeof reason === 'string') state.pauseReasons.set(ref, reason);
           }
         }
+        if (saved.balances && typeof saved.balances === 'object' && !Array.isArray(saved.balances)) {
+          if (!state.balances) state.balances = new Map();
+          for (const [ref, bal] of Object.entries(saved.balances)) {
+            if (bal && typeof bal === 'object') state.balances.set(ref, bal);
+          }
+        }
         if (saved.modelCapabilities && typeof saved.modelCapabilities === 'object' && !Array.isArray(saved.modelCapabilities)) {
           if (!state.modelCapabilities) state.modelCapabilities = new Map();
           for (const [ref, list] of Object.entries(saved.modelCapabilities)) {

package/lib/pool-state.js

@@ -3,7 +3,7 @@
 const MAP_FIELDS = [
   'failedUntil', 'failCounts', 'authFailCounts', 'brokenUntil', 'costPerKey',
   'lastUsedAt', 'usageCounts', 'byModel', 'usageDays', 'quotaWindows', 'costDays',
-  'tokenUsage', 'rpmWindows', 'tpmWindows', 'sessionSticky', 'modelCapabilities', 'pauseReasons', 'detectedLimits',
+  'tokenUsage', 'rpmWindows', 'tpmWindows', 'sessionSticky', 'modelCapabilities', 'pauseReasons', 'detectedLimits', 'balances', 'resetReasons', 'headerResetTimes',
 ];
 
 const SET_FIELDS = ['pausedRefs', 'revokedRefs'];

package/lib/rotate.js

@@ -197,16 +197,51 @@ export function createRotate(deps) {
         }
       }
 
-      const penalizeRef = (targetRef, errCode, errMsg) => {
+      const penalizeRef = (targetRef, errCode, errMsg, rawHeaders) => {
         if (!targetRef) return;
         if (sessionId) clearSessionSticky(pool, sessionId);
         if (errCode === LOCAL_QUOTA_CODE) return;
-        const _retry = parseRetryAfter(errMsg);
+        let headerResetMs = null;
+        if (rawHeaders) {
+          const limits = (typeof rawHeaders === 'object' && rawHeaders.rpmResetMs !== undefined)
+            ? rawHeaders
+            : extractProviderLimits(rawHeaders);
+          headerResetMs = limits?.retryAfterMs ?? limits?.rpmResetMs ?? null;
+        }
+        if (!headerResetMs) {
+          const pRetry = parseRetryAfter(errMsg);
+          if (pRetry) headerResetMs = pRetry;
+        }
+
         const _base = pool.cooldownMs ?? cooldownMs;
-        const _max = pool.maxCooldownMs ?? maxCooldownMs;
-        const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
+        const _max = pool.maxCooldownMs ?? maxCooldownMs ?? 3600000;
+        let _b = 0;
         const cls = classifyFailure({ status: Number(errCode) || 0, code: errCode, message: errMsg });
-        const _b = recordFailure(pool, targetRef, now(), _effBase, _max, cls.soft, true);
+
+        if (headerResetMs && headerResetMs > 0) {
+          const dynamicCooldownMs = Math.min(_max, Math.max(1000, Math.floor(headerResetMs) + 250));
+          pool.state.failedUntil.set(targetRef, now() + dynamicCooldownMs);
+          if (!pool.state.resetReasons) pool.state.resetReasons = new Map();
+          pool.state.resetReasons.set(targetRef, 'header');
+          if (!pool.state.headerResetTimes) pool.state.headerResetTimes = new Map();
+          pool.state.headerResetTimes.set(targetRef, dynamicCooldownMs);
+          _b = dynamicCooldownMs;
+          pushEvent(pool, targetRef, 'HEADER_RESET', dynamicCooldownMs);
+          logWarn(`[dsh-key-rotation] ${options.provider}: key ${targetRef} dynamic cooldown set to ${dynamicCooldownMs}ms via upstream headers`);
+
+          const recTimer = setTimeout(() => {
+            if ((pool.state.failedUntil?.get?.(targetRef) ?? 0) <= now()) {
+              pool.state.failedUntil.delete(targetRef);
+              pushEvent(pool, targetRef, 'DYNAMIC_UNBREAK', 0);
+            }
+          }, dynamicCooldownMs + 50);
+          if (typeof recTimer.unref === 'function') recTimer.unref();
+        } else {
+          const _effBase = _base;
+          _b = recordFailure(pool, targetRef, now(), _effBase, _max, cls.soft, true);
+          if (!pool.state.resetReasons) pool.state.resetReasons = new Map();
+          pool.state.resetReasons.set(targetRef, 'backoff');
+        }
 
         // #337 & #486 (GitHub #24): connect quota-window calendar reset for quota exhaustion
         // Transient rate limiting (RPM/TPM, requests/tokens per minute, concurrency)
@@ -409,7 +444,7 @@ export function createRotate(deps) {
               break;
             }
             const curRef = reqStore.pickedRef ?? pool.state.lastUsed;
-            penalizeRef(curRef, e?.code ?? 'TRANSPORT', String(e?.message ?? ''));
+            penalizeRef(curRef, e?.code ?? 'TRANSPORT', String(e?.message ?? ''), e?.headers ?? e?.response?.headers ?? pool.state.detectedLimits?.get?.(curRef));
             lastFailure = finishError(e?.code ?? 'TRANSPORT',
               `dsh-key-rotation: dispatch failed: ${String(e?.message ?? e)}`);
             logWarn(`[dsh-key-rotation] ${options.provider}: key ${String(curRef ?? '?')} threw ${String(e?.code ?? e?.message ?? e)}`);
@@ -434,6 +469,7 @@ export function createRotate(deps) {
                   continue;
                 }
                 if (chunk && chunk.type === 'finish') {
+                  const rawHdrs = chunk?.metadata?.headers ?? chunk?.headers ?? chunk?.reason?.failure?.headers;
                   const kind = chunk.reason?.kind;
                   const failure = chunk.reason?.failure;
                   const code = failure?.code;
@@ -442,7 +478,7 @@ export function createRotate(deps) {
                   const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
                   const activeRef = reqStore.pickedRef ?? pool.state.lastUsed;
                   if (switchable) {
-                    penalizeRef(activeRef, code ?? 'UNKNOWN', message);
+                    penalizeRef(activeRef, code ?? 'UNKNOWN', message, rawHdrs ?? failure?.headers ?? pool.state.detectedLimits?.get?.(activeRef));
                     pool.state.switches = (pool.state.switches ?? 0) + 1;
                     pool.state.lastReason = String(code ?? 'UNKNOWN');
                     pool.state.lastSwitchAt = now();
@@ -464,7 +500,7 @@ export function createRotate(deps) {
                   // content delta we may penalize, but must never replay the request.
                   if (kind === 'error' || kind === 'aborted') {
                     if (kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes)) {
-                      penalizeRef(activeRef, code ?? 'UNKNOWN', message);
+                      penalizeRef(activeRef, code ?? 'UNKNOWN', message, rawHdrs ?? failure?.headers ?? pool.state.detectedLimits?.get?.(activeRef));
                     }
                     yield chunk;
                     return;
@@ -528,7 +564,6 @@ export function createRotate(deps) {
                   // its quota or has retry-after, cool it down so the NEXT request starts on a different key.
                   // We do NOT re-run this (already successful) request — that would double-send.
                   // #503: Extract provider limits and reset headers
-                  const rawHdrs = chunk?.metadata?.headers ?? chunk?.headers;
                   if (rawHdrs && activeRef) {
                     const detected = extractProviderLimits(rawHdrs);
                     if (detected) {
@@ -574,7 +609,7 @@ export function createRotate(deps) {
               const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
               const activeRef = _pickedRef ?? reqStore.pickedRef ?? pool.state.lastUsed;
               if (!yielded && isSwitchableError(e, effectiveSwitchCodes)) {
-                penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+                penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e), e?.headers ?? e?.response?.headers ?? pool.state.detectedLimits?.get?.(activeRef));
                 pool.state.switches = (pool.state.switches ?? 0) + 1;
                 pool.state.lastReason = String(e?.code ?? 'TRANSPORT');
                 pool.state.lastSwitchAt = now();
@@ -591,7 +626,7 @@ export function createRotate(deps) {
                 continue; // Failover to next key!
               }
               if (yielded && isSwitchableError(e, effectiveSwitchCodes)) {
-                penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+                penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e), e?.headers ?? e?.response?.headers ?? pool.state.detectedLimits?.get?.(activeRef));
               }
               yield finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
               return;

package/lib/routes-ops.js

@@ -1,4 +1,5 @@
 import { registerMetricsRoutes } from './ops-metrics.js';
+import { registerBalanceRoutes } from './ops-balance.js';
 // lib/routes-ops.js — operational HTTP routes for dsh-key-rotation (#253/#312).
 // Thin orchestrator: each route family lives in its own lib/ops-*.js module.
 // Registration is injected with the live apply() dependencies.
@@ -20,4 +21,5 @@ export function registerOpsRoutes(ctx, deps) {
   registerTestRoutes(ctx, deps);
   registerWebhookActionRoute(ctx, deps);
   registerMetricsRoutes(ctx, deps);
+  registerBalanceRoutes(ctx, deps);
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@goodandready/dsh-key-rotation",
-  "version": "0.8.65",
+  "version": "0.8.66",
   "packageManager": "pnpm@10.33.2",
   "description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
   "keywords": [