@goodandready/dsh-key-rotation 0.7.36 → 0.7.38

package/README.md

@@ -253,6 +253,21 @@ All management routes require loopback authentication (`127.0.0.1` / `::1`) with
 
 MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
 
+### v0.7.38
+- **Hot-Path Stream Optimization**: Eliminated 4 redundant `buildRuntime()` calls inside the `rotate()` finish chunk handler by reusing the request-scoped `runtime0` snapshot.
+- **Allocation-Free Rate Limit Header Parsing**: Optimized `extractRateLimit()` with single-pass header inspection and length guards, completely removing dynamic lowercase/uppercase string allocations on every response chunk.
+- **Date ISO String Memoization**: Memoized `todayIso` string calculation once per finishing request instead of creating multiple `Date` instances for `costDays` and `usageDays`.
+- **Zero-Allocation Metrics Aggregation**: Replaced intermediate array allocation `[...values()].reduce()` with iterative summation for `totalUsage` in the `/dsh-key-rotation/status` endpoint.
+- **Stale Notification Cleanup**: Automatically purge stale entries in notification throttling maps (`budgetNotifiedAt`, `lowHealthNotifiedAt`) when provider pools are deleted.
+
+### v0.7.37
+- **Request Context Isolation via AsyncLocalStorage**: Scoped active key resolution (`pickedRef`), start timestamps, and retry counts strictly to each async dispatch context using Node's `node:async_hooks`. Eliminates race conditions where concurrent streaming requests could penalize healthy keys.
+- **Failover on Pre-Yield Stream Exceptions**: Fixed fatal stream termination where transport errors (e.g. HTTP 429 thrown before headers, socket hang-up) aborted the generator. The catch block now inspects `isSwitchableError` and cascades seamlessly to the next key if no content tokens have been yielded.
+- **Pure Local Candidate List in rotate()**: Generator uses an isolated local slice of keys (`attemptList`), eliminating shared mutations on `pool.weightedRefs`.
+- **Automatic Quarantine Release on Probe Success**: Successful sandbox model tests via Settings UI (`/dsh-key-rotation/test`) automatically lift `failedUntil` and `brokenUntil` quarantine flags.
+- **Long-Term Memory Compaction**: Wired `compactUsage(pool, 30, now)` into the periodic 30-second maintenance sweep to prevent memory growth on high-uptime servers.
+- **Request-Scoped Latency Recording**: Replaced module-global start timestamp with context-scoped `startMs` for accurate p50/p95 latency metrics under concurrent load.
+
 ### v0.7.36
 - **Architecture de-bloat & hardening**: Removed 6 unused/overengineered modules (`shadow`, `incident`, `agent-budget`, `region`, `canary`, `maintenance`) and dead route registrations.
 - **High-throughput buildRuntime memoization**: Eliminates per-token deep-cloning and schema validation on every streaming chunk.

package/README.ru.md

@@ -253,6 +253,21 @@ dsh-key-rotation:
 
 MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
 
+### v0.7.38
+- **Оптимизация горячего пути стриминга**: Устранены 4 избыточных вызова `buildRuntime()` при обработке завершающего чанка в `rotate()` за счёт повторного использования снимка `runtime0`.
+- **Парсинг заголовков лимитов без лишних аллокаций**: Оптимизирована функция `extractRateLimit()` — однократный проход по объекту с проверкой длины ключей без постоянных аллокаций строк `.toLowerCase()` и `.toUpperCase()`.
+- **Мемоизация строки даты ISO**: Дата текущего дня (`todayIso`) теперь вычисляется ровно один раз на запрос, исключая создание нескольких дублирующих объектов `Date` для `costDays` и `usageDays`.
+- **Итеративный подсчет totalUsage**: В эндпоинте статуса `/dsh-key-rotation/status` расчёт суммарного потребления переведён на прямой цикл без аллокации промежуточного массива `[...values()]`.
+- **Очистка устаревших записей уведомлений**: При удалении пулов провайдеров из конфигурации автоматически вычищаются связанные ключи из карт троттлинга уведомлений (`budgetNotifiedAt`, `lowHealthNotifiedAt`).
+
+### v0.7.37
+- **Изоляция контекста запросов через AsyncLocalStorage**: Полная привязка активного ключа (`pickedRef`), времени старта и попыток к асинхронному контексту вызова через `node:async_hooks`. Устранена гонка, при которой параллельные запросы могли ошибочно штрафовать здоровый ключ соседа.
+- **Failover при исключениях в потоке до первого чанка**: Устранено аварийное прерывание потока при транспортных сбоях (например, выброс HTTP 429 до отправки заголовков). Теперь блок перехвата проверяет `isSwitchableError` и прозрачно переключает поток на запасной ключ, если клиенту ещё не было отдано полезных данных.
+- **Чистый локальный список кандидатов в rotate()**: Устранена прямая мутация разделяемого массива `pool.weightedRefs` во время выполнения ротации.
+- **Автоматический вывод из карантина при успехе проверки**: Успешная проверка ключа через песочницу в настройках (`/dsh-key-rotation/test`) моментально сбрасывает флаги карантина (`failedUntil` и `brokenUntil`).
+- **Регулярная компактизация памяти**: Функция `compactUsage(pool, 30, now)` подключена в 30-секундный фоновый таймер пула, предотвращая утечки памяти при непрерывной многомесячной работе.
+- **Точные метрики задержки при параллельных вызовах**: Переход с глобальной переменной времени старта на контекстный замер `startMs` гарантирует достоверность p50/p95 latency под нагрузкой.
+
 ### v0.7.36
 - **Глубокий рефакторинг и деблоатинг**: Удалены 6 неиспользуемых модулей оверинжиниринга (`shadow`, `incident`, `agent-budget`, `region`, `canary`, `maintenance`) и сопутствующие устаревшие эндпоинты.
 - **Мемоизация buildRuntime**: Устранено глубокое клонирование и повторный парсинг схемы на каждом токене/чанге стрима.

package/README.zh.md

@@ -214,6 +214,21 @@ dsh-key-rotation:
 
 MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
 
+### v0.7.38
+- **热路径流处理优化**：在 `rotate()` 结束块处理中消除 4 次多余的 `buildRuntime()` 重复调用，直接复用请求作用域内的 `runtime0` 快照。
+- **零额外字符串分配的限流头解析**：重构 `extractRateLimit()`，采用单次遍历结合键长度检查，彻底消除对每个响应头执行 `.toLowerCase()` / `.toUpperCase()` 的内存碎片分配。
+- **日期 ISO 字符串记忆化**：在统计 `costDays` 与 `usageDays` 时将 `todayIso` 计算收敛为单次，杜绝重复创建 `Date` 实例。
+- **状态统计零数组分配**：在 `/dsh-key-rotation/status` 中将 `totalUsage` 的计算由 `[...values()].reduce()` 改为直接迭代累加，避免频繁轮询引发的垃圾回收波动。
+- **孤立通知记录自动清理**：在配置移除提供商模型池时，自动清理关联的通知限频缓存。
+
+### v0.7.37
+- **通过 AsyncLocalStorage 隔离请求上下文**：使用 Node.js 的 `node:async_hooks` 将解析后的密钥 (`pickedRef`)、启动时间和重试严格限定在单个请求上下文内，彻底消除并发请求间的竞态条件与误罚。
+- **流异常自动故障转移**：修复流在首个 token 返回前抛出传输异常（如 HTTP 429）直接终止的问题。若未发送内容块，现在会自动触发 `isSwitchableError` 并顺畅切换到备用密钥。
+- **避免在 rotate() 中直接修改共享状态**：遍历候选列表改用纯净的局部切片，不再直接覆盖修改 `pool.weightedRefs`。
+- **测试成功后自动解除隔离**：在设置界面通过沙箱成功验证密钥有效性后，自动清除 `failedUntil` 和 `brokenUntil` 惩罚标记。
+- **定期内存压缩清理**：将 `compactUsage(pool, 30, now)` 接入 30 秒后台巡检定时器，杜绝超长运行环境下的内存增长。
+- **并发环境下的精确延迟统计**：为每个请求独立计时，避免全局变量被并发请求覆盖导致 p50/p95 延迟失真。
+
 ### v0.7.36
 - **架构精简与稳定性加固**：移除 6 个过度设计的模块（`shadow`、`incident`、`agent-budget`、`region`、`canary`、`maintenance`）与废弃端点。
 - **buildRuntime 高性能记忆化**：消除每个流式 token/chunk 上的深拷贝与模式重解析开销。

package/lib/index.js

@@ -32,6 +32,7 @@
 //    cooldownMs: number      key cooldown after a switchable failure
 //    providers:  array       [{ provider, keys: [envName, ...] }]
 // ─────────────────────────────────────────────────────────────────────────────
+import { AsyncLocalStorage } from 'node:async_hooks';
 import Schema from '@deepseek-ai/schemastery';
 import { keyTail, isLoopbackAddress, isTrustedBridgeRequest, SWITCHABLE_MESSAGE_PATTERN, DEFAULT_SWITCH_CODES, isValidRef, pickNext, applyCooldown, recordFailure, recordSuccess, computeBackoff, envValue, sweepExpired, parseRetryAfter, computeHealthScore, extractRateLimit, isRateLimited, selectPool, isSwitchableError, formatExhaustionMessage, expiringSoon, shouldNotifyDaily, costForDay, costForWeek, budgetVerdict } from './pool.js';
 
@@ -61,7 +62,9 @@ import { nextQuotaReset } from './quota-window.js';
 import { QuotaStore } from './quota.js';
 import { WebhookSender } from './webhook.js';
 import { bucketAllow, bucketRetryMs, bucketSweep, bucketInfo } from './bucket.js';
-import { usageRows, usageCsv } from './usage-report.js';
+import { usageRows, usageCsv, compactUsage } from './usage-report.js';
+
+const dispatchStorage = new AsyncLocalStorage();
 import { findSecrets, looksLikeApiSecret } from './keycheck.js';
 
 /** The llm-pi-ai namespace whose provider profiles map providers to pools. */
@@ -449,6 +452,9 @@ export function apply(ctx, config = {}) {
       }
       const n = sweepExpired(poolState, now);
       if (n > 0) console.warn(`[dsh-key-rotation] sweep: cleared ${n} expired cooldown(s)`);
+      for (const pool of buildRuntime().poolByRef.values()) {
+        compactUsage(pool, 30, now);
+      }
       // #207 expiry pre-warning + #208 cost budget - piggybacked on this timer,
       // deduped to one notification per key/window per day (shouldNotifyDaily).
       try {
@@ -689,7 +695,11 @@ export function apply(ctx, config = {}) {
 
     // auto-cleanup: remove poolState for providers that are now empty or removed
     for (const key of [...poolState.keys()]) {
-      if (![...poolByRef.values()].some((p) => p.base === key)) poolState.delete(key);
+      if (![...poolByRef.values()].some((p) => p.base === key)) {
+        poolState.delete(key);
+        lowHealthNotifiedAt.delete(key);
+        budgetNotifiedAt.delete(key + ':budget');
+      }
     }
     // #192: drop RPM windows for refs that no longer belong to any pool
     for (const st of poolState.values()) {
@@ -761,6 +771,8 @@ export function apply(ctx, config = {}) {
         let hit = await original(candidate);
         if (hit && typeof hit.value === 'string' && hit.value.length > 0) {
           pool.state.lastUsed = candidate;
+          const store = dispatchStorage.getStore();
+          if (store && store.pool === pool) store.pickedRef = candidate;
           if (pool.state.failCounts) pool.state.failCounts.delete(candidate);
           pool.state.failedUntil.delete(candidate);
           if (pool.state.authFailCounts) pool.state.authFailCounts.delete(candidate);
@@ -780,6 +792,8 @@ export function apply(ctx, config = {}) {
         const envVal = envValue(candidate);
         if (envVal !== undefined) {
           pool.state.lastUsed = candidate;
+          const store = dispatchStorage.getStore();
+          if (store && store.pool === pool) store.pickedRef = candidate;
           if (pool.state.failCounts) pool.state.failCounts.delete(candidate);
           pool.state.failedUntil.delete(candidate);
           if (pool.state.authFailCounts) pool.state.authFailCounts.delete(candidate);
@@ -817,13 +831,14 @@ export function apply(ctx, config = {}) {
   // Latency recording (#6): record successful llm/stream latency per ref.
   // ponytail: only the true success path (finish-chunk). Failures are not recorded.
   let _rotateStartMs = Date.now();
-  function recordLatency(pool) {
+  function recordLatency(pool, reqStore) {
     try {
       const cfg = getConfig();
       if (!cfg || cfg.latencyEnabled === false) return;
-      const ref = pool && pool.state && pool.state.lastUsed;
+      const ref = reqStore?.pickedRef ?? pool?.state?.lastUsed;
       if (!ref) return;
-      const elapsed = Date.now() - _rotateStartMs;
+      const startMs = reqStore?.startMs ?? _rotateStartMs;
+      const elapsed = Date.now() - startMs;
       if (!Number.isFinite(elapsed) || elapsed < 0) return;
       latencyHistogram.record(ref, elapsed);
     } catch (_) { /* ponytail: never crash */ }
@@ -834,14 +849,15 @@ export function apply(ctx, config = {}) {
   // the resolve patch hands out the next key on each dispatch.
   function rotate(options, pool) {
     return (async function* () {
-      const { switchCodes, cooldownMs, maxCooldownMs } = buildRuntime();
-      let lastFailure = null;
-      _rotateStartMs = Date.now();
-
       const runtime0 = buildRuntime();
+      const { switchCodes, cooldownMs, maxCooldownMs, switchNotify, rateLimitThreshold } = runtime0;
+      let lastFailure = null;
+      const reqStore = { pool, pickedRef: undefined, startMs: Date.now() };
+      _rotateStartMs = reqStore.startMs;
+      let attemptList = (pool.weightedRefs ?? pool.refs).slice();
       if (runtime0.concurrencyLimit > 0 && concurrencyTracker.isEnabled()) {
         // #193: prefer least-loaded key within limit
-        const available = (pool.weightedRefs ?? pool.refs).filter((r) => {
+        const available = attemptList.filter((r) => {
           const fu = pool.state.failedUntil.get(r) ?? 0;
           if (fu > Date.now()) return false;
           const exp = pool.expiresAt ? pool.expiresAt[r] : undefined;
@@ -849,30 +865,54 @@ export function apply(ctx, config = {}) {
           return true;
         });
         const preferred = concurrencyTracker.pickLeastLoaded(available);
-        if (preferred && (pool.weightedRefs ?? pool.refs)[0] !== preferred) {
-          // Move preferred to front of the attempt list
-          const list = (pool.weightedRefs ?? pool.refs).slice();
+        if (preferred && attemptList[0] !== preferred) {
+          const list = attemptList.slice();
           const i = list.indexOf(preferred);
           if (i > 0) { list.splice(i, 1); list.unshift(preferred); }
-          pool.weightedRefs = list;
+          attemptList = list;
         }
       }
-      for (let attempt = 0; attempt < (pool.weightedRefs ?? pool.refs).length; attempt++) {
+
+      const penalizeRef = (targetRef, errCode, errMsg) => {
+        if (!targetRef) return;
+        const _retry = parseRetryAfter(errMsg);
+        const _base = pool.cooldownMs ?? cooldownMs;
+        const _max = pool.maxCooldownMs ?? maxCooldownMs;
+        const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
+        const _b = recordFailure(pool, targetRef, Date.now(), _effBase, _max);
+        pushEvent(pool, targetRef, errCode ?? 'UNKNOWN', _b);
+        if (!pool.state.authFailCounts) pool.state.authFailCounts = new Map();
+        if (!pool.state.brokenUntil) pool.state.brokenUntil = new Map();
+        const _cStr = String(errCode ?? '');
+        if (_cStr === 'AUTH' || /auth/i.test(errMsg)) {
+          const _c2 = (pool.state.authFailCounts.get(targetRef) ?? 0) + 1;
+          pool.state.authFailCounts.set(targetRef, _c2);
+          if (_c2 >= 3) {
+            pool.state.brokenUntil.set(targetRef, Date.now() + 86400000 * 30);
+            pool.state.failedUntil.set(targetRef, Date.now() + 86400000 * 30);
+          }
+        } else {
+          pool.state.authFailCounts.delete(targetRef);
+        }
+      };
+
+      for (let attempt = 0; attempt < attemptList.length; attempt++) {
         let yielded = false;
         let switching = false;
         let inner;
         try {
           // mark the internal dispatch so the interceptor does not re-rotate
-          inner = ctx.llm.stream({ ...options, [MARKER]: true });
+          inner = dispatchStorage.run(reqStore, () => ctx.llm.stream({ ...options, [MARKER]: true }));
         } catch (e) {
-          if (pool.state.lastUsed) { const _retry = parseRetryAfter(String(e?.message ?? '')); const _base = pool.cooldownMs ?? cooldownMs; const _max = pool.maxCooldownMs ?? maxCooldownMs; const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base; const _b = recordFailure(pool, pool.state.lastUsed, Date.now(), _effBase, _max); pushEvent(pool, pool.state.lastUsed, e?.code ?? 'TRANSPORT', _b); const _code = String(e?.code ?? ''); if (_code === 'AUTH' || /auth/i.test(String(e?.message ?? ''))) { const _c = (pool.state.authFailCounts.get(pool.state.lastUsed) ?? 0) + 1; pool.state.authFailCounts.set(pool.state.lastUsed, _c); if (_c >= 3) { pool.state.brokenUntil.set(pool.state.lastUsed, Date.now() + 86400000*30); pool.state.failedUntil.set(pool.state.lastUsed, Date.now() + 86400000*30); } } else { pool.state.authFailCounts.delete(pool.state.lastUsed); } }
+          const curRef = reqStore.pickedRef ?? pool.state.lastUsed;
+          penalizeRef(curRef, e?.code ?? 'TRANSPORT', String(e?.message ?? ''));
           lastFailure = finishError(e?.code ?? 'TRANSPORT',
             `dsh-key-rotation: dispatch failed: ${String(e?.message ?? e)}`);
-          console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(pool.state.lastUsed ?? '?')} threw ${String(e?.code ?? e?.message ?? e)}`);
+          console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(curRef ?? '?')} threw ${String(e?.code ?? e?.message ?? e)}`);
           continue;
         }
 
-        const _pickedRef = pool.state.lastUsed;
+        const _pickedRef = reqStore.pickedRef ?? pool.state.lastUsed;
         if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.acquire(_pickedRef);
         try {
           for await (const chunk of inner) {
@@ -890,39 +930,19 @@ export function apply(ctx, config = {}) {
               const message = failure?.message ?? '';
               const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
               const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
+              const activeRef = reqStore.pickedRef ?? pool.state.lastUsed;
               if (switchable) {
-                if (pool.state.lastUsed) {
-                  const _retry = parseRetryAfter(message);
-                  const _base = pool.cooldownMs ?? cooldownMs;
-                  const _max = pool.maxCooldownMs ?? maxCooldownMs;
-                  const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
-                  const _b = recordFailure(pool, pool.state.lastUsed, Date.now(), _effBase, _max);
-                  pushEvent(pool, pool.state.lastUsed, code ?? 'UNKNOWN', _b);
-                  // authFailCounts/brokenUntil: lazy-init if state was created by an older plugin version
-                  if (!pool.state.authFailCounts) pool.state.authFailCounts = new Map();
-                  if (!pool.state.brokenUntil) pool.state.brokenUntil = new Map();
-                  const _code2 = String(code ?? '');
-                  if (_code2 === 'AUTH' || /auth/i.test(message)) {
-                    const _c2 = (pool.state.authFailCounts.get(pool.state.lastUsed) ?? 0) + 1;
-                    pool.state.authFailCounts.set(pool.state.lastUsed, _c2);
-                    if (_c2 >= 3) {
-                      pool.state.brokenUntil.set(pool.state.lastUsed, Date.now() + 86400000*30);
-                      pool.state.failedUntil.set(pool.state.lastUsed, Date.now() + 86400000*30);
-                    }
-                  } else {
-                    pool.state.authFailCounts.delete(pool.state.lastUsed);
-                  }
-                }
+                penalizeRef(activeRef, code ?? 'UNKNOWN', message);
                 pool.state.switches = (pool.state.switches ?? 0) + 1;
                 pool.state.lastReason = String(code ?? 'UNKNOWN');
                 pool.state.lastSwitchAt = Date.now();
                 lastFailure = chunk;
-                console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(pool.state.lastUsed ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
+                console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
                 // #216: per-switch webhook (opt-in switchNotify), deduped per provider
-                if (buildRuntime().switchNotify && pool.state.lastUsed) {
-                  notifySwitch(buildRuntime(), pool, {
+                if (switchNotify && activeRef) {
+                  notifySwitch(runtime0, pool, {
                     provider: options.provider,
-                    from: pool.state.lastUsed,
+                    from: activeRef,
                     code: String(code ?? 'UNKNOWN'),
                     at: pool.state.lastSwitchAt,
                   });
@@ -931,59 +951,76 @@ export function apply(ctx, config = {}) {
                 break;
               }
               // cost tracking if provider returns usage.cost
-              if (chunk.usage?.cost != null && pool.state.lastUsed) {
+              const todayIso = activeRef ? new Date().toISOString().slice(0, 10) : undefined;
+              if (chunk.usage?.cost != null && activeRef) {
                 const c = Number(chunk.usage.cost);
                 if (!isNaN(c)) {
                   if (!pool.state.costPerKey) pool.state.costPerKey = new Map();
-                  pool.state.costPerKey.set(pool.state.lastUsed, (pool.state.costPerKey.get(pool.state.lastUsed) ?? 0) + c);
+                  pool.state.costPerKey.set(activeRef, (pool.state.costPerKey.get(activeRef) ?? 0) + c);
                   // #208: cost per day per key (mirrors usageDays) for budget checks
                   if (!pool.state.costDays) pool.state.costDays = new Map();
-                  const cday = new Date().toISOString().slice(0, 10);
-                  const cMap = pool.state.costDays.get(pool.state.lastUsed) || new Map();
-                  cMap.set(cday, (cMap.get(cday) ?? 0) + c);
-                  pool.state.costDays.set(pool.state.lastUsed, cMap);
+                  const cMap = pool.state.costDays.get(activeRef) || new Map();
+                  cMap.set(todayIso, (cMap.get(todayIso) ?? 0) + c);
+                  pool.state.costDays.set(activeRef, cMap);
                 }
               }
               // Usage by day (#119)
-              if (pool.state.lastUsed) {
+              if (activeRef) {
                 if (!pool.state.usageDays) pool.state.usageDays = new Map();
-                const day = new Date().toISOString().slice(0, 10);
-                const dayMap = pool.state.usageDays.get(pool.state.lastUsed) || new Map();
-                dayMap.set(day, (dayMap.get(day) ?? 0) + 1);
-                pool.state.usageDays.set(pool.state.lastUsed, dayMap);
+                const dayMap = pool.state.usageDays.get(activeRef) || new Map();
+                dayMap.set(todayIso, (dayMap.get(todayIso) ?? 0) + 1);
+                pool.state.usageDays.set(activeRef, dayMap);
               }
               // Per-model request detail (#121)
-              if (pool.state.lastUsed && options.model) {
+              if (activeRef && options.model) {
                 if (!pool.state.byModel) pool.state.byModel = new Map();
-                let byRef = pool.state.byModel.get(pool.state.lastUsed);
-                if (!byRef) { byRef = new Map(); pool.state.byModel.set(pool.state.lastUsed, byRef); }
+                let byRef = pool.state.byModel.get(activeRef);
+                if (!byRef) { byRef = new Map(); pool.state.byModel.set(activeRef, byRef); }
                 byRef.set(options.model, (byRef.get(options.model) ?? 0) + 1);
               }
               // Proactive rate-limit (#115): if response headers say this key is near
               // its quota, cool it down so the NEXT request starts on a different key.
               // We do NOT re-run this (already successful) request — that would double-send.
               const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
-              if (rate && pool.state.lastUsed) {
-                const { rateLimitThreshold } = buildRuntime();
+              if (rate && activeRef) {
                 if (isRateLimited(rate, rateLimitThreshold ?? 0.1)) {
                   const cool = rate.reset && rate.reset > Date.now() ? (rate.reset - Date.now()) : pool.cooldownMs;
-                  recordFailure(pool, pool.state.lastUsed, Date.now(), cool, pool.maxCooldownMs);
-                  pushEvent(pool, pool.state.lastUsed, 'RATE_LIMIT', cool);
-                  console.warn(`[dsh-key-rotation] ${options.provider}: key ${pool.state.lastUsed} near quota (remaining ${String(rate.remaining)}/${String(rate.limit)}) — next request will rotate`);
+                  recordFailure(pool, activeRef, Date.now(), cool, pool.maxCooldownMs);
+                  pushEvent(pool, activeRef, 'RATE_LIMIT', cool);
+                  console.warn(`[dsh-key-rotation] ${options.provider}: key ${activeRef} near quota (remaining ${String(rate.remaining)}/${String(rate.limit)}) — next request will rotate`);
                 }
               }
               // #7: persist quota snapshot regardless of threshold (so dashboard widget can show it).
-              if (rate && pool.state.lastUsed && Number.isFinite(rate.remaining)) {
-                quotaStore.set(pool.state.lastUsed, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: Date.now() });
+              if (rate && activeRef && Number.isFinite(rate.remaining)) {
+                quotaStore.set(activeRef, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: Date.now() });
               }
               yield chunk;
-              recordLatency(pool);
+              recordLatency(pool, reqStore);
               return;
             }
             yield chunk;
           }
         } catch (e) {
           if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.release(_pickedRef);
+          const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
+          const activeRef = _pickedRef ?? reqStore.pickedRef ?? pool.state.lastUsed;
+          if (!yielded && isSwitchableError(e, effectiveSwitchCodes)) {
+            penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+            pool.state.switches = (pool.state.switches ?? 0) + 1;
+            pool.state.lastReason = String(e?.code ?? 'TRANSPORT');
+            pool.state.lastSwitchAt = Date.now();
+            lastFailure = finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+            console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} stream threw ${String(e?.code ?? e?.message ?? e)} - failover to next key`);
+            if (switchNotify && activeRef) {
+              notifySwitch(runtime0, pool, {
+                provider: options.provider,
+                from: activeRef,
+                code: String(e?.code ?? 'TRANSPORT'),
+                at: pool.state.lastSwitchAt,
+              });
+            }
+            continue; // Failover to next key!
+          }
           yield finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
           return;
         }
@@ -1112,7 +1149,7 @@ export function apply(ctx, config = {}) {
           lastSwitchAt: pool.state.lastSwitchAt ?? null,
           lastExhaustionAt: pool.state.lastExhaustionAt ?? null,
           exhaustionCount: pool.state.exhaustionCount ?? 0,
-          totalUsage: [...(pool.state.usageCounts?.values() ?? [])].reduce((a, b) => a + b, 0),
+          totalUsage: (() => { let s = 0; if (pool.state.usageCounts) for (const v of pool.state.usageCounts.values()) s += v; return s; })(),
           // #225: aggregate p95 across the pool's keys
           p95: (() => {
             const vals = (pool.refs ?? []).map((r) => latencyHistogram.snapshot(r)).filter((s) => s && s.p95 != null).map((s) => s.p95);
@@ -1435,6 +1472,16 @@ export function apply(ctx, config = {}) {
           const result = probe === 'chat' ? await runner.probeChat(ref, keyForProbe) : await runner.probeModels(ref, keyForProbe);
           const cached = { ...result, at: Date.now() };
           lastTestCache.set(ref, cached);
+          if (cached.ok) {
+            for (const st of poolState.values()) {
+              if (st.failedUntil?.has(ref) || st.failCounts?.has(ref) || st.brokenUntil?.has(ref)) {
+                st.failedUntil?.delete(ref);
+                st.failCounts?.delete(ref);
+                st.authFailCounts?.delete(ref);
+                st.brokenUntil?.delete(ref);
+              }
+            }
+          }
           json(res, 200, { ok: cached.ok, ref, tail, source, probe, code: cached.code, latencyMs: cached.latencyMs, modelsCount: cached.modelsCount });
           return;
         }

package/lib/pool.js

@@ -248,14 +248,23 @@ export function computeHealthScore(state) {
 /** Extract rate-limit info from an object that may carry response headers. */
 export function extractRateLimit(headers) {
   if (!headers || typeof headers !== 'object') return null;
-  const get = (name) => {
-    const v = headers[name] ?? headers[name.toLowerCase()] ?? headers[name.toUpperCase()];
-    if (v === undefined || v === null) return undefined;
-    return Number(String(v));
-  };
-  const remaining = get('X-RateLimit-Remaining');
-  const limit = get('X-RateLimit-Limit');
-  const reset = get('X-RateLimit-Reset');
+  let remaining, limit, reset;
+  for (const k in headers) {
+    const lk = k.length;
+    if (lk === 21 || lk === 16 || lk === 17) {
+      const lower = k.toLowerCase();
+      if (lower === 'x-ratelimit-remaining') {
+        const v = headers[k];
+        if (v != null) remaining = Number(String(v));
+      } else if (lower === 'x-ratelimit-limit') {
+        const v = headers[k];
+        if (v != null) limit = Number(String(v));
+      } else if (lower === 'x-ratelimit-reset') {
+        const v = headers[k];
+        if (v != null) reset = Number(String(v));
+      }
+    }
+  }
   if (remaining === undefined && limit === undefined) return null;
   return { remaining, limit, reset };
 }

package/package.json

@@ -1,6 +1,7 @@
 {
   "name": "@goodandready/dsh-key-rotation",
-  "version": "0.7.36",
+  "version": "0.7.38",
+  "packageManager": "pnpm@10.33.2",
   "description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
   "keywords": [
     "deepseek-harness",
@@ -55,4 +56,4 @@
   "scripts": {
     "test": "node --test test/*.test.js test/*.test.mjs"
   }
-}
+}