@goodandready/dsh-key-rotation 0.8.8 → 0.8.10

package/README.md

@@ -22,12 +22,34 @@
   <a href="README.zh.md"><b>🇨🇳 中文说明</b></a>
 </p>
 
+<table align="center">
+  <tr>
+    <td align="center">
+      ⭐ <strong>If you like this plugin, please star it on GitHub</strong> — it shows me that the plugin is useful to you and motivates me to keep developing it.
+      <br><br>
+      🐛 <strong>If you find a bug or would like to request a feature</strong>, open a GitHub issue in any language — I will review your proposal and implement useful suggestions in a future plugin version.
+    </td>
+  </tr>
+</table>
+
 </div>
 
 ---
 
 ## ⚡ Overview & The Problem
 
+### 🚀 What's New in v0.8.10 (Stream Concurrency Hardening & Auto-Pruning)
+- **Zero Concurrency Leaks**: Guaranteed release of stream concurrency slots via deterministic `try ... finally` block, preventing key starvation during clean finishes or client stream aborts.
+- **Robust Probe Retry**: Added transient network socket error retry (`PROBE_RETRY_DELAY_MS`) in `SandboxRunner.probeModels` before marking keys as broken.
+- **Memory & State Pruning**: Automated pruning of removed/stale keys from internal pool maps during periodic sweep cycles.
+
+### 🚀 What's New in v0.8.9 (Routing Evolution & UI)
+- **Proactive Rate-Limit Guard**: Automatic key pausing based on `x-ratelimit-remaining-*` and `Retry-After` headers before hitting 429 errors.
+- **Self-Healing / Auto-Unbreak**: Periodic background probe via free `/models` endpoint to automatically revive broken keys without token burn.
+- **Latency-Aware Routing**: Selectable strategies: `round-robin`, `least-loaded` (concurrency), and `lowest-latency` (p95 latency).
+- **UI Evolution in `dsh-clinebot` Style**: Sequential batch "Test All Keys" runner with live progress, Live Event Stream drawer, and Quota Reset countdown badge.
+- **Native Dual Language Support**: Native English (`en`) and Chinese (`zh`) UI dictionaries.
+
 ### 🛠️ What's New in v0.8.0 (Stability)
 - **🔌 Circuit breaker**: after N consecutive provider failures the circuit opens and requests fail fast (`CIRCUIT_OPEN`) until a cool-down; half-open probes recover automatically.
 - **🕒 Monotonic clock**: cooldown/breaker durations use process monotonic time so NTP steps cannot invert remaining times.
@@ -38,7 +60,6 @@
 - **📡 Status extras**: per-provider `circuit` plus `meta.expectedClones` / `meta.notifyQueue`.
 - **🧪 Smoke harness**: scripted 429 → next-key → success path in `test/smoke-rotation-080.test.mjs`.
 
-
 ### 🛠️ What's New in v0.7.33 (Stability & Bugfix Release)
 - **🔍 Resolved Key Probing BaseURL**: Fixed `resolveBaseUrl` to map key credential refs to owning provider pools, restoring live `probeModels` testing.
 - **🛡️ Guarded Cascade Recursion**: Prevented call stack overflow in cross-provider failover when circular cascade chains occur.
@@ -49,7 +70,6 @@
 - **🎨 Native Design System (Changed in v0.8.3)**: Unified with `dsh-clinebot` baseline: modular section cards, live pool telemetry stat boxes, pill badges, and complete semantic theme token styling (#281).
 - **🌐 Localization (Changed in v0.8.2)**: Source strings are English-only. Russian/Chinese UI comes from the DSH core locale service and translation plugins (`props.t`). Active locale: host snapshot → first `navigator.languages` entry → `en` (#277).
 
-
 ### 🚀 What's New in v0.7.31
 - **⚡ O(1) TokenBucket Accumulator**: Upgraded rate limiting math to O(1) time and zero-allocation memory with adaptive header synchronization.
 - **🛡️ Soft vs Hard Backoff**: Differentiates transient infrastructure drops (502/503/timeouts: 10s flat cooldown) from hard quota errors (progressive doubling).
@@ -61,7 +81,6 @@
 - **🧹 30-Day Usage Compaction**: Automatic bounded memory management with 30-day rolling window data pruning.
 - **✨ Optimistic UI & Filter Pills**: Instant zero-latency UI updates on reset, plus `All`, `Ready`, `In Cooldown`, and `With Errors` quick filter chips.
 
-
 High-throughput autonomous agent workflows, parallel subagent swarms, and multi-turn tool loops inevitably hit upstream API rate limits (HTTP 429, RPM/TPM exhaustion, daily quotas, or sudden provider outages). In standard DeepSeek Harness deployments, a single exhausted API key breaks the entire agent execution chain, requiring manual intervention and destroying the session's replay state.
 
 **`dsh-key-rotation`** provides a seamless, enterprise-ready **transparent API key pooling, pre-emptive rate-limiting, and cross-provider failover engine** built natively on the Cordis microkernel architecture.

package/README.ru.md

@@ -22,12 +22,34 @@
   <a href="README.zh.md"><b>🇨🇳 中文说明</b></a>
 </p>
 
+<table align="center">
+  <tr>
+    <td align="center">
+      ⭐ <strong>Если вам нравится этот плагин, поставьте ему звезду на GitHub</strong> — это покажет мне, что плагин вам полезен, и будет мотивировать меня развивать его дальше.
+      <br><br>
+      🐛 <strong>Если вы нашли баг или хотите предложить новый функционал</strong>, создайте issue на GitHub на любом языке — я рассмотрю ваше предложение и реализую полезные идеи в одной из следующих версий плагина.
+    </td>
+  </tr>
+</table>
+
 </div>
 
 ---
 
 ## ⚡ Обзор и решаемая проблема
 
+### 🚀 Что нового в версии 0.8.10 (Надёжность стриминга и очистка памяти)
+- **Устранение утечек счётчика соединений**: Гарантированное освобождение слота конкурентности в `try ... finally`, исключающее блокировку ключей при успешном завершении стрима или прерывании клиентом.
+- **Повтор при сетевых сбоях проверки**: 1 автоматический повтор при кратковременных сокетных ошибках в `probeModels` перед дисквалификацией ключа.
+- **Очистка устаревших ключей**: Периодическая автоматическая очистка удалённых ключей из внутренних карт состояния пула при регулярном цикле `sweep`.
+
+### 🚀 Что нового в версии 0.8.9 (Развитие маршрутизации и UI)
+- **Превентивная защита от лимитов (Proactive Rate-Limit Guard)**: Автоматическая пауза ключей по заголовкам `x-ratelimit-remaining-*` и `Retry-After` до наступления ошибки 429.
+- **Авто-восстановление (Self-Healing / Auto-Unbreak)**: Периодический фоновый опрос эндпоинта `/models` для автоматического снятия статуса `broken` без расхода чат-токенов.
+- **Маршрутизация по задержке (Latency-Aware Routing)**: Выбор стратегии балансировки: `round-robin`, `least-loaded` (по активным запросам) и `lowest-latency` (по p95 задержки).
+- **Обновление интерфейса в стиле `dsh-clinebot`**: Пакетное тестирование всех ключей («Test All Keys») с прогрессом, аккордеон живого потока событий ротации (`.krot-event-stream`) и таймер обратного отсчета сброса квот.
+- **Встроенная поддержка двух языков**: Готовые словари интерфейса на английском (`en`) и китайском (`zh`).
+
 ### 🛠️ Что нового в версии 0.8.0 (Стабильность)
 - **🔌 Автомат выключения провайдера**: после N подряд ошибок контур открывается, запросы быстро завершаются (`CIRCUIT_OPEN`); half-open пробы восстанавливают сервис.
 - **🕒 Монотонные часы**: cooldown/breaker считаются по процессным часам, NTP не ломает оставшееся время.
@@ -38,7 +60,6 @@
 - **📡 Status**: `circuit` у провайдера + `meta.expectedClones` / `meta.notifyQueue`.
 - **🧪 Smoke**: сценарий 429 → следующий ключ → успех.
 
-
 ### 🛠️ Что нового в версии 0.7.33 (Хотфикс и повышение стабильности)
 - **🔍 Исправление проверки ключей**: Функция `resolveBaseUrl` теперь сопоставляет ref ключа с его пулом, восстанавливая работу живого тестирования моделей.
 - **🛡️ Защита от зацикливания каскада**: Устранена возможность переполнения стека при взаимных кольцевых цепочках failover.
@@ -47,7 +68,6 @@
 - **⚡ Сброс зависших блокировок**: Алгоритм `pickLeastLoaded` теперь учитывает протухшие блокировки при выборе наименее нагруженного ключа.
 - **🌐 Полная китайская локализация**: Добавлен словарь `zh` в веб-интерфейс настроек для соблюдения стандарта трёх языков (EN/RU/ZH).
 
-
 ### 🚀 Что нового в версии 0.7.31
 - **⚡ $O(1)$ Накопитель TokenBucket**: Оптимизация математики лимитов до $O(1)$ по времени и без аллокаций памяти с адаптивной синхронизацией по заголовкам.
 - **🛡️ Разделение Soft / Hard сбоев**: Кратковременные сетевые сбои (502/503/таймауты) получают короткий 10-секундный кулдаун без штрафного удвоения.
@@ -59,7 +79,6 @@
 - **🧹 30-дневная компактизация**: Автоматическая очистка истории старше 30 дней для защиты от утечек памяти.
 - **✨ Оптимистичный UI и фильтры**: Мгновенный отклик кнопок сброса и статус-пилюли (`Все`, `Готовы`, `В кулдауне`, `С ошибками`) над списком ключей.
 
-
 При активной работе автономных агентов, параллельном запуске субагентов и циклических вызовах инструментов запросы неизбежно упираются в ограничения провайдеров (ошибки HTTP 429 Too Many Requests, исчерпание суточных квот или временные сбои на стороне апстрима). В стандартной конфигурации DeepSeek Harness исчерпание одного API-ключа полностью блокирует цепочку рассуждений агента, ломает сохранённое состояние диалога (Replay State) и требует ручного вмешательства администратора.
 
 **`dsh-key-rotation`** реализует отказоустойчивую корпоративную архитектуру **пулов API-ключей с предиктивным контролем лимитов (Token Bucket) и каскадным переключением на резервных провайдеров**, нативно интегрированную в микроядро Cordis.

package/README.zh.md

@@ -22,12 +22,34 @@
   <a href="README.zh.md"><b>🇨🇳 中文说明</b></a>
 </p>
 
+<table align="center">
+  <tr>
+    <td align="center">
+      ⭐ <strong>如果您喜欢这个插件，请在 GitHub 上为它点亮 Star</strong> — 这能让我知道插件对您有用，并鼓励我继续开发和维护它。
+      <br><br>
+      🐛 <strong>如果您发现 Bug 或希望增加功能</strong>，请使用任意语言在 GitHub 上提交 Issue — 我会评估您的建议，并在后续版本中实现有价值的改进。
+    </td>
+  </tr>
+</table>
+
 </div>
 
 ---
 
 ## ⚡ 概述与核心痛点
 
+### 🚀 v0.8.10 版本新特性（流并发加固与状态自动修剪）
+- **杜绝并发计数泄漏**：在 `try ... finally` 中强制释放流并发占用，防止在正常完成或客户端中断时密钥被永久锁定。
+- **探测抗网络抖动重试**：在 `probeModels` 中遇到套接字网络临时错误时自动重试一次，避免密钥被误判损坏。
+- **内存与过期状态自动清理**：在定期清理周期中自动清除已删除密钥的内部映射记录。
+
+### 🚀 v0.8.9 版本新特性（智能路由与界面升级）
+- **前瞻性限流防护 (Proactive Rate-Limit Guard)**：根据 `x-ratelimit-remaining-*` 和 `Retry-After` 响应头在触发 429 错误前自动预冷密钥。
+- **自愈恢复 (Self-Healing / Auto-Unbreak)**：通过免费的 `/models` 接口进行定期后台探测，自动恢复处于 `broken` 状态的密钥，无需消耗聊天代币。
+- **延迟感知路由 (Latency-Aware Routing)**：可选路由策略：`round-robin`（轮询）、`least-loaded`（并发负载最低）与 `lowest-latency`（p95 延迟最低）。
+- **`dsh-clinebot` 风格界面升级**：支持带进度提示的批量“测试所有密钥”（Test All Keys）、实时事件流折叠抽屉（Live Event Stream）及配额重置倒计时徽标。
+- **原生中英文双语支持**：内置英文（`en`）与中文（`zh`）用户界面词典。
+
 ### 🛠️ v0.8.0 版本新特性（稳定性）
 - **🔌 熔断器**：连续失败后快速失败（`CIRCUIT_OPEN`），半开探测自动恢复。
 - **🕒 单调时钟**：冷却/熔断使用进程单调时间，NTP 校时不会颠倒剩余时间。
@@ -38,7 +60,6 @@
 - **📡 Status**：提供商 `circuit` + `meta`。
 - **🧪 Smoke**：429 → 切换密钥 → 成功。
 
-
 ### 🛠️ v0.7.33 版本新特性 (稳定性与问题修复)
 - **🔍 修复密钥探测 BaseURL 解析**：`resolveBaseUrl` 现已支持从密钥 ref 反查归属提供商池，恢复在线模型连通性探测。
 - **🛡️ 防御级联无限递归**：在跨提供商故障转移中增加递归深度防护，彻底杜绝循环级联导致的堆栈溢出。
@@ -47,7 +68,6 @@
 - **⚡ 负载均衡超时锁自动释放**：`pickLeastLoaded` 算法现已检测过期连接锁，确保最小连接调度不发生偏移。
 - **🌐 完整中文界面本地化**：为 React 设置面板补充全部 `zh` 语言包，实现标准的三语（英/俄/中）无缝对齐。
 
-
 ### 🚀 v0.7.31 版本新特性
 - **⚡ O(1) 令牌桶累加器**：将速率限制计算升级为 O(1) 时间复杂度与零内存分配，并支持响应头自适应同步。
 - **🛡️ 软/硬故障分级退避**：区分临时网络抖动（502/503/超时获得 10 秒平缓冷却）与硬性配额超限（指数退避倍增）。
@@ -59,7 +79,6 @@
 - **🧹 30 天用量压缩**：自动清理超过 30 天的历史统计数据，保障长期运行内存上限。
 - **✨ 乐观 UI 与快速筛选标签**：一键重置即时生效，密钥列表新增 `全部`、`就绪`、`冷却中`、`故障` 状态筛选胶囊。
 
-
 在高吞吐量自主智能体运行、多子智能体并行执行与多轮工具调用场景下，API 极易触发上游服务商的速率限制（HTTP 429 Too Many Requests、RPM/TPM 耗尽、每日配额限制或网络抖动）。在原生的 DeepSeek Harness 中，单个密钥耗尽会导致整个智能体执行链路崩溃，破坏会话的 Replay 状态并要求人工干预。
 
 **`dsh-key-rotation`** 基于 Cordis 微内核架构构建，提供了无缝透明的 **API 密钥池轮换、客户端预判限流（Token Bucket）与跨提供商故障转移（Failover Cascade）** 解决方案。
@@ -263,7 +282,6 @@ MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
 - **本地化与文案 (Changed in v0.8.2)**: 插件仅注册英文源字符串 `en`；俄文/中文由 DSH 核心 locale 与 translation 插件通过 `props.t` 提供。活动语言回退：snapshot → 首个 `navigator.languages` → `en`。已移除 `settings.section` 回退与内置 `ru`/`zh` 表 (#236, #275, #277)。
 - **死代码清理**: 移除 header-chip 迁移后残留的废弃 `mountDashboard` 函数 (#240)。
 
-
 ### 熔断器参数（v0.8.0）
 
 | 参数 | 类型 | 默认 | 说明 |

package/lib/client.js

@@ -143,8 +143,162 @@ window.__ModuleLoader__.load({
       recentFailures: 'Recent failures ({n})',
       noActivePools: 'No active pools',
       headerPoolsTitle: 'Key rotation pools',
-      pausedBudget: 'paused',
+            pausedBudget: 'paused',
       sortUsage: 'Sort by usage',
+      testingAllProgress: 'Testing {n}/{total}…',
+      liveEventStream: 'Live Event Stream',
+      liveEventStreamEmpty: 'No rotation events recorded',
+      eventAt: 'Time',
+      eventKey: 'Key',
+      eventReason: 'Reason',
+      eventCooldown: 'Cooldown',
+      quotaResetIn: 'Reset in {time}',
+      routingStrategyLabel: 'Routing Strategy',
+      routingStrategyRoundRobin: 'Round-Robin (default)',
+      routingStrategyLeastLoaded: 'Least-Loaded (concurrency)',
+      routingStrategyLowestLatency: 'Lowest-Latency (p95)',
+      proactiveGuardLabel: 'Proactive Rate-Limit Guard',
+      proactiveGuardHint: 'Pre-emptively pause keys when remaining quota or Retry-After header indicates limit before hitting 429',
+      selfHealingLabel: 'Auto-Unbreak Interval (min)',
+      selfHealingHint: 'Periodic free model check to revive broken keys (0 to disable)',
+    };
+
+    
+    const zh = {
+      title: '密钥轮换',
+      'header.title': '密钥轮换与故障转移',
+      'header.sub': '自动化多密钥轮换、429/5xx指数退避冷却、断路器保护及凭证安全 (~/.dsh/.credentials.yaml)。',
+      'header.pools_badge': '个活跃密钥池',
+      'header.keys_badge': '个已配置密钥',
+      sectionStats: '密钥池健康与遥测',
+      sectionStatsDesc: '活跃密钥池、健康凭证和故障转移状态的实时遥测。',
+      sectionFailover: '故障转移触发器与错误代码',
+      sectionFailoverDesc: '选择触发自动切换至下一个可用密钥的错误条件。',
+      sectionTiming: '时间与轮换计划',
+      sectionTimingDesc: '配置冷却持续时间和定期轮换计划窗口。',
+      sectionBackup: '备份与快照',
+      sectionBackupDesc: '导出或导入提供商密钥池，并创建完整插件配置快照。',
+      statPools: '已配置密钥池',
+      statKeys: '总密钥数',
+      statHealthy: '健康 / 就绪',
+      loadDistribution: '流量分布',
+      statRequests: '请求',
+      noTrafficYet: '暂无流量记录',
+      confirm: '确认',
+      cancel: '取消',
+      confirmResetTitle: '重置 {p} 的冷却状态？',
+      confirmResetDesc: '这将立即清除所有故障计数、冷却计时器并重置断路器。',
+      confirmRemoveProvTitle: '移除提供商 {p}？',
+      confirmRemoveProvDesc: '确定要从轮换中移除该提供商及其所有已配置密钥吗？',
+      filterAll: '全部',
+      filterReady: '就绪',
+      filterCooldown: '冷却中',
+      filterErrors: '存在错误',
+      subtitle: '按提供商 API 密钥轮换：密钥池、配额/限流错误自动故障转移、冷却与恢复。',
+      cardDesc: '按提供商 API 密钥轮换：密钥池、配额/限流错误自动故障转移、冷却与恢复。',
+      loading: '加载中…',
+      notRegistered: '未注册',
+      removeKey: '移除密钥',
+      removeProvider: '移除提供商',
+      addKey: '+ 添加密钥',
+      addKeyTitle: '添加 API 密钥',
+      noProviders: 'DSH 尚未注册任何提供商 — 暂无可选项目。',
+      addProvider: '+ 添加提供商',
+      desc: '按提供商的 API 密钥轮换。为每个提供商列出其 API 密钥（环境变量名，存储在 DSH 凭据中）。插件按顺序通过提供商的密钥路由模型，并在配额/速率限制失败时切换到下一个。',
+      cooldown: '故障后冷却时间 (毫秒)',
+      scheduleDays: '轮换计划 (天数，0=关闭)',
+      switchCodes: '切换代码 (逗号分隔)',
+      providersTitle: '提供商与其密钥',
+      save: '保存',
+      discard: '放弃',
+      saving: '保存中…',
+      moveUp: '上移',
+      moveDown: '下移',
+      keyActive: '使用中',
+      keyReady: '就绪',
+      keyCooling: '冷却中，{s}秒',
+      keyMissing: '凭证不存在',
+      switchesNone: '暂无切换',
+      switchesSome: '切换次数: {n} · 上次: {reason}, {ago}',
+      justNow: '刚刚',
+      minutesAgo: '{n} 分钟前',
+      hoursAgo: '{n} 小时前',
+      codesTitle: '在这些故障时切换',
+      keyValuePlaceholder: '粘贴密钥，然后保存',
+      keySave: '保存密钥',
+      keySaved: '已保存',
+      keyFromEnv: '来自环境变量，此处只读',
+      keyWriteFailed: '无法存储密钥: {msg}',
+      notSecretShape: '已保存，但该值看起来不像 API 密钥 - 请检查是否有拼写错误',
+      rpmTitle: '请求/分钟: 已用 {u}, 剩余 {r}',
+      budgetLabel: '预算:',
+      exportCsv: '导出 CSV',
+      weightHint: '轮换权重：该密钥参与循环的次数 (1 = 等分)',
+      retestBroken: '重新测试',
+      retestFail: '重新测试失败 - 密钥仍然不可用',
+      snapshotExport: '快照 ⬇',
+      snapshotImport: '恢复',
+      keyHint: '该值存储在 DSH 凭据中，绝不会发送回浏览器 — 仅显示其最后 5 个字符。名称已自动生成；悬停在密钥上可查看其使用的名称。',
+      brokenKey: '已损坏 (3× 认证失败)',
+      keyExpired: '已过期',
+      keyExpiringSoon: '{n} 天后过期',
+      exportPools: '导出',
+      exportOne: '⬇',
+      usedAgo: '{ago}前使用',
+      importPools: '导入',
+      importEnv: '导入 .env',
+      resetCooldown: '重置冷却',
+      testAll: '测试所有密钥',
+      testing: '测试中…',
+      testKey: '测试',
+      testOk: '正常',
+      testFail: '失败',
+      poolExhausted: '密钥池耗尽 — 所有密钥冷却中',
+      resetting: '重置中…',
+      keyLabel: '密钥 {n}',
+      undoProvider: '提供商已移除',
+      undoKey: '密钥已移除',
+      undo: '撤销',
+      emptyTitle: '暂无密钥池',
+      emptyDesc: '添加提供商及至少一个 API 密钥名称以开始轮换。',
+      errorTitle: '无法加载设置',
+      retry: '重试',
+      unavailableTitle: '设置桥接不可用',
+      unavailableDesc: '宿主未暴露 dsh-key-rotation 设置命名空间。请重新打开设置或检查插件是否已启用。',
+      modalClose: '关闭对话框',
+      dialogLabel: '确认',
+      loadChartAria: '密钥流量分布',
+      bulkCooldownPlaceholder: '批量冷却时间 (毫秒)',
+      bulkApply: '应用到所选项',
+      bulkRemove: '删除所选项',
+      confirmBulkRemoveTitle: '删除 {n} 个提供商？',
+      confirmBulkRemoveDesc: '所选提供商及其所有密钥将从轮换中移除。操作后可立即撤销一次。',
+      selectProvider: '选择提供商 {p}',
+      activeBadge: '活跃',
+      healthScoreTitle: '健康评分',
+      totalRequestsTitle: '总请求数',
+      exportProvider: '导出此提供商',
+      recentFailures: '近期故障 ({n})',
+      noActivePools: '无活跃密钥池',
+      headerPoolsTitle: '密钥轮换池',
+      pausedBudget: '已暂停',
+      sortUsage: '按使用量排序',
+      testingAllProgress: '正在测试 {n}/{total}…',
+      liveEventStream: '实时事件流',
+      liveEventStreamEmpty: '暂无轮换事件记录',
+      eventAt: '时间',
+      eventKey: '密钥',
+      eventReason: '原因',
+      eventCooldown: '冷却',
+      quotaResetIn: '{time} 后重置',
+      routingStrategyLabel: '路由策略',
+      routingStrategyRoundRobin: '轮询 (默认)',
+      routingStrategyLeastLoaded: '最低负载 (并发)',
+      routingStrategyLowestLatency: '最低延迟 (p95)',
+      proactiveGuardLabel: '前瞻性限流防护',
+      proactiveGuardHint: '当响应中的剩余配额或 Retry-After 头提示即将超限时，预先冷却密钥以防触发 429',
+      selfHealingLabel: '自愈检查间隔 (分钟)',
+      selfHealingHint: '定期轻量探测以自动恢复损坏密钥 (0 为禁用)',
     };
 
     // Коды, на которых имеет смысл переключать ключ. Список из хоста
@@ -315,6 +469,9 @@ window.__ModuleLoader__.load({
       '.krot-pill-err{border-color:rgba(239,68,68,0.3);color:var(--dsw-alias-state-error-primary)}',
       '.krot-alert-ok{padding:10px 14px;border-radius:8px;background:rgba(16,185,129,0.1);color:var(--dsw-alias-state-success-primary);font-size:13px;display:flex;align-items:center;gap:10px}',
       '.krot-alert-bad{padding:10px 14px;border-radius:8px;background:rgba(239,68,68,0.1);color:var(--dsw-alias-state-error-primary);font-size:13px;display:flex;align-items:center;gap:10px}',
+      '.krot-event-stream{display:flex;flex-direction:column;gap:6px;margin-top:8px;padding:10px 12px;background:var(--dsw-alias-bg-layer-3);border:1px solid var(--dsw-alias-border-l2);border-radius:8px}',
+      '.krot-event-row{display:flex;align-items:center;gap:8px;font-size:12px;padding:4px 0;border-bottom:1px solid var(--dsw-alias-border-l1)}',
+      '.krot-event-time{font-family:ui-monospace,Menlo,Consolas,monospace;color:var(--dsw-alias-label-tertiary);font-size:11px}',
       '.krot-card{border:1px solid var(--dsw-alias-border-l2);background:var(--dsw-alias-bg-layer-3);border-radius:12px;list-style:none}',
       '.krot-card-header{appearance:none;width:100%;font:inherit;color:inherit;text-align:left;cursor:pointer;background:0 0;border:0;border-radius:12px;display:flex;align-items:center;gap:12px;padding:14px 16px}',
       '.krot-card-head-text{display:flex;flex-direction:column;flex:1;gap:4px;min-width:0}',
@@ -500,6 +657,7 @@ window.__ModuleLoader__.load({
       const [testing, setTesting] = React.useState('');
       const [testResult, setTestResult] = React.useState({});
       const [testAllProvider, setTestAllProvider] = React.useState('');
+      const [testingProgress, setTestingProgress] = React.useState({});
       const [secretDraft, setSecretDraft] = React.useState({});
       const [secretError, setSecretError] = React.useState('');
       const stashUndo = (u) => { setUndo(u); if (undoTimer.current) clearTimeout(undoTimer.current); undoTimer.current = setTimeout(() => setUndo(null), 5000); };
@@ -1124,6 +1282,24 @@ window.__ModuleLoader__.load({
               return h('span', { className: 'krot-tail', title: t('healthScoreTitle'), style: { flex: 'none', color, fontWeight: 700 } }, String(score));
             })(),
             (() => { const ps = status[entry.provider]; const tot = ps && typeof ps.totalUsage === 'number' ? ps.totalUsage : null; return tot !== null ? h('span', { className: 'krot-tail', title: t('totalRequestsTitle'), style: { flex: 'none' } }, String(tot)) : null; })(),
+            (() => {
+              const q = status.quota || (providerStatus && providerStatus.quota);
+              let nearestReset = null;
+              if (q && typeof q === 'object') {
+                for (const k of (entry.keys || [])) {
+                  const entryQ = q[k];
+                  if (entryQ && typeof entryQ.reset === 'number' && entryQ.reset > Date.now()) {
+                    if (!nearestReset || entryQ.reset < nearestReset) nearestReset = entryQ.reset;
+                  }
+                }
+              }
+              if (!nearestReset) return null;
+              const diffMs = nearestReset - Date.now();
+              if (diffMs <= 0) return null;
+              const mins = Math.ceil(diffMs / 60000);
+              const timeStr = mins > 60 ? Math.floor(mins / 60) + 'h ' + (mins % 60) + 'm' : mins + 'm';
+              return h('span', { className: 'krot-badge krot-badge-warn', title: 'Quota window reset' }, t('quotaResetIn').replace('{time}', timeStr));
+            })(),
             btn('✕', () => setConfirmModal({
               title: t('confirmRemoveProvTitle').replace('{p}', entry.provider),
               desc: t('confirmRemoveProvDesc'),
@@ -1144,7 +1320,26 @@ window.__ModuleLoader__.load({
             (providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('div', { style: { display: 'flex', gap: '2px', alignItems: 'end', height: '24px', marginTop: '4px' } }, (() => { const now = Date.now(); const buckets = Array(24).fill(0); for (const ev of providerStatus.events) { const h = Math.floor((now - ev.at) / 3600000); if (h >= 0 && h < 24) buckets[23 - h]++; } const max = Math.max(1, ...buckets); return buckets.map((c, i) => h('div', { key: i, title: c + ' switches', style: { flex: 1, background: c ? 'var(--dsw-alias-state-warning-primary)' : 'var(--dsw-alias-border-l2)', height: (c / max * 24) + 'px', minHeight: '2px', borderRadius: '2px' } })); })()) : null),
             // #224: 7-day switches per day (client-side, from the same events)
             (providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('div', { style: { display: 'flex', gap: '2px', alignItems: 'end', height: '16px', marginTop: '2px' } }, (() => { const now = Date.now(); const days = Array(7).fill(0); for (const ev of providerStatus.events) { const d = Math.floor((now - ev.at) / 86400000); if (d >= 0 && d < 7) days[6 - d]++; } const max = Math.max(1, ...days); return days.map((c, i) => h('div', { key: i, title: c + ' switches · day -' + (6 - i), style: { flex: 1, background: c ? 'var(--dsw-alias-state-info-primary, var(--dsw-alias-state-warning-primary))' : 'var(--dsw-alias-border-l2)', height: (c / max * 16) + 'px', minHeight: '2px', borderRadius: '2px' } })); })()) : null),
-            (providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('details', { style: { fontSize: '11px', marginTop: '6px' } }, h('summary', null, t('recentFailures').replace('{n}', String(providerStatus.events.length))), h('ul', { style: { margin: '4px 0 0', paddingLeft: '16px' } }, providerStatus.events.slice().reverse().map((ev, i) => h('li', { key: i, style: ev.type === 'probe' ? { opacity: .5 } : null }, new Date(ev.at).toLocaleTimeString() + ' ' + (ev.type === 'probe' ? '[probe] ' : '') + ev.ref + ' ' + ev.reason + ' cd=' + ev.cooldownMs)) )) : null),
+            (providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('details', { className: 'krot-event-stream' },
+              h('summary', { style: { display: 'flex', alignItems: 'center', gap: '8px', cursor: 'pointer', fontWeight: 600, fontSize: '12px' } },
+                h('span', null, '📜 ' + t('liveEventStream')),
+                h('span', { className: 'krot-badge' }, String(providerStatus.events.length))
+              ),
+              h('div', { style: { display: 'flex', flexDirection: 'column', gap: '4px', marginTop: '8px' } },
+                providerStatus.events.slice().reverse().map((ev, i) => {
+                  const isHeal = ev.type === 'heal' || ev.reason === 'auto-unbreak' || ev.reason === 'self-heal';
+                  const isBad = ev.code === 'AUTH' || ev.reason === 'AUTH' || ev.type === 'broken';
+                  const badgeClass = isHeal ? 'krot-badge-ok' : isBad ? 'krot-badge-bad' : 'krot-badge-warn';
+                  const timeStr = ev.at ? new Date(ev.at).toLocaleTimeString() : '';
+                  return h('div', { key: i, className: 'krot-event-row' },
+                    h('span', { className: 'krot-event-time' }, timeStr),
+                    h('span', { className: 'krot-tail' }, ev.ref || 'pool'),
+                    h('span', { className: 'krot-badge ' + badgeClass }, String(ev.reason || ev.code || ev.type)),
+                    ev.cooldownMs > 0 ? h('span', { style: { fontSize: '11px', color: 'var(--dsw-alias-label-tertiary)' } }, 'cd: ' + Math.round(ev.cooldownMs / 1000) + 's') : null
+                  );
+                })
+              )
+            ) : null),
             btn(t('resetCooldown'), () => setConfirmModal({
               title: t('confirmResetTitle').replace('{p}', entry.provider),
               desc: t('confirmResetDesc'),
@@ -1152,7 +1347,7 @@ window.__ModuleLoader__.load({
               danger: false,
               onConfirm: () => doReset(entry.provider),
             }), { disabled: !(providerStatus && providerStatus.switches > 0) || resetting === entry.provider, title: t('resetCooldown') }),
-            btn(testAllProvider === entry.provider ? t('testing') : t('testAll'), () => doTestAll(entry.provider), { disabled: testAllProvider === entry.provider, title: t('testAll') }),
+            btn(testAllProvider === entry.provider ? (testingProgress[entry.provider] ? t('testingAllProgress').replace('{n}', testingProgress[entry.provider].split('/')[0]).replace('{total}', testingProgress[entry.provider].split('/')[1]) : t('testing')) : t('testAll'), () => doTestAll(entry.provider), { disabled: testAllProvider === entry.provider, title: t('testAll') }),
             exportCsv,
           ),
         );
@@ -1276,6 +1471,26 @@ window.__ModuleLoader__.load({
         h('div', { className: 'krot-grid-2' },
           field(t('cooldown'), textInput(String(val.cooldownMs ?? 60000), (v) => setField((cur) => ({ ...cur, cooldownMs: Number(v) || 0 })))),
           field(t('scheduleDays'), textInput(String(val.rotationScheduleDays ?? 0), (v) => setField((cur) => ({ ...cur, rotationScheduleDays: Number(v) || 0 }))))
+        ),
+        h('div', { className: 'krot-grid-3', style: { marginTop: '10px' } },
+          field(t('routingStrategyLabel'), h('select', {
+            className: 'krot-in',
+            value: val.routingStrategy ?? 'round-robin',
+            onChange: (e) => setField((cur) => ({ ...cur, routingStrategy: e.target.value })),
+          }, [
+            h('option', { key: 'round-robin', value: 'round-robin' }, t('routingStrategyRoundRobin')),
+            h('option', { key: 'least-loaded', value: 'least-loaded' }, t('routingStrategyLeastLoaded')),
+            h('option', { key: 'lowest-latency', value: 'lowest-latency' }, t('routingStrategyLowestLatency')),
+          ])),
+          field(t('selfHealingLabel'), textInput(String(val.selfHealingIntervalMinutes ?? 30), (v) => setField((cur) => ({ ...cur, selfHealingIntervalMinutes: Number(v) || 0 })))),
+          field(t('proactiveGuardLabel'), h('label', { style: { display: 'flex', alignItems: 'center', gap: '8px', height: '36px', cursor: 'pointer' } }, [
+            h('input', {
+              type: 'checkbox',
+              checked: val.proactiveRateLimitGuard ?? true,
+              onChange: (e) => setField((cur) => ({ ...cur, proactiveRateLimitGuard: e.target.checked })),
+            }),
+            h('span', { style: { fontSize: '13px', color: 'var(--dsw-alias-label-secondary)' } }, t('proactiveGuardLabel')),
+          ]))
         )
       );
 
@@ -1475,7 +1690,7 @@ window.__ModuleLoader__.load({
     }
 
     function apply(ctx) {
-      ctx.effect(() => ctx.locale.register(NS, { en }), 'dsh-key-rotation: dictionaries');
+      ctx.effect(() => ctx.locale.register(NS, { en, zh }), 'dsh-key-rotation: dictionaries');
       // Header chip (#201): status dot in the session header utilities slot,
       // same slot dsh-gitea / dsh-subscriptions use for their header widgets.
       ctx.effect(() => {

package/lib/concurrency.js

@@ -40,6 +40,15 @@ export class ConcurrencyTracker {
     e.lastAcquired = now;
   }
 
+  getActive(ref, now) {
+    now = now || Date.now();
+    if (!ref) return 0;
+    const e = this._inFlight.get(ref);
+    if (!e) return 0;
+    if (now - e.lastAcquired > this._staleMs) return 0;
+    return e.count;
+  }
+
   snapshot() {
     const out = {};
     for (const [k, v] of this._inFlight) out[k] = { count: v.count };

package/lib/heal.js

@@ -33,3 +33,50 @@ export function healIdleCooldowns(pools, idleMs, now = Date.now()) {
   }
   return healed;
 }
+
+/**
+ * Background auto-recovery for keys in brokenUntil status.
+ * Tests broken keys with a lightweight free probe (e.g. probeModels).
+ * If the probe succeeds, clears broken and failure states and records a 'heal' event.
+ *
+ * @param {Array} pools
+ * @param {Function} probeFn async (ref) => { ok: boolean, code?: string, status?: number }
+ * @param {number} [now=Date.now()]
+ * @returns {Promise<Array<{ ref: string, poolBase: string, ok: boolean }>>}
+ */
+export async function autoUnbreakBrokenKeys(pools, probeFn, now = Date.now()) {
+  if (!Array.isArray(pools) || pools.length === 0 || typeof probeFn !== 'function') return [];
+  const results = [];
+  for (const pool of pools) {
+    if (!pool || !pool.state || !pool.base) continue;
+    const bu = pool.state.brokenUntil;
+    if (!bu || bu.size === 0) continue;
+    const brokenRefs = [];
+    for (const [ref, until] of bu.entries()) {
+      if (Number.isFinite(until) && until > now) {
+        brokenRefs.push(ref);
+      }
+    }
+    for (const ref of brokenRefs) {
+      try {
+        const res = await probeFn(ref);
+        if (res && res.ok) {
+          bu.delete(ref);
+          pool.state.failedUntil?.delete(ref);
+          pool.state.failCounts?.delete(ref);
+          pool.state.authFailCounts?.delete(ref);
+          if (Array.isArray(pool.state.events)) {
+            pool.state.events.push({ at: now, ref, reason: 'auto-unbreak', cooldownMs: 0, type: 'heal' });
+            if (pool.state.events.length > 50) pool.state.events.shift();
+          }
+          results.push({ ref, poolBase: pool.base, ok: true });
+        } else {
+          results.push({ ref, poolBase: pool.base, ok: false });
+        }
+      } catch (_) {
+        results.push({ ref, poolBase: pool.base, ok: false });
+      }
+    }
+  }
+  return results;
+}

package/lib/index.js

@@ -27,8 +27,9 @@ const HEALTH_PATH = '/dsh-key-rotation/health';
 const USAGE_PATH = '/dsh-key-rotation/usage';
 const TEST_PATH = '/dsh-key-rotation/test';
 const SANDBOX_CACHE_PATH = '/dsh-key-rotation/sandbox-cache';
+import { sortAttemptList } from './pool.js';
 import { LastTestCache, SandboxRunner } from './sandbox.js';
-import { healIdleCooldowns } from './heal.js';
+import { healIdleCooldowns, autoUnbreakBrokenKeys } from './heal.js';
 import { LatencyHistogram } from './histogram.js';
 import { pickCascadeFallback } from './cascade.js';
 import { ConcurrencyTracker } from './concurrency.js';
@@ -199,6 +200,9 @@ export const Config = Schema.object({
     hour: Schema.number().default(0),
   }),
   rateLimitThreshold: Schema.number().default(0.1),
+  proactiveRateLimitGuard: Schema.boolean().default(true),
+  selfHealingIntervalMinutes: Schema.number().default(30),
+  routingStrategy: Schema.union(['round-robin', 'least-loaded', 'lowest-latency']).default('round-robin'),
   rpmLimit: Schema.number().default(0),
   webhookActionToken: Schema.string().role('secret').default(''),
   expiryWarnDays: Schema.number().default(7),
@@ -230,6 +234,8 @@ export const Config = Schema.object({
     })).default({}),
     cooldownMs: Schema.number(),
     maxCooldownMs: Schema.number(),
+    routingStrategy: Schema.union(['round-robin', 'least-loaded', 'lowest-latency']),
+    proactiveRateLimitGuard: Schema.boolean(),
   })).default([...DEFAULT_PROVIDERS]),
 });
 
@@ -339,6 +345,29 @@ export function apply(ctx, config = {}) {
     if (snap) statePersistence.save(snap);
   }
 
+    // Auto-healing / auto-unbreak for broken keys (#303)
+  ctx.effect(() => {
+    const cfg = getConfig();
+    const intervalMin = cfg?.selfHealingIntervalMinutes ?? 30;
+    if (!intervalMin || intervalMin <= 0) return () => {};
+    const intervalMs = intervalMin * 60 * 1000;
+    const timer = setInterval(async () => {
+      try {
+        const c = getConfig();
+        if (!c || !c.selfHealingIntervalMinutes || c.selfHealingIntervalMinutes <= 0) return;
+        const { pools } = buildRuntime();
+        const runner = ensureSandboxRunner(ctx);
+        await autoUnbreakBrokenKeys(pools, async (ref) => {
+          let val = (await ctx.credentials?.resolve?.(ref))?.value;
+          if (!val) return { ok: false };
+          return runner.probeModels(ref, val);
+        });
+      } catch (_) {}
+    }, intervalMs);
+    if (typeof timer.unref === 'function') timer.unref();
+    return () => clearInterval(timer);
+  }, 'dsh-key-rotation: auto-unbreak');
+
   ctx.effect(() => {
     const timer = setInterval(() => { try { schedulePersist(); } catch (_) {} }, 15000);
     if (typeof timer.unref === 'function') timer.unref();
@@ -372,9 +401,10 @@ export function apply(ctx, config = {}) {
           }
         }
       }
-      const n = sweepExpired(poolState, now);
-      if (n > 0) console.warn(`[dsh-key-rotation] sweep: cleared ${n} expired cooldown(s)`);
       const runtime = buildRuntime();
+      const allActiveRefs = Array.from(runtime.poolByRef.keys());
+      const n = sweepExpired(poolState, now, allActiveRefs);
+      if (n > 0) console.warn(`[dsh-key-rotation] sweep: cleared ${n} expired cooldown(s)`);
       for (const pool of runtime.poolByRef.values()) {
         compactUsage(pool, 30, now);
       }
@@ -564,7 +594,7 @@ export function apply(ctx, config = {}) {
       if (typeof v === 'string' && v.length > 0) { const ts = Date.parse(v); return Number.isNaN(ts) ? undefined : ts; }
       return undefined;
     };
-    const buildPool = (base, keys, weights, poolCooldown, poolMax, expiresAt) => {
+    const buildPool = (base, keys, weights, poolCooldown, poolMax, expiresAt, poolStrategy, poolGuard) => {
       const refs = (keys ?? []).filter((ref) => typeof ref === 'string' && ref.length > 0);
       if (refs.length === 0) return null;
       const w = Array.isArray(weights) ? weights : [];
@@ -582,7 +612,7 @@ export function apply(ctx, config = {}) {
       }
       return { base, refs, weights: refs.map((_, i) => (typeof w[i] === 'number' && w[i] > 0 ? Math.floor(w[i]) : 1)),
                weightedRefs: weightedRefs.length > 0 ? weightedRefs : refs,
-               state: makeState(base), cooldownMs: poolCooldown, maxCooldownMs: poolMax, expiresAt: parsedExpiry, rpmLimit };
+               state: makeState(base), cooldownMs: poolCooldown, maxCooldownMs: poolMax, expiresAt: parsedExpiry, rpmLimit, routingStrategy: poolStrategy, proactiveRateLimitGuard: poolGuard };
     };
     for (const p of cfg.providers ?? []) {
       const poolCooldown = typeof p.cooldownMs === 'number' ? p.cooldownMs : (cfg.cooldownMs ?? 60000);
@@ -656,7 +686,10 @@ export function apply(ctx, config = {}) {
         }
       }
     }
-    cachedRuntime = { switchCodes, cooldownMs, maxCooldownMs, notifyWebhook, notifyThreshold, concurrencyLimit, cascade, quotaResetWindow, rateLimitThreshold, rpmLimit, webhookActionToken, expiryWarnDays: cfg.expiryWarnDays ?? 7, switchNotify: cfg.switchNotify ?? false, verboseLogging: cfg.verboseLogging ?? false, switchNotifyThrottleMs: cfg.switchNotifyThrottleMs ?? 60000, warnBelowHealthy: cfg.warnBelowHealthy ?? 0, latencySloMs: cfg.latencySloMs ?? 0, providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider, cloneIds, expectedClones,
+    cachedRuntime = { switchCodes, cooldownMs, maxCooldownMs, notifyWebhook, notifyThreshold, concurrencyLimit, cascade, quotaResetWindow, rateLimitThreshold, rpmLimit, webhookActionToken, expiryWarnDays: cfg.expiryWarnDays ?? 7, switchNotify: cfg.switchNotify ?? false, verboseLogging: cfg.verboseLogging ?? false,
+      proactiveRateLimitGuard: cfg.proactiveRateLimitGuard ?? true,
+      selfHealingIntervalMinutes: cfg.selfHealingIntervalMinutes ?? 30,
+      routingStrategy: cfg.routingStrategy ?? 'round-robin', switchNotifyThrottleMs: cfg.switchNotifyThrottleMs ?? 60000, warnBelowHealthy: cfg.warnBelowHealthy ?? 0, latencySloMs: cfg.latencySloMs ?? 0, providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider, cloneIds, expectedClones,
       circuitBreakerEnabled: cfg.circuitBreakerEnabled ?? true,
       circuitBreakerThreshold: cfg.circuitBreakerThreshold ?? 5,
       circuitBreakerOpenMs: cfg.circuitBreakerOpenMs ?? 30000,
@@ -684,8 +717,12 @@ export function apply(ctx, config = {}) {
       const pool = poolByRef.get(ref);
       if (!pool) return original(ref);
       const now = Date.now();
-      const list = pool.weightedRefs ?? pool.refs;
-      const start = pool.state.pointer ?? 0;
+      const strat = pool.routingStrategy ?? buildRuntime().routingStrategy ?? 'round-robin';
+      let list = pool.weightedRefs ?? pool.refs;
+      if (strat === 'lowest-latency' || strat === 'least-loaded') {
+        list = sortAttemptList(list, strat, { latencyHistogram, concurrencyTracker });
+      }
+      const start = (strat === 'round-robin') ? (pool.state.pointer ?? 0) : 0;
       for (let i = 0; i < list.length; i++) {
         const index = (start + i) % list.length;
         const candidate = list[index];
@@ -805,6 +842,7 @@ export function apply(ctx, config = {}) {
     notifySwitch: (runtime, pool, info) => notifySwitch(runtime, pool, info, { webhookSender, notifyQueue: moduleNotifyQueue, now: () => Date.now() }),
     notifyExhaustion,
     recordLatency,
+    latencyHistogram,
     concurrencyTracker,
     MARKER,
     finishError,

package/lib/pool.js

@@ -184,9 +184,10 @@ export function envValue(ref) {
   return typeof v === 'string' && v.length > 0 ? v : undefined;
 }
 
-/** Sweep expired cooldown entries from poolState. Returns count of cleared refs. */
-export function sweepExpired(poolState, now = Date.now()) {
+/** Sweep expired cooldown entries and prune stale deleted refs from poolState. Returns count of cleared refs. */
+export function sweepExpired(poolState, now = Date.now(), activeRefs = null) {
   let cleared = 0;
+  const activeSet = activeRefs ? new Set(activeRefs) : null;
   for (const st of poolState.values()) {
     for (const [ref, until] of [...st.failedUntil.entries()]) {
       if (until <= now) {
@@ -195,6 +196,16 @@ export function sweepExpired(poolState, now = Date.now()) {
         cleared++;
       }
     }
+    // Prune stale entries for keys no longer in active configuration
+    if (activeSet) {
+      for (const map of [st.failedUntil, st.failCounts, st.authFailCounts, st.brokenUntil, st.costPerKey, st.lastUsedAt, st.usageCounts, st.byModel, st.usageDays, st.quotaWindows, st.rpmWindows, st.probedAt]) {
+        if (map && typeof map.keys === 'function') {
+          for (const k of [...map.keys()]) {
+            if (!activeSet.has(k)) map.delete(k);
+          }
+        }
+      }
+    }
   }
   return cleared;
 }
@@ -248,33 +259,105 @@ export function computeHealthScore(state) {
 /** Extract rate-limit info from an object that may carry response headers. */
 export function extractRateLimit(headers) {
   if (!headers || typeof headers !== 'object') return null;
-  let remaining, limit, reset;
+  let remaining, limit, reset, retryAfter;
   for (const k in headers) {
-    const lk = k.length;
-    if (lk === 21 || lk === 16 || lk === 17) {
-      const lower = k.toLowerCase();
-      if (lower === 'x-ratelimit-remaining') {
-        const v = headers[k];
-        if (v != null) remaining = Number(String(v));
-      } else if (lower === 'x-ratelimit-limit') {
-        const v = headers[k];
-        if (v != null) limit = Number(String(v));
-      } else if (lower === 'x-ratelimit-reset') {
-        const v = headers[k];
-        if (v != null) reset = Number(String(v));
+    const lower = k.toLowerCase();
+    const v = headers[k];
+    if (v == null) continue;
+    if (lower === 'retry-after') {
+      const num = Number(v);
+      if (Number.isFinite(num) && num >= 0) {
+        retryAfter = Math.ceil(num);
+      } else {
+        const parsed = Date.parse(v);
+        if (!Number.isNaN(parsed)) {
+          retryAfter = Math.max(0, Math.ceil((parsed - Date.now()) / 1000));
+        }
+      }
+    } else if (
+      lower === 'x-ratelimit-remaining' ||
+      lower === 'x-ratelimit-remaining-requests' ||
+      lower === 'anthropic-ratelimit-requests-remaining' ||
+      lower === 'openai-ratelimit-remaining-requests' ||
+      lower === 'ratelimit-remaining'
+    ) {
+      const n = Number(String(v));
+      if (Number.isFinite(n)) remaining = n;
+    } else if (
+      (lower === 'x-ratelimit-remaining-tokens' || lower === 'anthropic-ratelimit-tokens-remaining') &&
+      remaining === undefined
+    ) {
+      const n = Number(String(v));
+      if (Number.isFinite(n)) remaining = n;
+    } else if (
+      lower === 'x-ratelimit-limit' ||
+      lower === 'x-ratelimit-limit-requests' ||
+      lower === 'anthropic-ratelimit-requests-limit' ||
+      lower === 'openai-ratelimit-limit-requests' ||
+      lower === 'ratelimit-limit'
+    ) {
+      const n = Number(String(v));
+      if (Number.isFinite(n)) limit = n;
+    } else if (
+      lower === 'x-ratelimit-reset' ||
+      lower === 'x-ratelimit-reset-requests' ||
+      lower === 'anthropic-ratelimit-requests-reset' ||
+      lower === 'ratelimit-reset'
+    ) {
+      const n = Number(String(v));
+      if (Number.isFinite(n)) {
+        reset = n;
+      } else {
+        const parsed = Date.parse(v);
+        if (!Number.isNaN(parsed)) reset = parsed;
       }
     }
   }
-  if (remaining === undefined && limit === undefined) return null;
-  return { remaining, limit, reset };
+  if (remaining === undefined && limit === undefined && retryAfter === undefined && reset === undefined) return null;
+  const out = { remaining, limit, reset };
+  if (retryAfter !== undefined) out.retryAfter = retryAfter;
+  return out;
 }
 
-/** True if remaining is below the given threshold fraction of limit (e.g. 0.1). */
+/** True if remaining is below the given threshold fraction of limit (e.g. 0.1) or retry-after is active. */
 export function isRateLimited(rate, threshold = 0.1) {
   if (!rate) return false;
-  if (rate.remaining === undefined) return false;
-  if (rate.limit && rate.limit > 0) return rate.remaining < rate.limit * threshold;
-  return rate.remaining <= 0;
+  if (typeof rate.retryAfter === 'number' && rate.retryAfter > 0) return true;
+  if (rate.remaining !== undefined && rate.remaining <= 1) return true;
+  if (rate.limit && rate.limit > 0 && rate.remaining !== undefined) {
+    return rate.remaining < rate.limit * threshold;
+  }
+  if (rate.remaining !== undefined) return rate.remaining <= 0;
+  return false;
+}
+
+/**
+ * Sort or prioritize an attempt list of refs according to routing strategy.
+ * Strategies:
+ * - 'round-robin': preserves order
+ * - 'least-loaded': keys with lowest active concurrency count first
+ * - 'lowest-latency': keys with lowest p95 (or avg) latency first; unsampled keys neutral
+ */
+export function sortAttemptList(refs, strategy = 'round-robin', deps = {}) {
+  if (!Array.isArray(refs) || refs.length <= 1) return (refs ?? []).slice();
+  const list = refs.slice();
+  if (strategy === 'least-loaded' && deps.concurrencyTracker && typeof deps.concurrencyTracker.getActive === 'function') {
+    return list.sort((a, b) => {
+      const ca = deps.concurrencyTracker.getActive(a) ?? 0;
+      const cb = deps.concurrencyTracker.getActive(b) ?? 0;
+      return ca - cb;
+    });
+  }
+  if (strategy === 'lowest-latency' && deps.latencyHistogram && typeof deps.latencyHistogram.snapshot === 'function') {
+    return list.sort((a, b) => {
+      const sa = deps.latencyHistogram.snapshot(a);
+      const sb = deps.latencyHistogram.snapshot(b);
+      const la = (sa && Number.isFinite(sa.p95)) ? sa.p95 : ((sa && Number.isFinite(sa.avg)) ? sa.avg : 500);
+      const lb = (sb && Number.isFinite(sb.p95)) ? sb.p95 : ((sb && Number.isFinite(sb.avg)) ? sb.avg : 500);
+      return la - lb;
+    });
+  }
+  return list;
 }
 
 /**

package/lib/rotate.js

@@ -9,6 +9,7 @@ import {
   extractRateLimit,
   isRateLimited,
   formatExhaustionMessage,
+  sortAttemptList,
 } from './pool.js';
 import { pickCascadeFallback } from './cascade.js';
 
@@ -25,6 +26,7 @@ export function createRotate(deps) {
     notifySwitch,
     notifyExhaustion,
     recordLatency,
+    latencyHistogram,
     concurrencyTracker,
     MARKER,
     finishError,
@@ -42,7 +44,11 @@ export function createRotate(deps) {
       const reqStore = { pool, pickedRef: undefined, startMs: now() };
       setRotateStartMs(reqStore.startMs);
       let attemptList = (pool.weightedRefs ?? pool.refs).slice();
-      if (runtime0.concurrencyLimit > 0 && concurrencyTracker.isEnabled()) {
+      const strategy = pool.routingStrategy ?? runtime0.routingStrategy ?? 'round-robin';
+      if (strategy !== 'round-robin') {
+        attemptList = sortAttemptList(attemptList, strategy, { latencyHistogram, concurrencyTracker });
+      }
+      if (strategy !== 'least-loaded' && runtime0.concurrencyLimit > 0 && concurrencyTracker.isEnabled()) {
         // #193: prefer least-loaded key within limit
         const available = attemptList.filter((r) => {
           const fu = pool.state.failedUntil.get(r) ?? 0;
@@ -115,122 +121,136 @@ export function createRotate(deps) {
         }
 
         const _pickedRef = reqStore.pickedRef ?? pool.state.lastUsed;
-        if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.acquire(_pickedRef);
+        const acquired = (_pickedRef && runtime0.concurrencyLimit > 0) ? concurrencyTracker.acquire(_pickedRef) : false;
         try {
-          for await (const chunk of inner) {
-            // Only actual content deltas lock the stream (no more rotation).
-            // Structural/metadata chunks (block-start/end, usage) do not.
-            if (chunk && (chunk.type === 'text-delta' || chunk.type === 'reasoning-delta' || chunk.type === 'tool-call-delta')) {
-              yielded = true;
-              yield chunk;
-              continue;
-            }
-            if (chunk && chunk.type === 'finish') {
-              const kind = chunk.reason?.kind;
-              const failure = chunk.reason?.failure;
-              const code = failure?.code;
-              const message = failure?.message ?? '';
-              const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
-              const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
-              const activeRef = reqStore.pickedRef ?? pool.state.lastUsed;
-              if (switchable) {
-                penalizeRef(activeRef, code ?? 'UNKNOWN', message);
-                pool.state.switches = (pool.state.switches ?? 0) + 1;
-                pool.state.lastReason = String(code ?? 'UNKNOWN');
-                pool.state.lastSwitchAt = now();
-                lastFailure = chunk;
-                console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
-                // #216: per-switch webhook (opt-in switchNotify), deduped per provider
-                if (switchNotify && activeRef) {
-                  notifySwitch(runtime0, pool, {
-                    provider: options.provider,
-                    from: activeRef,
-                    code: String(code ?? 'UNKNOWN'),
-                    at: pool.state.lastSwitchAt,
-                  });
-                }
-                switching = true;
-                break;
+          try {
+            for await (const chunk of inner) {
+              // Only actual content deltas lock the stream (no more rotation).
+              // Structural/metadata chunks (block-start/end, usage) do not.
+              if (chunk && (chunk.type === 'text-delta' || chunk.type === 'reasoning-delta' || chunk.type === 'tool-call-delta')) {
+                yielded = true;
+                yield chunk;
+                continue;
               }
-              // cost tracking if provider returns usage.cost
-              const todayIso = activeRef ? new Date().toISOString().slice(0, 10) : undefined;
-              if (chunk.usage?.cost != null && activeRef) {
-                const c = Number(chunk.usage.cost);
-                if (!isNaN(c)) {
-                  if (!pool.state.costPerKey) pool.state.costPerKey = new Map();
-                  pool.state.costPerKey.set(activeRef, (pool.state.costPerKey.get(activeRef) ?? 0) + c);
-                  // #208: cost per day per key (mirrors usageDays) for budget checks
-                  if (!pool.state.costDays) pool.state.costDays = new Map();
-                  const cMap = pool.state.costDays.get(activeRef) || new Map();
-                  cMap.set(todayIso, (cMap.get(todayIso) ?? 0) + c);
-                  pool.state.costDays.set(activeRef, cMap);
+              if (chunk && chunk.type === 'finish') {
+                const kind = chunk.reason?.kind;
+                const failure = chunk.reason?.failure;
+                const code = failure?.code;
+                const message = failure?.message ?? '';
+                const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
+                const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
+                const activeRef = reqStore.pickedRef ?? pool.state.lastUsed;
+                if (switchable) {
+                  penalizeRef(activeRef, code ?? 'UNKNOWN', message);
+                  pool.state.switches = (pool.state.switches ?? 0) + 1;
+                  pool.state.lastReason = String(code ?? 'UNKNOWN');
+                  pool.state.lastSwitchAt = now();
+                  lastFailure = chunk;
+                  console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
+                  // #216: per-switch webhook (opt-in switchNotify), deduped per provider
+                  if (switchNotify && activeRef) {
+                    notifySwitch(runtime0, pool, {
+                      provider: options.provider,
+                      from: activeRef,
+                      code: String(code ?? 'UNKNOWN'),
+                      at: pool.state.lastSwitchAt,
+                    });
+                  }
+                  switching = true;
+                  break;
                 }
-              }
-              // Usage by day (#119)
-              if (activeRef) {
-                if (!pool.state.usageDays) pool.state.usageDays = new Map();
-                const dayMap = pool.state.usageDays.get(activeRef) || new Map();
-                dayMap.set(todayIso, (dayMap.get(todayIso) ?? 0) + 1);
-                pool.state.usageDays.set(activeRef, dayMap);
-              }
-              // Per-model request detail (#121)
-              if (activeRef && options.model) {
-                if (!pool.state.byModel) pool.state.byModel = new Map();
-                let byRef = pool.state.byModel.get(activeRef);
-                if (!byRef) { byRef = new Map(); pool.state.byModel.set(activeRef, byRef); }
-                byRef.set(options.model, (byRef.get(options.model) ?? 0) + 1);
-              }
-              // Proactive rate-limit (#115): if response headers say this key is near
-              // its quota, cool it down so the NEXT request starts on a different key.
-              // We do NOT re-run this (already successful) request — that would double-send.
-              const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
-              if (rate && activeRef) {
-                if (isRateLimited(rate, rateLimitThreshold ?? 0.1)) {
-                  const cool = rate.reset && rate.reset > now() ? (rate.reset - now()) : pool.cooldownMs;
-                  recordFailure(pool, activeRef, now(), cool, pool.maxCooldownMs);
-                  pushEvent(pool, activeRef, 'RATE_LIMIT', cool);
-                  console.warn(`[dsh-key-rotation] ${options.provider}: key ${activeRef} near quota (remaining ${String(rate.remaining)}/${String(rate.limit)}) — next request will rotate`);
+                // cost tracking if provider returns usage.cost
+                const todayIso = activeRef ? new Date().toISOString().slice(0, 10) : undefined;
+                if (chunk.usage?.cost != null && activeRef) {
+                  const c = Number(chunk.usage.cost);
+                  if (!isNaN(c)) {
+                    if (!pool.state.costPerKey) pool.state.costPerKey = new Map();
+                    pool.state.costPerKey.set(activeRef, (pool.state.costPerKey.get(activeRef) ?? 0) + c);
+                    // #208: cost per day per key (mirrors usageDays) for budget checks
+                    if (!pool.state.costDays) pool.state.costDays = new Map();
+                    const cMap = pool.state.costDays.get(activeRef) || new Map();
+                    cMap.set(todayIso, (cMap.get(todayIso) ?? 0) + c);
+                    pool.state.costDays.set(activeRef, cMap);
+                  }
                 }
-              }
-              // #7: persist quota snapshot regardless of threshold (so dashboard widget can show it).
-              if (rate && activeRef && Number.isFinite(rate.remaining) && quotaStore) {
-                quotaStore.set(activeRef, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: now() });
+                // Usage by day (#119)
+                if (activeRef) {
+                  if (!pool.state.usageDays) pool.state.usageDays = new Map();
+                  const dayMap = pool.state.usageDays.get(activeRef) || new Map();
+                  dayMap.set(todayIso, (dayMap.get(todayIso) ?? 0) + 1);
+                  pool.state.usageDays.set(activeRef, dayMap);
+                }
+                // Per-model request detail (#121)
+                if (activeRef && options.model) {
+                  if (!pool.state.byModel) pool.state.byModel = new Map();
+                  let byRef = pool.state.byModel.get(activeRef);
+                  if (!byRef) { byRef = new Map(); pool.state.byModel.set(activeRef, byRef); }
+                  byRef.set(options.model, (byRef.get(options.model) ?? 0) + 1);
+                }
+                // Proactive rate-limit (#115, #303): if response headers say this key is near
+                // its quota or has retry-after, cool it down so the NEXT request starts on a different key.
+                // We do NOT re-run this (already successful) request — that would double-send.
+                const guardEnabled = pool.proactiveRateLimitGuard ?? runtime0.proactiveRateLimitGuard ?? true;
+                const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
+                if (guardEnabled && rate && activeRef) {
+                  if (isRateLimited(rate, rateLimitThreshold ?? 0.1)) {
+                    let cool;
+                    if (typeof rate.retryAfter === 'number' && rate.retryAfter > 0) {
+                      cool = rate.retryAfter * 1000;
+                    } else if (rate.reset && rate.reset > now()) {
+                      cool = rate.reset - now();
+                    } else {
+                      cool = pool.cooldownMs ?? cooldownMs;
+                    }
+                    const maxCool = pool.maxCooldownMs ?? maxCooldownMs;
+                    const effCool = Math.min(cool, maxCool ?? cool);
+                    recordFailure(pool, activeRef, now(), effCool, maxCool);
+                    pushEvent(pool, activeRef, 'RATE_LIMIT', effCool);
+                    console.warn(`[dsh-key-rotation] ${options.provider}: key ${activeRef} proactive pause (remaining ${String(rate.remaining ?? '?')}/${String(rate.limit ?? '?')}, cool ${Math.round(effCool / 1000)}s) — next request will rotate`);
+                  }
+                }
+                // #7: persist quota snapshot regardless of threshold (so dashboard widget can show it).
+                if (rate && activeRef && Number.isFinite(rate.remaining) && quotaStore) {
+                  quotaStore.set(activeRef, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: now() });
+                }
+                yield chunk;
+                recordLatency(pool, reqStore);
+                if (circuitBreaker) circuitBreaker.onSuccess(options.provider);
+                return;
               }
               yield chunk;
-              recordLatency(pool, reqStore);
-              if (circuitBreaker) circuitBreaker.onSuccess(options.provider);
-              return;
             }
-            yield chunk;
-          }
-        } catch (e) {
-          if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.release(_pickedRef);
-          const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
-          const activeRef = _pickedRef ?? reqStore.pickedRef ?? pool.state.lastUsed;
-          if (!yielded && isSwitchableError(e, effectiveSwitchCodes)) {
-            penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
-            pool.state.switches = (pool.state.switches ?? 0) + 1;
-            pool.state.lastReason = String(e?.code ?? 'TRANSPORT');
-            pool.state.lastSwitchAt = now();
-            lastFailure = finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
-            console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} stream threw ${String(e?.code ?? e?.message ?? e)} - failover to next key`);
-            if (switchNotify && activeRef) {
-              notifySwitch(runtime0, pool, {
-                provider: options.provider,
-                from: activeRef,
-                code: String(e?.code ?? 'TRANSPORT'),
-                at: pool.state.lastSwitchAt,
-              });
+          } catch (e) {
+            const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
+            const activeRef = _pickedRef ?? reqStore.pickedRef ?? pool.state.lastUsed;
+            if (!yielded && isSwitchableError(e, effectiveSwitchCodes)) {
+              penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+              pool.state.switches = (pool.state.switches ?? 0) + 1;
+              pool.state.lastReason = String(e?.code ?? 'TRANSPORT');
+              pool.state.lastSwitchAt = now();
+              lastFailure = finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+              console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} stream threw ${String(e?.code ?? e?.message ?? e)} - failover to next key`);
+              if (switchNotify && activeRef) {
+                notifySwitch(runtime0, pool, {
+                  provider: options.provider,
+                  from: activeRef,
+                  code: String(e?.code ?? 'TRANSPORT'),
+                  at: pool.state.lastSwitchAt,
+                });
+              }
+              continue; // Failover to next key!
             }
-            continue; // Failover to next key!
+            yield finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+            return;
           }
-          yield finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
-          return;
-        }
 
-        if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.release(_pickedRef);
-        if (switching) continue; // try the next key
-        return; // clean end — served
+          if (switching) continue; // try the next key
+          return; // clean end — served
+        } finally {
+          if (acquired && _pickedRef) {
+            concurrencyTracker.release(_pickedRef);
+          }
+        }
       }
 
       // pool exhausted — all keys cooling or missing

package/lib/routes-ops.js

@@ -163,6 +163,8 @@ export function registerOpsRoutes(ctx, deps) {
           budgetDaily: (providerBudgets?.get ? providerBudgets.get(pool.base) : providerBudgets?.[pool.base])?.costBudgetDaily ?? 0,
           budgetWeekly: (providerBudgets?.get ? providerBudgets.get(pool.base) : providerBudgets?.[pool.base])?.costBudgetWeekly ?? 0,
           pauseOnBudget: (providerBudgets?.get ? providerBudgets.get(pool.base) : providerBudgets?.[pool.base])?.pauseOnBudget ?? false,
+          routingStrategy: pool.routingStrategy ?? runtime.routingStrategy ?? 'round-robin',
+          proactiveRateLimitGuard: pool.proactiveRateLimitGuard ?? runtime.proactiveRateLimitGuard ?? true,
         });
         } catch (e) {
           console.warn(`[dsh-key-rotation] status: pool ${pool.base} failed: ${String(e?.message ?? e)} ${e?.stack ?? ''}`);

package/lib/sandbox.js

@@ -83,7 +83,15 @@ export class SandboxRunner {
         res = await doFetch();
       } catch (e) {
         if (e && e.name === 'AbortError') return { ok: false, code: 'timeout', latencyMs: Date.now() - started };
-        return { ok: false, code: 'network', latencyMs: Date.now() - started };
+        // ponytail: 1 retry on transient socket network errors before marking broken
+        await new Promise((r) => setTimeout(r, PROBE_RETRY_DELAY_MS));
+        if (ctrl.signal.aborted) return { ok: false, code: 'timeout', latencyMs: Date.now() - started };
+        try {
+          res = await doFetch();
+        } catch (e2) {
+          if (e2 && e2.name === 'AbortError') return { ok: false, code: 'timeout', latencyMs: Date.now() - started };
+          return { ok: false, code: 'network', latencyMs: Date.now() - started };
+        }
       }
       // ponytail: 1 retry on 5xx — naive; classifier refines if needed
       if (res.status >= 500 && res.status < 600) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@goodandready/dsh-key-rotation",
-  "version": "0.8.8",
+  "version": "0.8.10",
   "packageManager": "pnpm@10.33.2",
   "description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
   "keywords": [