@goodandready/dsh-key-rotation 0.7.35 → 0.7.36

package/README.md

@@ -253,6 +253,14 @@ All management routes require loopback authentication (`127.0.0.1` / `::1`) with
 
 MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
 
+### v0.7.36
+- **Architecture de-bloat & hardening**: Removed 6 unused/overengineered modules (`shadow`, `incident`, `agent-budget`, `region`, `canary`, `maintenance`) and dead route registrations.
+- **High-throughput buildRuntime memoization**: Eliminates per-token deep-cloning and schema validation on every streaming chunk.
+- **Atomic round-robin pointer rotation**: Concurrent requests advance pointer immediately on candidate selection, eliminating race conditions on simultaneous tool calls.
+- **Enhanced switchable error detection**: Direct parsing of HTTP status codes (`429`, `401`, `403`, `5xx`) and gRPC codes (`RESOURCE_EXHAUSTED`, `UNAVAILABLE`) alongside regex fallback.
+- **Informative user exhaustion messaging**: Clear countdown notice with next key recovery ETA when all keys in a pool are cooling down.
+- **Smart polling**: Client background polling paused when browser tab is inactive (`document.visibilityState`).
+
 ### v0.7.35
 - **Lifecycle Cleanups**: Wrapped `credentials.resolve` patch and `ctx.on` event handlers (`llm/stream`, `agent/request-error`) in `ctx.effect` scopes with guaranteed unmount cleanup (#238, #239).
 - **Settings & Secret Roles**: Added `.role('secret')` to `incidentGitHubToken` and `webhookActionToken` in `Config` schema for automatic UI masking (#237).

package/README.ru.md

@@ -253,6 +253,14 @@ dsh-key-rotation:
 
 MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
 
+### v0.7.36
+- **Глубокий рефакторинг и деблоатинг**: Удалены 6 неиспользуемых модулей оверинжиниринга (`shadow`, `incident`, `agent-budget`, `region`, `canary`, `maintenance`) и сопутствующие устаревшие эндпоинты.
+- **Мемоизация buildRuntime**: Устранено глубокое клонирование и повторный парсинг схемы на каждом токене/чанге стрима.
+- **Атомарная ротация указателя**: Устранено состояние гонки при параллельных вызовах инструментов — указатель смещается сразу при выборе кандидата.
+- **Точная детекция ошибок**: Прямая проверка кодов HTTP (`429`, `401`, `403`, `5xx`) и gRPC (`RESOURCE_EXHAUSTED`, `UNAVAILABLE`) с фоллбеком на регулярные выражения.
+- **Понятные сообщения при исчерпании пула**: Информативное уведомление с таймером восстановления ближайшего ключа.
+- **Энергоэффективный опрос (Smart Polling)**: Автоматическая приостановка фонового опроса при неактивной вкладке браузера (`document.visibilityState`).
+
 ### v0.7.35
 - **Очистка жизненного цикла**: Патч `credentials.resolve` и слушатели событий `ctx.on` (`llm/stream`, `agent/request-error`) переведены в скоупы `ctx.effect` с автоматическим восстановлением функций и отпиской при выгрузке плагина (#238, #239).
 - **Роли секретов в схеме**: Полям `incidentGitHubToken` и `webhookActionToken` в схеме `Config` присвоена роль `.role('secret')` для маскирования в UI (#237).

package/README.zh.md

@@ -214,6 +214,14 @@ dsh-key-rotation:
 
 MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
 
+### v0.7.36
+- **架构精简与稳定性加固**：移除 6 个过度设计的模块（`shadow`、`incident`、`agent-budget`、`region`、`canary`、`maintenance`）与废弃端点。
+- **buildRuntime 高性能记忆化**：消除每个流式 token/chunk 上的深拷贝与模式重解析开销。
+- **原子轮询指针推进**：并发请求在选定候选密钥时立即推进指针，消除并发工具调用中的竞争条件。
+- **增强的可切换错误检测**：直接解析 HTTP 状态码（`429`, `401`, `403`, `5xx`）与 gRPC 状态码（`RESOURCE_EXHAUSTED`, `UNAVAILABLE`）。
+- **用户友好的耗尽提示**：密钥池耗尽时返回带恢复倒计时的清晰通知。
+- **智能轮询（Smart Polling）**：标签页不活动时暂停客户端后台轮询。
+
 ### v0.7.35
 - **生命周期清理**: 将 `credentials.resolve` 猴子补丁和 `ctx.on` 事件监听器 (`llm/stream`, `agent/request-error`) 封装在 `ctx.effect` 作用域内，确保卸载时自动注销并恢复原始方法 (#238, #239)。
 - **配置密钥角色**: 在 `Config` Schema 中为 `incidentGitHubToken` 和 `webhookActionToken` 增加 `.role('secret')`，避免明文泄露并在 UI 中掩码显示 (#237)。

package/lib/client.js

@@ -240,12 +240,13 @@ window.__ModuleLoader__.load({
     // отдельными отмеченными галочками, чтобы правило нельзя было потерять.
     const KNOWN_CODES = ['QUOTA', 'RATE_LIMIT', 'SERVER', 'TIMEOUT', 'TRANSPORT', 'EMPTY_RESPONSE', 'UNKNOWN_MODEL', 'AUTH'];
 
-    /** Опрос статуса ротации, пока раздел настроек открыт. */
+    /** Опрос статуса ротации, пока раздел настроек открыт (smart polling). */
     function useRotationStatus() {
       const [byProvider, setByProvider] = React.useState({});
       React.useEffect(() => {
         let alive = true;
         const pull = () => {
+          if (typeof document !== 'undefined' && document.visibilityState !== 'visible') return;
           fetch('/dsh-key-rotation/status', { headers: { accept: 'application/json' } })
             .then((r) => (r.ok ? r.json() : null))
             .then((data) => {
@@ -258,17 +259,24 @@ window.__ModuleLoader__.load({
         };
         pull();
         const id = setInterval(pull, 4000);
-        return () => { alive = false; clearInterval(id); };
+        const onVis = () => { if (typeof document !== 'undefined' && document.visibilityState === 'visible') pull(); };
+        if (typeof document !== 'undefined') document.addEventListener('visibilitychange', onVis);
+        return () => {
+          alive = false;
+          clearInterval(id);
+          if (typeof document !== 'undefined') document.removeEventListener('visibilitychange', onVis);
+        };
       }, []);
       return byProvider;
     }
 
-    /** Последний probe-результат по каждому ключу (#219): /sandbox-cache. */
+    /** Последний probe-результат по каждому ключу (#219): /sandbox-cache (smart polling). */
     function useProbeCache() {
       const [cache, setCache] = React.useState({});
       React.useEffect(() => {
         let alive = true;
         const pull = () => {
+          if (typeof document !== 'undefined' && document.visibilityState !== 'visible') return;
           fetch('/dsh-key-rotation/sandbox-cache', { headers: { accept: 'application/json' } })
             .then((r) => (r.ok ? r.json() : null))
             .then((data) => { if (alive && data) setCache(data); })
@@ -276,7 +284,13 @@ window.__ModuleLoader__.load({
         };
         pull();
         const id = setInterval(pull, 4000);
-        return () => { alive = false; clearInterval(id); };
+        const onVis = () => { if (typeof document !== 'undefined' && document.visibilityState === 'visible') pull(); };
+        if (typeof document !== 'undefined') document.addEventListener('visibilitychange', onVis);
+        return () => {
+          alive = false;
+          clearInterval(id);
+          if (typeof document !== 'undefined') document.removeEventListener('visibilitychange', onVis);
+        };
       }, []);
       return cache;
     }
@@ -1076,14 +1090,21 @@ window.__ModuleLoader__.load({
       React.useEffect(() => {
         let alive = true;
         const load = () => {
+          if (typeof document !== 'undefined' && document.visibilityState !== 'visible') return;
           fetch('/dsh-key-rotation/health', { headers: { accept: 'application/json' }, credentials: 'same-origin' })
             .then((r) => (r.ok ? r.json() : null))
             .then((d) => { if (alive) setSnap(d); })
             .catch(() => {});
         };
         load();
-        const id = setInterval(load, 4000);
-        return () => { alive = false; clearInterval(id); };
+        const id = setInterval(load, 5000);
+        const onVis = () => { if (typeof document !== 'undefined' && document.visibilityState === 'visible') load(); };
+        if (typeof document !== 'undefined') document.addEventListener('visibilitychange', onVis);
+        return () => {
+          alive = false;
+          clearInterval(id);
+          if (typeof document !== 'undefined') document.removeEventListener('visibilitychange', onVis);
+        };
       }, []);
 
       React.useEffect(() => {

package/lib/index.js

@@ -33,11 +33,11 @@
 //    providers:  array       [{ provider, keys: [envName, ...] }]
 // ─────────────────────────────────────────────────────────────────────────────
 import Schema from '@deepseek-ai/schemastery';
-import { keyTail, isLoopbackAddress, isTrustedBridgeRequest, SWITCHABLE_MESSAGE_PATTERN, DEFAULT_SWITCH_CODES, isValidRef, pickNext, applyCooldown, recordFailure, recordSuccess, computeBackoff, envValue, sweepExpired, parseRetryAfter, computeHealthScore, extractRateLimit, isRateLimited, selectPool } from './pool.js';
+import { keyTail, isLoopbackAddress, isTrustedBridgeRequest, SWITCHABLE_MESSAGE_PATTERN, DEFAULT_SWITCH_CODES, isValidRef, pickNext, applyCooldown, recordFailure, recordSuccess, computeBackoff, envValue, sweepExpired, parseRetryAfter, computeHealthScore, extractRateLimit, isRateLimited, selectPool, isSwitchableError, formatExhaustionMessage, expiringSoon, shouldNotifyDaily, costForDay, costForWeek, budgetVerdict } from './pool.js';
 
 export const name = 'dsh-key-rotation';
 export const inject = ['llm', 'webServer', 'settings', 'credentials'];
-export { keyTail, isLoopbackAddress, isTrustedBridgeRequest, DEFAULT_SWITCH_CODES };
+export { keyTail, isLoopbackAddress, isTrustedBridgeRequest, DEFAULT_SWITCH_CODES, isSwitchableError, formatExhaustionMessage };
 
 /** Settings namespace owning the GUI-editable section (settingsNamespace-valid). */
 const NS = 'dsh-key-rotation';
@@ -52,27 +52,15 @@ const HEALTH_PATH = '/dsh-key-rotation/health';
 const USAGE_PATH = '/dsh-key-rotation/usage';
 const TEST_PATH = '/dsh-key-rotation/test';
 const SANDBOX_CACHE_PATH = '/dsh-key-rotation/sandbox-cache';
-const AGENT_BUDGET_PATH = '/dsh-key-rotation/agent-budget';
-const REGIONS_PATH = '/dsh-key-rotation/regions';
-const INCIDENT_RESET_PATH = '/dsh-key-rotation/incident-reset';
-const SHADOW_PATH = '/dsh-key-rotation/shadow';
-const WEBHOOK_TEST_PATH = '/dsh-key-rotation/webhook-test';
-const TEST_MATRIX_PATH = '/dsh-key-rotation/test-matrix';
 import { LastTestCache, SandboxRunner } from './sandbox.js';
 import { healIdleCooldowns } from './heal.js';
 import { LatencyHistogram } from './histogram.js';
 import { pickCascadeFallback } from './cascade.js';
 import { ConcurrencyTracker } from './concurrency.js';
 import { nextQuotaReset } from './quota-window.js';
-import { CanaryProber } from './canary.js';
 import { QuotaStore } from './quota.js';
-import { AgentBudget } from './agent-budget.js';
-import { RegionMap } from './region.js';
-import { IncidentReporter } from './incident.js';
-import { ShadowRouter } from './shadow.js';
 import { WebhookSender } from './webhook.js';
 import { bucketAllow, bucketRetryMs, bucketSweep, bucketInfo } from './bucket.js';
-import { expiringSoon, shouldNotifyDaily, costForDay, budgetVerdict, costForWeek } from './maintenance.js';
 import { usageRows, usageCsv } from './usage-report.js';
 import { findSecrets, looksLikeApiSecret } from './keycheck.js';
 
@@ -130,28 +118,12 @@ let lastTestCacheRunnerCtx = null;
 const lastTestCache = new LastTestCache();
 const latencyHistogram = new LatencyHistogram();
 const quotaStore = new QuotaStore();
-const agentBudget = new AgentBudget();
-const regionMap = new RegionMap();
 // Global config accessor safe against early initialization
 let getConfig = () => null;
 let getRuntime = () => null;
-
-// IncidentReporter: lazily built when Config provides incidentGitHubToken + incidentGitHubBaseUrl.
-// ponytail: never bake the token into source; repo is hardcoded (this plugin's home repo) but token is per-deploy.
-let incidentReporter = null;
-function ensureIncidentReporter() {
-  if (incidentReporter) return incidentReporter;
-  const cfg = getConfig();
-  const token = cfg ? cfg.incidentGitHubToken : '';
-  const baseUrl = cfg ? cfg.incidentGitHubBaseUrl : '';
-  if (!token || !baseUrl) return null;
-  incidentReporter = new IncidentReporter({ token, baseUrl, repo: 'goodandready/dsh-key-rotation', fetchImpl: globalThis.fetch });
-  return incidentReporter;
-}
-const shadowRouter = new ShadowRouter({ primary: '', secondary: '', percent: 0 });let sandboxRunner = null;
+let sandboxRunner = null;
 const webhookSender = new WebhookSender({ fetchImpl: globalThis.fetch });
 const concurrencyTracker = new ConcurrencyTracker();
-let canaryProber = null;
 function ensureSandboxRunner(ctx) {
   if (sandboxRunner) return sandboxRunner;
   // provider id or key ref -> baseUrl (stripped of trailing /) for fetch /models probe
@@ -211,20 +183,11 @@ export const Config = Schema.object({
   maxCooldownMs: Schema.number(),
   notifyWebhook: Schema.string().default(''),
   notifyThreshold: Schema.number().default(3),
-  backupDir: Schema.string().default(''),
-  backupIntervalMs: Schema.number().default(86400000),
-  backupKeep: Schema.number().default(7),
-  rotationScheduleDays: Schema.number().default(0),
   selfHealCooldown: Schema.boolean().default(true),
   selfHealIdleMs: Schema.number().default(3600000),
   latencyEnabled: Schema.boolean().default(true),
   latencyWindow: Schema.number().default(200),
-  incidentGitHubToken: Schema.string().role('secret').default(''),
-  incidentGitHubBaseUrl: Schema.string().default(''),
-  incidentThreshold: Schema.number().default(5),
   concurrencyLimit: Schema.number().default(0),
-  canaryProbingEnabled: Schema.boolean().default(false),
-  canaryIntervalMs: Schema.number().default(30000),
   cascade: Schema.array(Schema.object({
     provider: Schema.string().required(),
     model: Schema.string(),
@@ -391,7 +354,6 @@ async function handleConfigBridge(ctx, request, res, getCloneIds) {
       // fields that legitimately hold tokens are masked before scanning.
       try {
         const masked = structuredClone(section);
-        if (masked.incidentGitHubToken) masked.incidentGitHubToken = '***';
         if (masked.webhookActionToken) masked.webhookActionToken = '***';
         // notifyWebhook legitimately carries bot tokens inside URLs
         // (api.telegram.org/bot<token>/...) - scan it for nothing.
@@ -447,39 +409,6 @@ export function apply(ctx, config = {}) {
   // interval, low cost; skipped when selfHealCooldown is disabled in config.
   // ponytail: keep handle on the same ctx via closure so buildRuntime() reads
   // fresh config on every tick. Naive but correct: 60s cadence is cheap.
-  // #196: canary probing before key release from cooldown.
-  // Every canaryIntervalMs, probe refs that are in cooldown and close to expiry.
-  // Canary prober lifecycle effect
-  ctx.effect(() => {
-    const cfg = getConfig();
-    if (!cfg || !cfg.canaryProbingEnabled) return () => {};
-    const timer = setInterval(() => {
-      try {
-        const c = getConfig();
-        if (!c || !c.canaryProbingEnabled) return;
-        const runner = ensureSandboxRunner(ctx);
-        if (!runner) return;
-        if (!canaryProber) {
-          canaryProber = new CanaryProber({ sandboxRunner: runner, intervalMs: c.canaryIntervalMs });
-        }
-        const providers = Array.isArray(c.providers) ? c.providers : [];
-        for (const p of providers) {
-          const pool = buildRuntime().providerToPool.get(p.provider);
-          if (!pool) continue;
-          for (const ref of pool.refs) {
-            const until = pool.state.failedUntil.get(ref) ?? 0;
-            const now = Date.now();
-            if (until > now && until - now < (c.canaryIntervalMs ?? 30000)) {
-              canaryProber.probe(ref, ref);
-            }
-          }
-        }
-      } catch (_) { /* ponytail: never crash the timer */ }
-    }, cfg.canaryIntervalMs ?? 30000);
-    if (typeof timer.unref === 'function') timer.unref();
-    return () => clearInterval(timer);
-  }, 'dsh-key-rotation: canary prober');
-
   // Self-healing idle cooldowns lifecycle effect
   ctx.effect(() => {
     const cfg = getConfig();
@@ -500,96 +429,9 @@ export function apply(ctx, config = {}) {
     return () => clearInterval(timer);
   }, 'dsh-key-rotation: self-healing idle');
 
-  // Dashboard widget now lives in client.js (mountDashboard, see issue #152).
-  const DASH_HTML = '';
-    // ── key-pool state, persisted across config reloads ──
+  // ── key-pool state, persisted across config reloads ──
   // base provider -> { failedUntil: Map<ref, epochMs>, pointer: number, lastUsed: ref }
   const poolState = new Map();
-  // Periodic backup of pools config
-  ctx.effect(() => {
-    const { backupDir, backupIntervalMs, backupKeep } = buildRuntime();
-    if (!backupDir) return;
-    const id = setInterval(() => {
-      try {
-        const fs = require('node:fs');
-        const path = require('node:path');
-        const dir = backupDir;
-        fs.mkdirSync(dir, { recursive: true });
-        const now = new Date();
-        const dateStr = now.toISOString().slice(0,10).replace(/-/g,'');
-        const file = path.join(dir, 'pools-' + dateStr + '.json');
-        const data = JSON.stringify({ backup: now.toISOString(), providers: getConfig()?.providers ?? [] }, null, 2);
-        fs.writeFileSync(file, data, 'utf8');
-        // prune old backups
-        const keep = backupKeep || 7;
-        const files = fs.readdirSync(dir).filter((f) => f.startsWith('pools-') && f.endsWith('.json')).sort();
-        while (files.length > keep) {
-          const old = files.shift();
-          fs.unlinkSync(path.join(dir, old));
-        }
-      } catch (e) {
-        console.warn('[dsh-key-rotation] backup failed:', String(e?.message ?? e));
-      }
-    }, backupIntervalMs || 86400000);
-    return () => clearInterval(id);
-  }, 'dsh-key-rotation: backup pools');
-  // Periodic save of usage/cost stats to file
-  ctx.effect(() => {
-    const { backupDir } = buildRuntime();
-    if (!backupDir) return;
-    try {
-      const fs = require('node:fs');
-      const path = require('node:path');
-      const statsFile = path.join(backupDir, 'stats.json');
-      // Load existing stats at startup
-      try {
-        if (fs.existsSync(statsFile)) {
-          const saved = JSON.parse(fs.readFileSync(statsFile, 'utf8'));
-          for (const st of poolState.values()) {
-            if (saved.usageCounts && st.usageCounts) { for (const [k, v] of Object.entries(saved.usageCounts)) st.usageCounts.set(k, (st.usageCounts.get(k) ?? 0) + v); }
-            if (saved.costPerKey && st.costPerKey) { for (const [k, v] of Object.entries(saved.costPerKey)) st.costPerKey.set(k, (st.costPerKey.get(k) ?? 0) + v); }
-            if (saved.lastUsedAt && st.lastUsedAt) { for (const [k, v] of Object.entries(saved.lastUsedAt)) { if (!st.lastUsedAt.has(k) || v > st.lastUsedAt.get(k)) st.lastUsedAt.set(k, v); } }
-          }
-        }
-      } catch {}
-      // Periodic save
-      const id = setInterval(() => {
-        try {
-          const usageCounts = {}; const costPerKey = {}; const lastUsedAt = {};
-          for (const [base, st] of poolState) {
-            if (st.usageCounts) for (const [k, v] of st.usageCounts) usageCounts[k] = v;
-            if (st.costPerKey) for (const [k, v] of st.costPerKey) costPerKey[k] = v;
-            if (st.lastUsedAt) for (const [k, v] of st.lastUsedAt) lastUsedAt[k] = v;
-          }
-          fs.writeFileSync(statsFile, JSON.stringify({ t: Date.now(), usageCounts, costPerKey, lastUsedAt }), 'utf8');
-        } catch {}
-      }, 60000);
-      return () => clearInterval(id);
-    } catch { return () => {}; }
-  }, 'dsh-key-rotation: persist stats');
-  // Rotation schedule: shift pointer every N days
-  ctx.effect(() => {
-    const { rotationScheduleDays } = buildRuntime();
-    if (!rotationScheduleDays || rotationScheduleDays <= 0) return;
-    const intervalMs = Math.min(rotationScheduleDays * 86400000, 2147483647);
-    const id = setInterval(() => {
-      try {
-        const rt = buildRuntime();
-        let shifted = 0;
-        for (const pool of rt.poolByRef.values()) {
-          if (pool.refs.length < 2) continue;
-          const oldPtr = pool.state.pointer ?? 0;
-          pool.state.pointer = (oldPtr + 1) % pool.refs.length;
-          shifted++;
-          console.warn(`[dsh-key-rotation] ${pool.base}: scheduled rotation -> ${pool.refs[pool.state.pointer]} (day ${rotationScheduleDays})`);
-        }
-        if (shifted) console.warn(`[dsh-key-rotation] schedule: rotated ${shifted} pools`);
-      } catch (e) {
-        console.warn('[dsh-key-rotation] schedule error:', String(e?.message ?? e));
-      }
-    }, intervalMs);
-    return () => clearInterval(id);
-  }, 'dsh-key-rotation: rotation schedule');
   // Periodic sweep of expired cooldowns — keeps health probe cheap and avoids waiting for next user request
   ctx.effect(() => {
     const id = setInterval(() => {
@@ -719,27 +561,34 @@ export function apply(ctx, config = {}) {
   }, 'dsh-key-rotation: sweep expired cooldowns');
 
   // ── runtime snapshot: config + llm-pi-ai profile mapping ──
+  let cachedRuntime = null;
+  let lastConfigRef = null;
+  let lastProfilesRef = null;
+
   getRuntime = buildRuntime;
   function buildRuntime() {
-    // Deep-clone before resolving: the frozen snapshot from settings.register
-    // must never be written to by schemastery's dict resolver.
-    const cfg = Config(structuredClone(getConfig() ?? {})) ?? {};
+    const rawConfig = getConfig() ?? {};
+    let currentProfiles = null;
+    try {
+      currentProfiles = ctx.get('settings')?.get(PIAI_NS)?.providers ?? null;
+    } catch {
+      /* settings not mounted yet — empty mapping */
+    }
+
+    if (cachedRuntime && lastConfigRef === rawConfig && lastProfilesRef === currentProfiles) {
+      return cachedRuntime;
+    }
+
+    const cfg = Config(rawConfig) ?? {};
     const switchCodes = new Set(cfg.switchCodes ?? DEFAULT_SWITCH_CODES);
     const cooldownMs = cfg.cooldownMs ?? 60000;
     const maxCooldownMs = cfg.maxCooldownMs ?? undefined;
     const notifyWebhook = cfg.notifyWebhook ?? '';
     const notifyThreshold = cfg.notifyThreshold ?? 3;
-    const backupDir = cfg.backupDir ?? '';
-    const backupIntervalMs = cfg.backupIntervalMs ?? 86400000;
-    const backupKeep = cfg.backupKeep ?? 7;
-    const rotationScheduleDays = cfg.rotationScheduleDays ?? 0;
     const rateLimitThreshold = cfg.rateLimitThreshold ?? 0.1;
     const rpmLimit = cfg.rpmLimit ?? 0;
     const webhookActionToken = cfg.webhookActionToken ?? '';
-    const incidentThreshold = cfg.incidentThreshold ?? 5;
     const concurrencyLimit = cfg.concurrencyLimit ?? 0;
-    const canaryProbingEnabled = cfg.canaryProbingEnabled ?? false;
-    const canaryIntervalMs = cfg.canaryIntervalMs ?? 30000;
     const cascade = Array.isArray(cfg.cascade) ? cfg.cascade : [];
     const quotaResetWindow = cfg.quotaResetWindow || null;
 
@@ -856,7 +705,10 @@ export function apply(ctx, config = {}) {
       const weekly = typeof p.costBudgetWeekly === 'number' ? p.costBudgetWeekly : 0;
       if (daily > 0 || weekly > 0) providerBudgets.set(p.provider, { costBudgetDaily: daily, costBudgetWeekly: weekly, pauseOnBudget: p.pauseOnBudget ?? false });
     }
-    return { switchCodes, cooldownMs, maxCooldownMs, notifyWebhook, notifyThreshold, incidentThreshold, concurrencyLimit, canaryProbingEnabled, canaryIntervalMs, cascade, quotaResetWindow, backupDir, backupIntervalMs, backupKeep, rotationScheduleDays, rateLimitThreshold, rpmLimit, webhookActionToken, expiryWarnDays: cfg.expiryWarnDays ?? 7, switchNotify: cfg.switchNotify ?? false, switchNotifyThrottleMs: cfg.switchNotifyThrottleMs ?? 60000, warnBelowHealthy: cfg.warnBelowHealthy ?? 0, latencySloMs: cfg.latencySloMs ?? 0, providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider, cloneIds };
+    cachedRuntime = { switchCodes, cooldownMs, maxCooldownMs, notifyWebhook, notifyThreshold, concurrencyLimit, cascade, quotaResetWindow, rateLimitThreshold, rpmLimit, webhookActionToken, expiryWarnDays: cfg.expiryWarnDays ?? 7, switchNotify: cfg.switchNotify ?? false, switchNotifyThrottleMs: cfg.switchNotifyThrottleMs ?? 60000, warnBelowHealthy: cfg.warnBelowHealthy ?? 0, latencySloMs: cfg.latencySloMs ?? 0, providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider, cloneIds };
+    lastConfigRef = rawConfig;
+    lastProfilesRef = currentProfiles;
+    return cachedRuntime;
   }
 
   // ── patch credentials.resolve: pool refs resolve to the next healthy key ──
@@ -904,9 +756,10 @@ export function apply(ctx, config = {}) {
             continue;
           }
         }
+        // Advance pointer immediately so concurrent requests round-robin across distinct healthy keys
+        pool.state.pointer = (index + 1) % list.length;
         let hit = await original(candidate);
         if (hit && typeof hit.value === 'string' && hit.value.length > 0) {
-          pool.state.pointer = (index + 1) % list.length;
           pool.state.lastUsed = candidate;
           if (pool.state.failCounts) pool.state.failCounts.delete(candidate);
           pool.state.failedUntil.delete(candidate);
@@ -926,7 +779,6 @@ export function apply(ctx, config = {}) {
         // fallback: env var (transient, not persisted)
         const envVal = envValue(candidate);
         if (envVal !== undefined) {
-          pool.state.pointer = (index + 1) % list.length;
           pool.state.lastUsed = candidate;
           if (pool.state.failCounts) pool.state.failCounts.delete(candidate);
           pool.state.failedUntil.delete(candidate);
@@ -1037,8 +889,7 @@ export function apply(ctx, config = {}) {
               const code = failure?.code;
               const message = failure?.message ?? '';
               const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
-              const switchable = !yielded && kind === 'error' &&
-                (effectiveSwitchCodes.has(code) || SWITCHABLE_MESSAGE_PATTERN.test(message));
+              const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
               if (switchable) {
                 if (pool.state.lastUsed) {
                   const _retry = parseRetryAfter(message);
@@ -1167,7 +1018,8 @@ export function apply(ctx, config = {}) {
         }
       }
 
-      yield lastFailure ?? finishError('TRANSPORT', 'dsh-key-rotation: all keys failed');
+      const exhaustionMsg = formatExhaustionMessage(options.provider, pool);
+      yield lastFailure ?? finishError('QUOTA', exhaustionMsg);
     })();
   }
 
@@ -1337,7 +1189,6 @@ export function apply(ctx, config = {}) {
         const exportable = { ...value };
         // token-shaped fields stay empty in the file; refs are names, not secrets
         exportable.webhookActionToken = '';
-        if (exportable.incidentGitHubToken) exportable.incidentGitHubToken = '';
         json(res, 200, { at: Date.now(), version: 1, snapshot: exportable });
         return;
       }
@@ -1349,7 +1200,6 @@ export function apply(ctx, config = {}) {
       // #200 leak guard applies to imported content too
       try {
         const masked = structuredClone(snap);
-        if (masked.incidentGitHubToken) masked.incidentGitHubToken = '***';
         if (masked.webhookActionToken) masked.webhookActionToken = '***';
         if (masked.notifyWebhook) masked.notifyWebhook = '***';
         const findings = findSecrets(JSON.stringify(masked));
@@ -1363,7 +1213,6 @@ export function apply(ctx, config = {}) {
       // empty token fields in the file keep the current values (never wipe a secret)
       const merged = { ...cur, ...snap };
       if (!snap.webhookActionToken) merged.webhookActionToken = cur.webhookActionToken ?? '';
-      if (!snap.incidentGitHubToken) merged.incidentGitHubToken = cur.incidentGitHubToken ?? '';
       try {
         await settings.replace(NS, merged, desc.revision);
         const after = descriptorOf(ctx, NS);
@@ -1609,61 +1458,7 @@ export function apply(ctx, config = {}) {
     },
   }), 'dsh-key-rotation: sandbox cache');
 
-  // Auto-incident reset (#8).
-  ctx.effect(() => ctx.webServer.register({
-    kind: 'exact',
-    path: INCIDENT_RESET_PATH,
-    handler: (req, res) => {
-      if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: incident-reset is local-only' } }); return; }
-      if (req.method !== 'POST') { json(res, 405, { error: { code: 'method', message: 'POST only' } }); return; }
-      readJson(req).then((body) => {
-        const provider = typeof body?.provider === 'string' ? body.provider : '';
-        if (provider) incidentReporter.resetCooldown(provider);
-        else incidentReporter.resetCooldown();
-        json(res, 200, { ok: true, reset: provider || 'all' });
-      }).catch((e) => json(res, 400, { error: { code: 'bad-request', message: String(e?.message ?? e) } }));
-    },
-  }), 'dsh-key-rotation: incident-reset');
 
-  // #198: 1-click Health Matrix — parallel probe of all configured keys.
-  ctx.effect(() => ctx.webServer.register({
-    kind: 'exact',
-    path: TEST_MATRIX_PATH,
-    handler: async (req, res) => {
-      if (req.method !== 'POST') { json(res, 405, { error: { code: 'method', message: 'POST only' } }); return; }
-      if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: matrix is local-only' } }); return; }
-      const cfg = getConfig();
-      const runner = ensureSandboxRunner();
-      if (!runner) { json(res, 500, { error: { code: 'no-runner', message: 'sandbox runner unavailable' } }); return; }
-      const providers = Array.isArray(cfg?.providers) ? cfg.providers : [];
-      const jobs = [];
-      for (const p of providers) {
-        for (const ref of (p.keys ?? [])) {
-          if (typeof ref !== 'string' || !ref) continue;
-          jobs.push((async () => {
-            try {
-              const probeResult = await runner.probeModels(ref, ref);
-              return { provider: p.provider, ref, ok: probeResult.ok, code: probeResult.code, latencyMs: probeResult.latencyMs, modelsCount: probeResult.modelsCount ?? 0 };
-            } catch (e) {
-              return { provider: p.provider, ref, ok: false, code: 'error', latencyMs: 0, modelsCount: 0 };
-            }
-          })());
-        }
-      }
-      const results = await Promise.all(jobs);
-      json(res, 200, { at: Date.now(), total: results.length, ok: results.filter(r => r.ok).length, results });
-    },
-  }), 'dsh-key-rotation: test-matrix');
-
-  // Webhook test endpoint (#10): dry-run that validates webhookSender setup.
-  ctx.effect(() => ctx.webServer.register({
-    kind: 'exact',
-    path: WEBHOOK_TEST_PATH,
-    handler: (req, res) => {
-      if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: webhook-test is local-only' } }); return; }
-      json(res, 200, { ok: true, snapshot: webhookSender.snapshot() });
-    },
-  }), 'dsh-key-rotation: webhook-test');
 
   // #199 webhook-action: interactive webhook buttons call back here.
   // Auth: bearer token from Config (external services like Telegram/Discord
@@ -1752,39 +1547,7 @@ export function apply(ctx, config = {}) {
     },
   }), 'dsh-key-rotation: webhook-action');
 
-  // Shadow A/B sampling snapshot (#9).
-  ctx.effect(() => ctx.webServer.register({
-    kind: 'exact',
-    path: SHADOW_PATH,
-    handler: (req, res) => {
-      if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: shadow is local-only' } }); return; }
-      json(res, 200, shadowRouter.snapshot());
-    },
-  }), 'dsh-key-rotation: shadow');
 
-  // Region tags + failover chain (#4).
-  ctx.effect(() => ctx.webServer.register({
-    kind: 'exact',
-    path: REGIONS_PATH,
-    handler: (req, res) => {
-      if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: regions is local-only' } }); return; }
-      const body = regionMap.snapshot();
-      // Add pickFallback hints per provider for inspection.
-      const out = {};
-      for (const p of Object.keys(body)) out[p] = { region: body[p], fallback: regionMap.pickFallback(p) };
-      json(res, 200, out);
-    },
-  }), 'dsh-key-rotation: regions');
-
-  // Per-agent rate budget snapshot (#3).
-  ctx.effect(() => ctx.webServer.register({
-    kind: 'exact',
-    path: AGENT_BUDGET_PATH,
-    handler: (req, res) => {
-      if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: agent-budget is local-only' } }); return; }
-      json(res, 200, { enabled: agentBudget.isEnabled(), agents: agentBudget.snapshot() });
-    },
-  }), 'dsh-key-rotation: agent-budget');
 
   ctx.effect(() => ctx.on('llm/stream', (options, next) => {
     if (options[MARKER]) return next();
@@ -1812,7 +1575,7 @@ export function apply(ctx, config = {}) {
     const code = String(payload?.failure?.code ?? payload?.code ?? '');
     const message = String(payload?.failure?.message ?? payload?.message ?? '');
     const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
-    const switchable = effectiveSwitchCodes.has(code) || SWITCHABLE_MESSAGE_PATTERN.test(message);
+    const switchable = isSwitchableError(payload, effectiveSwitchCodes);
     if (!switchable) return next();
     const ref = pool.state.lastUsed;
     if (ref) {
@@ -1835,18 +1598,15 @@ export function apply(ctx, config = {}) {
   });
 }
 
-// Notify on exhaustion: webhook + (optional) GitHub incident.
+// Notify on exhaustion: webhook notification.
 // Extracted at module scope for testability. No I/O outside the injected hooks.
 // ponytail: thresholds and URLs are runtime-resolved per call, so changing Config is reflected immediately.
-export function notifyExhaustion(runtime, pool, options, hooks = { webhookSender, ensureIncidentReporter }) {
+export function notifyExhaustion(runtime, pool, options, hooks = { webhookSender }) {
   if (!runtime || !pool) return;
   const count = pool.state ? (pool.state.exhaustionCount ?? 0) : 0;
   if (count <= 0) return;
   try {
     if (runtime.notifyWebhook && count >= (runtime.notifyThreshold ?? 0)) {
-      // #199: interactive payload when an action token is configured - the
-      // platform formatter (webhook.js) turns `actions` into buttons whose
-      // callback carries the token back to /dsh-key-rotation/webhook-action.
       const token = runtime.webhookActionToken ?? '';
       const payload = {
         title: `Key pool exhausted: ${options.provider}`,
@@ -1863,10 +1623,6 @@ export function notifyExhaustion(runtime, pool, options, hooks = { webhookSender
       };
       hooks.webhookSender.send(runtime.notifyWebhook, payload);
     }
-    if (runtime.incidentThreshold && count >= runtime.incidentThreshold) {
-      const reporter = hooks.ensureIncidentReporter();
-      if (reporter) reporter.open(options.provider, pool.state.lastExhaustionAt);
-    }
   } catch (_) { /* ponytail: never crash rotate() */ }
 }
 

package/lib/pool.js

@@ -44,6 +44,10 @@ export const SWITCHABLE_MESSAGE_PATTERN = new RegExp([
   /\bout[\s_-]+of[\s_-]+(?:credits?|budget)\b/i,
   /\b(?:exceeded|exhausted)[\s_-]+(?:quota|limit|budget)\b/i,
   /\bbilling\b/i,
+  /\bresource[\s_-]+exhausted\b/i,
+  /\bcapacity[\s_-]+(?:limit|exceeded|reached)\b/i,
+  /\bfree[\s_-]+tier[\s_-]+(?:limit|exceeded)\b/i,
+  /\bconcurrent[\s_-]+(?:requests?|limit)[\s_-]+exceeded\b/i,
   /\b429\b|\b5\d\d\b/i,
   /\btime(?:d)?\s*out\b|timeout/i,
   /\b(?:network|connection|socket|fetch|ECONN[A-Z]+)\b/i,
@@ -52,6 +56,7 @@ export const SWITCHABLE_MESSAGE_PATTERN = new RegExp([
   /\b(?:invalid|expired|revoked|unauthorized)[\s_-]+(?:api[\s_-]?key|token)\b/i,
   /\bapi[\s_-]?key[\s_-]+(?:is[\s_-]+)?(?:invalid|expired|revoked|unauthorized)\b/i,
   /\b(?:authentication|unauthorized|not[\s_-]+authorized)\b/i,
+  /\b(?:overloaded|server[\s_-]+busy)\b/i,
 ].map((r) => r.source).join('|'), 'i');
 
 /** Default switch codes used by lib/index.js. */
@@ -261,4 +266,108 @@ export function isRateLimited(rate, threshold = 0.1) {
   if (rate.remaining === undefined) return false;
   if (rate.limit && rate.limit > 0) return rate.remaining < rate.limit * threshold;
   return rate.remaining <= 0;
-}
+}
+
+/**
+ * Unified check whether an error or payload represents a switchable failure.
+ * Prioritizes explicit HTTP status codes and gRPC codes before text matching.
+ */
+export function isSwitchableError(failureOrPayload, switchCodes = new Set(DEFAULT_SWITCH_CODES)) {
+  if (!failureOrPayload) return false;
+  const failure = failureOrPayload.failure ?? failureOrPayload;
+  const status = Number(failure.status ?? failure.statusCode ?? failure.httpStatus ?? 0);
+  const code = String(failure.code ?? failure.reason ?? '').toUpperCase();
+  const message = String(failure.message ?? failureOrPayload.message ?? '');
+
+  // 1. Direct HTTP status codes
+  if (status === 429 && (switchCodes.has('RATE_LIMIT') || switchCodes.has('QUOTA') || switchCodes.has('429'))) return true;
+  if ((status === 401 || status === 403) && (switchCodes.has('AUTH') || switchCodes.has('401'))) return true;
+  if ((status >= 500 && status <= 504) && (switchCodes.has('SERVER') || switchCodes.has(String(status)))) return true;
+
+  // 2. Standard gRPC / cloud error codes
+  if (code === 'RESOURCE_EXHAUSTED' && (switchCodes.has('QUOTA') || switchCodes.has('RATE_LIMIT'))) return true;
+  if ((code === 'UNAVAILABLE' || code === 'INTERNAL') && switchCodes.has('SERVER')) return true;
+  if (code === 'DEADLINE_EXCEEDED' && switchCodes.has('TIMEOUT')) return true;
+  if ((code === 'UNAUTHENTICATED' || code === 'PERMISSION_DENIED') && switchCodes.has('AUTH')) return true;
+
+  // 3. Named switch code matching
+  if (code && switchCodes.has(code)) return true;
+
+  // 4. Fallback text pattern matching
+  return SWITCHABLE_MESSAGE_PATTERN.test(message);
+}
+
+/**
+ * Format an informative user-facing exhaustion message with recovery countdown.
+ */
+export function formatExhaustionMessage(provider, pool, now = Date.now()) {
+  const list = pool.refs ?? [];
+  const total = list.length;
+  let minWaitMs = Infinity;
+  if (pool.state && pool.state.failedUntil) {
+    for (const ref of list) {
+      const until = pool.state.failedUntil.get(ref) ?? 0;
+      if (until > now) {
+        const wait = until - now;
+        if (wait < minWaitMs) minWaitMs = wait;
+      }
+    }
+  }
+  const sec = Number.isFinite(minWaitMs) && minWaitMs > 0 ? Math.ceil(minWaitMs / 1000) : 60;
+  return `[dsh-key-rotation] All ${total} keys for provider '${provider}' are temporarily exhausted. Next key recovers in ~${sec}s.`;
+}
+
+
+/**
+ * Keys of pool whose expiresAt falls within the next warnDays.
+ * Returns [{ ref, expiresInDays, expiresAt }], soonest first.
+ */
+export function expiringSoon(pool, warnDays = 7, now = Date.now()) {
+  if (!pool || !pool.expiresAt) return [];
+  const DAY_MS = 86400000;
+  const horizon = now + Math.max(1, warnDays) * DAY_MS;
+  return Object.entries(pool.expiresAt)
+    .filter(([, at]) => at > now && at <= horizon)
+    .map(([ref, at]) => ({ ref, expiresAt: at, expiresInDays: Math.max(0, Math.floor((at - now) / DAY_MS)) }))
+    .sort((a, b) => a.expiresAt - b.expiresAt);
+}
+
+/** Dedupe: true when a day-level notification for key is due. */
+export function shouldNotifyDaily(lastNotified, key, now = Date.now()) {
+  const DAY_MS = 86400000;
+  if (!lastNotified.has(key)) {
+    lastNotified.set(key, now);
+    return true;
+  }
+  const last = lastNotified.get(key);
+  if (now - last < DAY_MS) return false;
+  lastNotified.set(key, now);
+  return true;
+}
+
+/** Total spend of a pool on ISO day day (defaults to today) across costDays Map<ref, Map<day, cost>>. */
+export function costForDay(costDays, day) {
+  const d = day ?? new Date().toISOString().slice(0, 10);
+  let total = 0;
+  for (const perRef of (costDays?.values() ?? [])) {
+    total += perRef.get(d) ?? 0;
+  }
+  return total;
+}
+
+/** Total spend over the last 7 ISO days ending today. */
+export function costForWeek(costDays, now = Date.now()) {
+  let total = 0;
+  for (let i = 0; i < 7; i++) {
+    const d = new Date(now - i * 86400000).toISOString().slice(0, 10);
+    total += costForDay(costDays, d);
+  }
+  return total;
+}
+
+/** Budget verdict for a pool: { spend, budget, ratio, warn, exceeded }. */
+export function budgetVerdict(spend, budget) {
+  if (!budget || budget <= 0) return { spend, budget: 0, ratio: 0, warn: false, exceeded: false };
+  const ratio = spend / budget;
+  return { spend, budget, ratio, warn: ratio >= 0.8, exceeded: ratio >= 1 };
+}

package/package.json

@@ -1,7 +1,6 @@
 {
   "name": "@goodandready/dsh-key-rotation",
-  "version": "0.7.35",
-  "packageManager": "pnpm@10.33.2",
+  "version": "0.7.36",
   "description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
   "keywords": [
     "deepseek-harness",
@@ -56,4 +55,4 @@
   "scripts": {
     "test": "node --test test/*.test.js test/*.test.mjs"
   }
-}
+}

package/lib/agent-budget.js

@@ -1,68 +0,0 @@
-// lib/agent-budget.js — per-agent rate cap.
-// ponytail: in-memory counter per (agent, window), thread-safe-ish via timer map.
-
-export const AGENT_BUDGET_DEFAULT_WINDOW_MS = 3600_000; // 1h
-export const AGENT_BUDGET_DEFAULT_LIMIT = 0;             // 0 = disabled
-export const AGENT_BUDGET_MAX = 50000;                  // hard ceiling per agent
-
-export class AgentBudget {
-  constructor({ windowMs = AGENT_BUDGET_DEFAULT_WINDOW_MS, limit = AGENT_BUDGET_DEFAULT_LIMIT } = {}) {
-    const w = Number.isFinite(windowMs) && windowMs > 0 ? Math.floor(windowMs) : AGENT_BUDGET_DEFAULT_WINDOW_MS;
-    const l = Number.isFinite(limit) && limit >= 0 ? Math.min(AGENT_BUDGET_MAX, Math.floor(limit)) : 0;
-    this._windowMs = w;
-    this._limit = l;
-    this._state = new Map(); // agent -> { hits: number[], windowStart: epochMs }
-  }
-
-  isEnabled() {
-    return this._limit > 0;
-  }
-
-  // Decide if request from this agent is allowed. Returns { allowed, remaining, resetAt }.
-  // Records the hit only when allowed.
-  check(agentId, now = Date.now()) {
-    if (!this.isEnabled()) return { allowed: true, remaining: Infinity, resetAt: null };
-    if (!agentId || typeof agentId !== 'string') return { allowed: false, remaining: 0, resetAt: now };
-    let s = this._state.get(agentId);
-    if (!s) {
-      s = { hits: [], windowStart: now };
-      this._state.set(agentId, s);
-    }
-    // Window: prune hits older than windowStart + windowMs
-    const cutoff = now - this._windowMs;
-    while (s.hits.length > 0 && s.hits[0] < cutoff) s.hits.shift();
-    s.windowStart = s.hits.length ? s.hits[0] : now;
-    if (s.hits.length >= this._limit) {
-      const resetAt = s.hits[0] + this._windowMs;
-      return { allowed: false, remaining: 0, resetAt };
-    }
-    s.hits.push(now);
-    return { allowed: true, remaining: this._limit - s.hits.length, resetAt: now + this._windowMs };
-  }
-
-  // Reset single agent or all
-  reset(agentId) {
-    if (agentId) this._state.delete(agentId);
-    else this._state.clear();
-  }
-
-  // Inspect-only: return remaining without recording.
-  peek(agentId, now = Date.now()) {
-    if (!this.isEnabled()) return { remaining: Infinity, resetAt: null };
-    const s = this._state.get(agentId);
-    if (!s) return { remaining: this._limit, resetAt: null };
-    const cutoff = now - this._windowMs;
-    let count = 0;
-    for (let i = 0; i < s.hits.length; i++) {
-      if (s.hits[i] >= cutoff) count += 1;
-    }
-    const oldest = s.hits[0];
-    return { remaining: this._limit - count, resetAt: oldest ? oldest + this._windowMs : null };
-  }
-
-  snapshot() {
-    const out = {};
-    for (const [k, v] of this._state) out[k] = { hits: v.hits.length };
-    return out;
-  }
-}

package/lib/canary.js

@@ -1,63 +0,0 @@
-// canary.js — canary probing before releasing a key from cooldown (issue #196, #7).
-
-export const CANARY_PROBE_TIMEOUT_MS = 5000;
-export const CANARY_DEFAULT_INTERVAL_MS = 30 * 1000;
-
-export class CanaryProber {
-  constructor(opts) {
-    opts = opts || {};
-    if (!opts.sandboxRunner) throw new Error('canary: sandboxRunner required');
-    this._runner = opts.sandboxRunner;
-    this._intervalMs = opts.intervalMs || CANARY_DEFAULT_INTERVAL_MS;
-    this._probeTargetModel = Boolean(opts.probeTargetModel);
-    this._results = new Map();
-    this._inProgress = new Set();
-  }
-
-  get intervalMs() { return this._intervalMs; }
-  get probeTargetModel() { return this._probeTargetModel; }
-
-  async probe(ref, key, targetModel = null) {
-    if (!ref || this._inProgress.has(ref)) return null;
-    this._inProgress.add(ref);
-    try {
-      let result;
-      if (this._probeTargetModel && targetModel && typeof this._runner.probeChatCompletion === 'function') {
-        result = await this._runner.probeChatCompletion(ref, key, targetModel);
-      } else {
-        result = await this._runner.probeModels(ref, key);
-      }
-      this._results.set(ref, Object.assign({}, result, { at: Date.now() }));
-      return result;
-    } catch (e) {
-      return { ok: false, code: 'error', at: Date.now() };
-    } finally {
-      this._inProgress.delete(ref);
-    }
-  }
-
-  lastResult(ref) {
-    return this._results.get(ref) || null;
-  }
-
-  isHealthy(ref) {
-    const r = this._results.get(ref);
-    return Boolean(r && r.ok);
-  }
-
-  clear(ref) {
-    if (ref) {
-      this._results.delete(ref);
-      this._inProgress.delete(ref);
-    } else {
-      this._results.clear();
-      this._inProgress.clear();
-    }
-  }
-
-  snapshot() {
-    const out = {};
-    for (const [k, v] of this._results) out[k] = v;
-    return out;
-  }
-}

package/lib/incident.js

@@ -1,76 +0,0 @@
-// lib/incident.js — auto-create Gitea issue when pool exhausted > threshold.
-// ponytail: minimal — caller provides a token + base URL. No retries on rate-limit.
-
-export const INCIDENT_DEFAULT_THRESHOLD_MS = 5 * 60 * 1000; // 5 min
-export const INCIDENT_DEFAULT_COOLDOWN_MS = 30 * 60 * 1000; // 30 min between incidents per provider
-export const INCIDENT_TIMEOUT_MS = 5000;
-
-export class IncidentReporter {
-  constructor({ token, baseUrl, repo, thresholdMs = INCIDENT_DEFAULT_THRESHOLD_MS, cooldownMs = INCIDENT_DEFAULT_COOLDOWN_MS, fetchImpl } = {}) {
-    if (!token) throw new Error('incident: token required');
-    if (!baseUrl) throw new Error('incident: baseUrl required');
-    if (!repo || !repo.includes('/')) throw new Error('incident: repo (owner/name) required');
-    this._token = token;
-    this._baseUrl = baseUrl.replace(/\/+$/, '');
-    this._repo = repo;
-    this._thresholdMs = thresholdMs;
-    this._cooldownMs = cooldownMs;
-    this._lastIncidentAt = new Map(); // provider -> epochMs
-    this._fetch = fetchImpl || (typeof fetch !== 'undefined' ? fetch : () => { throw new Error('incident: no fetch available'); });
-  }
-
-  // Should we report now? Pure; does not perform I/O.
-  shouldReport(provider, exhaustedSince, now = Date.now()) {
-    if (!provider) return false;
-    if (!Number.isFinite(exhaustedSince)) return false;
-    if (now - exhaustedSince < this._thresholdMs) return false;
-    const last = this._lastIncidentAt.get(provider);
-    if (Number.isFinite(last) && now - last < this._cooldownMs) return false;
-    return true;
-  }
-
-  markReported(provider, at = Date.now()) {
-    this._lastIncidentAt.set(provider, at);
-  }
-
-  resetCooldown(provider) {
-    if (provider) this._lastIncidentAt.delete(provider);
-    else this._lastIncidentAt.clear();
-  }
-
-  // Open a Gitea issue. ponytail: minimal payload, ignore failures.
-  async open(provider, exhaustedSince, now = Date.now()) {
-    if (!this.shouldReport(provider, exhaustedSince, now)) return { reported: false };
-    const url = `${this._baseUrl}/api/v1/repos/${this._repo}/issues`;
-    const body = {
-      title: `prod-incident: pool ${provider} exhausted since ${new Date(exhaustedSince).toISOString()}`,
-      body: [
-        'Auto-generated by `dsh-key-rotation`.',
-        '',
-        `- provider: \`${provider}\``,
-        `- exhaustedSince: \`${new Date(exhaustedSince).toISOString()}\``,
-        '',
-        'All keys in the pool are in cooldown or missing. Check OpenCode provider status and rotate keys.',
-      ].join('\n'),
-      labels: ['prod-incident'],
-    };
-    const ctrl = new AbortController();
-    const timer = setTimeout(() => ctrl.abort(), INCIDENT_TIMEOUT_MS);
-    try {
-      const res = await this._fetch(url, {
-        method: 'POST',
-        headers: { authorization: `token ${this._token}`, 'content-type': 'application/json' },
-        body: JSON.stringify(body),
-        signal: ctrl.signal,
-      });
-      if (!res.ok) return { reported: false, status: res.status };
-      const data = await res.json();
-      this.markReported(provider, now);
-      return { reported: true, number: data.number, url: data.html_url };
-    } catch (_) {
-      return { reported: false };
-    } finally {
-      clearTimeout(timer);
-    }
-  }
-}

package/lib/maintenance.js

@@ -1,64 +0,0 @@
-// lib/maintenance.js - pure helpers for #207 (expiry pre-warning) and
-// #208 (cost budget). No I/O; the timers in index.js call these and decide
-// whether to send webhooks.
-
-const DAY_MS = 86400000;
-
-/**
- * #207: keys of `pool` whose expiresAt falls within the next `warnDays`.
- * Returns [{ ref, expiresInDays, expiresAt }], soonest first.
- */
-export function expiringSoon(pool, warnDays, now = Date.now()) {
-  if (!pool || !pool.expiresAt) return [];
-  const horizon = now + Math.max(1, warnDays ?? 7) * DAY_MS;
-  return Object.entries(pool.expiresAt)
-    .filter(([, at]) => at > now && at <= horizon)
-    .map(([ref, at]) => ({ ref, expiresAt: at, expiresInDays: Math.max(0, Math.floor((at - now) / DAY_MS)) }))
-    .sort((a, b) => a.expiresAt - b.expiresAt);
-}
-
-/** #207 dedupe: true when a day-level notification for `key` is due. */
-export function shouldNotifyDaily(lastNotified, key, now = Date.now()) {
-  if (!lastNotified.has(key)) {
-    lastNotified.set(key, now);
-    return true;
-  }
-  const last = lastNotified.get(key);
-  if (now - last < DAY_MS) return false;
-  lastNotified.set(key, now);
-  return true;
-}
-
-/**
- * #208: total spend of a pool on ISO day `day` (defaults to today)
- * across costDays Map<ref, Map<day, cost>>.
- */
-export function costForDay(costDays, day) {
-  const d = day ?? new Date().toISOString().slice(0, 10);
-  let total = 0;
-  for (const perRef of (costDays?.values() ?? [])) {
-    total += perRef.get(d) ?? 0;
-  }
-  return total;
-}
-
-/** #208: total spend over the last 7 ISO days ending today. */
-export function costForWeek(costDays, now = Date.now()) {
-  let total = 0;
-  for (let i = 0; i < 7; i++) {
-    const d = new Date(now - i * 86400000).toISOString().slice(0, 10);
-    total += costForDay(costDays, d);
-  }
-  return total;
-}
-
-/**
- * #208 budget verdict for a pool: { spend, budget, ratio, warn, exceeded }.
- * budget <= 0 -> never warn.
- */
-export function budgetVerdict(spend, budget) {
-  if (!budget || budget <= 0) return { spend, budget: 0, ratio: 0, warn: false, exceeded: false };
-  const ratio = spend / budget;
-  // warn from 80%, exceeded at 100%
-  return { spend, budget, ratio, warn: ratio >= 0.8, exceeded: ratio >= 1 };
-}

package/lib/region.js

@@ -1,50 +0,0 @@
-// lib/region.js — region tag + failover helper.
-// ponytail: simple — providers declare an optional 'region' (string).
-// When the primary provider hits exhaustion AND a same-region fallback is
-// configured, the plugin picks that as next fallback.
-
-export const REGION_NONE = '';
-export const REGION_GLOBAL = 'global';
-
-export class RegionMap {
-  constructor() {
-    this._byProvider = new Map(); // provider id -> region
-  }
-
-  set(provider, region = REGION_GLOBAL) {
-    if (!provider) return;
-    if (!region) region = REGION_GLOBAL;
-    this._byProvider.set(provider, region);
-  }
-
-  get(provider) {
-    return this._byProvider.get(provider) || REGION_GLOBAL;
-  }
-
-  // Pick a fallback for `provider`. Returns another provider in the same
-  // region if available; otherwise null. Returns null for unknown providers
-  // (we don't know their region -> conservative).
-  pickFallback(provider) {
-    if (!this._byProvider.has(provider)) return null;
-    const region = this.get(provider);
-    for (const [p, r] of this._byProvider) {
-      if (p === provider) continue;
-      if (r === region) return p;
-    }
-    return null;
-  }
-
-  snapshot() {
-    const out = {};
-    for (const [k, v] of this._byProvider) out[k] = v;
-    return out;
-  }
-
-  clear() {
-    this._byProvider.clear();
-  }
-
-  get size() {
-    return this._byProvider.size;
-  }
-}

package/lib/shadow.js

@@ -1,81 +0,0 @@
-// lib/shadow.js — shadow A/B traffic sampling.
-// ponytail: per-provider counter, simple percent gating.
-
-export const SHADOW_DEFAULT_PERCENT = 0;  // 0 = disabled
-export const SHADOW_BUCKET = 100;        // percent base
-
-export class ShadowRouter {
-  constructor({ primary, secondary, percent = SHADOW_DEFAULT_PERCENT } = {}) {
-    this._primary = primary || '';
-    this._secondary = secondary || '';
-    this._percent = Number.isFinite(percent) && percent > 0 ? Math.min(SHADOW_BUCKET, Math.floor(percent)) : 0;
-    this._sent = 0;
-    this._shadowed = 0;
-    this._latencySumPrimary = 0;
-    this._latencySumSecondary = 0;
-    this._latencyCountPrimary = 0;
-    this._latencyCountSecondary = 0;
-  }
-
-  isEnabled() {
-    return this._percent > 0 && Boolean(this._primary) && Boolean(this._secondary) && this._primary !== this._secondary;
-  }
-
-  pick(requestHash = Math.random()) {
-    if (!this.isEnabled()) return { primary: this._primary, secondary: null, sampled: false };
-    // Convert requestHash to [0, SHADOW_BUCKET)
-    let h;
-    if (typeof requestHash === 'number') {
-      h = Math.floor(requestHash * SHADOW_BUCKET);
-    } else {
-      // Stable hash: fnv1a-lite on string
-      let str = String(requestHash);
-      let x = 2166136261;
-      for (let i = 0; i < str.length; i++) {
-        x ^= str.charCodeAt(i);
-        x = (x * 16777619) >>> 0;
-      }
-      h = x % SHADOW_BUCKET;
-    }
-    const sampled = h < this._percent;
-    this._sent += 1;
-    if (sampled) this._shadowed += 1;
-    return { primary: this._primary, secondary: sampled ? this._secondary : null, sampled };
-  }
-
-  recordLatency(target, ms) {
-    if (!Number.isFinite(ms) || ms < 0) return;
-    if (target === this._primary) {
-      this._latencySumPrimary += ms;
-      this._latencyCountPrimary += 1;
-    } else if (target === this._secondary) {
-      this._latencySumSecondary += ms;
-      this._latencyCountSecondary += 1;
-    }
-  }
-
-  snapshot() {
-    const avg = (sum, count) => (count > 0 ? sum / count : null);
-    return {
-      primary: this._primary,
-      secondary: this._secondary,
-      percent: this._percent,
-      enabled: this.isEnabled(),
-      sent: this._sent,
-      shadowed: this._shadowed,
-      avgLatencyMs: {
-        primary: avg(this._latencySumPrimary, this._latencyCountPrimary),
-        secondary: avg(this._latencySumSecondary, this._latencyCountSecondary),
-      },
-    };
-  }
-
-  reset() {
-    this._sent = 0;
-    this._shadowed = 0;
-    this._latencySumPrimary = 0;
-    this._latencySumSecondary = 0;
-    this._latencyCountPrimary = 0;
-    this._latencyCountSecondary = 0;
-  }
-}