@goodandready/dsh-key-rotation 0.8.17 → 0.8.19

package/CHANGELOG.md

@@ -3,6 +3,25 @@
 All notable changes to `@goodandready/dsh-key-rotation` are documented here.
 User-facing feature notes also appear in README (en is source of truth).
 
+## 0.8.19 - 2026-09-22
+
+### Fixed
+- **Restored pool state hydration & telemetry preservation (GitHub PR #14)**: extracted `initializePoolState()` and `hasPoolActivity()` into `lib/pool-state.js`. `StatePersistence.restorePools()` now hydrates existing runtime pools in-place, ensuring transient fields like `events` array are never undefined. `pushEvent()` defensively guards `events` array so telemetry errors cannot throw `TypeError` and mask underlying provider errors as `TRANSPORT`.
+- **Circuit breaker probe lifecycle & stream accounting (GitHub PR #15)**: added once-settled request permits with generation tokens in `CircuitBreaker`. Correctly counts the first half-open probe, checks admission before each key attempt, prevents late failures while open from extending cooldowns, and distinguishes error/aborted terminals from successes in `rotate()`.
+- **Accurate quota failure classification & backoff overflow saturation (GitHub PR #16)**: narrowed `penalizeRef()` quota detection regex in `lib/rotate.js` to explicit quota-exhaustion phrases (`insufficient_quota`, `quota exhausted`, `You exceeded your current quota`), preventing ordinary rate limits (e.g. `rate limit exceeded`, `requests per minute exceeded`, `deadline exceeded`) from being held until midnight UTC. Replaced 32-bit signed integer shifts in `computeBackoff()` with `Math.pow(2, failCount - 1)`, preventing negative delay wrap-around on 32+ failures.
+- **Quarantined key auto-unbreak isolation (GitHub PR #17)**: in `setupAutoUnbreakEffect()`, quarantined keys are now probed using the un-wrapped original credentials resolver handle, ensuring the exact broken key is tested rather than a healthy neighbor chosen by the rotation wrapper. Added concurrency lock and state-observation checks to prevent stale probes from clearing newer failures.
+- **Client error message visibility & empty ref validation (GitHub Issue #13)**: `validateBeforeSave()` in `lib/client.js` now normalizes and extracts `vres.message ?? vres.error?.message` from pre-save validation failures, displaying the server-provided reason instead of the generic "validation failed" toast. `buildPoolItem()` in `lib/pool-builder.js` filters out empty, whitespace-only, and invalid ref names using `isValidRef()`.
+
+## 0.8.18 - 2026-09-21
+
+### Fixed
+- **Stream success failCount reset (#336)**: `rotate()` now invokes `recordSuccess(pool, activeRef, now())` upon successful stream completion and clean EOF, clearing accumulated `failCounts`, removing stale `failedUntil`, and updating `lastSuccessAt`.
+- **Calendar-based quota reset connection (#337)**: connected `quota-window.js` (`nextQuotaReset`) to `penalizeRef` in `lib/rotate.js` when `QUOTA` / `RESOURCE_EXHAUSTED` / quota errors occur and `quotaResetWindow` is configured. Ensures exhausted keys remain penalized until calendar reset window instead of rapidly failing over back into penalty. Enhanced `nextQuotaReset` to safely handle both object and string window configurations.
+- **Penalty decay during maintenance sweeps (#338)**: `sweepExpired` in `lib/pool.js` now invokes `decayPenalties({ state: st }, now)`, allowing keys with historic transient failures that have run stably to gradually recover their base backoff levels. Also cleans up expired `quotaWindows`. Removed redundant duplicate check in `isSwitchableError`.
+- **State persistence during failover (#339)**: `schedulePersist()` is now passed into `createRotate()` and invoked whenever a key failover occurs, ensuring key penalty and switch state is promptly written to disk across restarts.
+- **Dead imports cleanup (#340)**: removed 16 unused named imports from `lib/index.js` (`pickCascadeFallback`, `nextQuotaReset`, `usageRows`, `usageCsv`, `findSecrets`, `readJson`, `writeSection`, etc.) ensuring lean bundle evaluation.
+- **Remote configuration import timeout (#341)**: added 10-second timeout via `AbortSignal.timeout(10000)` to remote `fetch(url)` in `/dsh-key-rotation/import` route (`lib/ops-keys.js`), returning HTTP 504 Gateway Timeout with structured error code `'timeout'` on network stalls.
+
 ## 0.8.17 - 2026-09-21
 
 ### Fixed

package/lib/circuit-breaker.js

@@ -11,18 +11,24 @@ export class CircuitBreaker {
    * @param {{ threshold?: number, openMs?: number, halfOpenProbes?: number, now?: () => number }} opts
    */
   constructor({ threshold = 5, openMs = 30_000, halfOpenProbes = 1, now = Date.now } = {}) {
-    this.threshold = Math.max(1, threshold | 0);
-    this.openMs = Math.max(100, openMs | 0);
-    this.halfOpenProbes = Math.max(1, halfOpenProbes | 0);
+    this.configure({ threshold, openMs, halfOpenProbes });
     this._now = now;
-    /** @type {Map<string, {state:string, fails:number, openedAt:number, probes:number}>} */
+    /** @type {Map<string, {state:string, fails:number, openedAt:number, probes:number, generation:number}>} */
     this._st = new Map();
   }
 
+  configure({ threshold = 5, openMs = 30_000, halfOpenProbes = 1 } = {}) {
+    const integer = (value, fallback, min) => Number.isFinite(value)
+      ? Math.max(min, Math.min(Number.MAX_SAFE_INTEGER, Math.floor(value))) : fallback;
+    this.threshold = integer(threshold, 5, 1);
+    this.openMs = integer(openMs, 30_000, 100);
+    this.halfOpenProbes = integer(halfOpenProbes, 1, 1);
+  }
+
   _entry(provider) {
     let e = this._st.get(provider);
     if (!e) {
-      e = { state: BREAKER_CLOSED, fails: 0, openedAt: 0, probes: 0 };
+      e = { state: BREAKER_CLOSED, fails: 0, openedAt: 0, probes: 0, generation: 0 };
       this._st.set(provider, e);
     }
     return e;
@@ -36,9 +42,10 @@ export class CircuitBreaker {
       if (now - e.openedAt >= this.openMs) {
         e.state = BREAKER_HALF_OPEN;
         e.probes = 0;
-        return true;
+        e.generation += 1;
+      } else {
+        return false;
       }
-      return false;
     }
     if (e.state === BREAKER_HALF_OPEN) {
       if (e.probes < this.halfOpenProbes) {
@@ -50,26 +57,58 @@ export class CircuitBreaker {
     return true;
   }
 
+  // A permit is settled exactly once. Late outcomes from an older circuit
+  // generation (or a removed provider) cannot close/reopen a newer circuit.
+  acquire(provider) {
+    if (!this.canRequest(provider)) return null;
+    const entry = this._entry(provider);
+    const generation = entry.generation;
+    const probing = entry.state === BREAKER_HALF_OPEN;
+    let settled = false;
+    const settle = (fn) => {
+      if (settled) return;
+      settled = true;
+      if (this._st.get(provider) === entry && entry.generation === generation) fn();
+    };
+    return {
+      success: () => settle(() => this.onSuccess(provider)),
+      failure: () => settle(() => this.onFailure(provider)),
+      release: () => settle(() => {
+        if (probing && entry.state === BREAKER_HALF_OPEN) {
+          entry.probes = Math.max(0, entry.probes - 1);
+        }
+      }),
+    };
+  }
+
   onSuccess(provider) {
     const e = this._entry(provider);
+    if (e.state !== BREAKER_CLOSED) e.generation += 1;
     e.fails = 0;
     e.probes = 0;
+    e.openedAt = 0;
     e.state = BREAKER_CLOSED;
   }
 
   onFailure(provider) {
     const e = this._entry(provider);
     const now = this._now();
+    // In-flight failures arriving after opening must not extend the cooldown.
+    if (e.state === BREAKER_OPEN) return e.state;
     if (e.state === BREAKER_HALF_OPEN) {
       e.state = BREAKER_OPEN;
       e.openedAt = now;
       e.fails = this.threshold;
+      e.probes = 0;
+      e.generation += 1;
       return e.state;
     }
     e.fails += 1;
     if (e.fails >= this.threshold) {
       e.state = BREAKER_OPEN;
       e.openedAt = now;
+      e.probes = 0;
+      e.generation += 1;
     }
     return e.state;
   }
@@ -81,7 +120,7 @@ export class CircuitBreaker {
   snapshot() {
     const out = {};
     for (const [k, e] of this._st) {
-      out[k] = { state: e.state, fails: e.fails, openedAt: e.openedAt || null };
+      out[k] = { state: e.state, fails: e.fails, openedAt: e.state === BREAKER_CLOSED ? null : e.openedAt };
     }
     return out;
   }
@@ -92,20 +131,22 @@ export class CircuitBreaker {
   }
 
   /** Restore entries from a persistence snapshot (#287). */
-  restore(snapshot) {
-    if (!snapshot || typeof snapshot !== 'object') return 0;
+  restore(snapshot, { preserveExisting = false } = {}) {
+    if (!snapshot || typeof snapshot !== 'object' || Array.isArray(snapshot)) return 0;
     let n = 0;
     for (const [provider, e] of Object.entries(snapshot)) {
-      if (!e || typeof e !== 'object') continue;
+      if (!e || typeof e !== 'object' || Array.isArray(e)) continue;
+      if (preserveExisting && this._st.has(provider)) continue;
       const state = e.state === BREAKER_OPEN || e.state === BREAKER_HALF_OPEN ? e.state : BREAKER_CLOSED;
       this._st.set(provider, {
         state,
         fails: Number.isFinite(e.fails) ? e.fails : 0,
         openedAt: Number.isFinite(e.openedAt) ? e.openedAt : 0,
         probes: 0,
+        generation: 0,
       });
       n += 1;
     }
     return n;
   }
-}
+}

package/lib/client.js

@@ -947,9 +947,15 @@ window.__ModuleLoader__.load({
           headers: { 'content-type': 'application/json' },
           body: JSON.stringify({ ref, value }),
         })
-          .then((r) => r.json())
-          .then((data) => { setValidationResult((m) => ({ ...m, [ref]: data })); return data; })
-          .catch(() => null)
+          .then(async (r) => {
+            const data = (await r.json().catch(() => null)) ?? {};
+            const resData = !r.ok && data.error
+              ? { ok: false, code: data.error.code, message: data.error.message }
+              : { ok: r.ok, ...data };
+            setValidationResult((m) => ({ ...m, [ref]: resData }));
+            return resData;
+          })
+          .catch((e) => ({ ok: false, message: String(e?.message ?? e) }))
           .finally(() => setValidating(''));
       };
       const saveSecret = async (ref, rowKey) => {
@@ -963,7 +969,8 @@ window.__ModuleLoader__.load({
         if (vres && vres.ok === false && vres.code === 'no-credential') {
           // No credential yet is fine for a new key being saved
         } else if (vres && !vres.ok) {
-          setSecretError(t('keyWriteFailed').replace('{msg}', vres.message || 'validation failed'));
+          const msg = vres.message ?? vres.error?.message ?? 'validation failed';
+          setSecretError(t('keyWriteFailed').replace('{msg}', msg));
           return;
         }
         fetch('/dsh-key-rotation/key', {

package/lib/heal.js

@@ -42,9 +42,10 @@ export function healIdleCooldowns(pools, idleMs, now = Date.now()) {
  * @param {Array} pools
  * @param {Function} probeFn async (ref) => { ok: boolean, code?: string, status?: number }
  * @param {number} [now=Date.now()]
+ * @param {{ isActive?: () => boolean }} [options] lifecycle guard for delayed results
  * @returns {Promise<Array<{ ref: string, poolBase: string, ok: boolean }>>}
  */
-export async function autoUnbreakBrokenKeys(pools, probeFn, now = Date.now()) {
+export async function autoUnbreakBrokenKeys(pools, probeFn, now = Date.now(), { isActive = () => true } = {}) {
   if (!Array.isArray(pools) || pools.length === 0 || typeof probeFn !== 'function') return [];
   const results = [];
   for (const pool of pools) {
@@ -58,8 +59,18 @@ export async function autoUnbreakBrokenKeys(pools, probeFn, now = Date.now()) {
       }
     }
     for (const ref of brokenRefs) {
+      if (!isActive()) return results;
+      const state = pool.state;
+      const fields = ['brokenUntil', 'failedUntil', 'failCounts', 'authFailCounts'];
+      const observed = fields.map(field => [field, state[field], state[field]?.get(ref)]);
+      // An earlier await may have removed this key from the captured worklist.
+      if (state.brokenUntil !== bu || !bu.has(ref)) continue;
       try {
         const res = await probeFn(ref);
+        if (!isActive()) return results;
+        // Do not let a late success erase a newer failure or replaced state.
+        if (pool.state !== state || observed.some(([field, map, value]) =>
+          state[field] !== map || map?.get(ref) !== value)) continue;
         if (res && res.ok) {
           bu.delete(ref);
           pool.state.failedUntil?.delete(ref);

package/lib/index.js

@@ -15,22 +15,18 @@ import {
   budgetVerdict, sortAttemptList,
 } from './pool.js';
 import { LatencyHistogram } from './histogram.js';
-import { pickCascadeFallback } from './cascade.js';
 import { ConcurrencyTracker } from './concurrency.js';
-import { nextQuotaReset } from './quota-window.js';
 import { QuotaStore } from './quota.js';
 import { WebhookSender } from './webhook.js';
-import { bucketAllow, bucketRetryMs, bucketSweep, bucketInfo } from './bucket.js';
-import { usageRows, usageCsv, compactUsage } from './usage-report.js';
-import { findSecrets, looksLikeApiSecret } from './keycheck.js';
-import { json, readJson, descriptorOf, viewOf, writeSection, providerCatalog, handleConfigBridge, NS as BRIDGE_NS } from './http-bridge.js';
+import { bucketAllow, bucketRetryMs } from './bucket.js';
+import { compactUsage } from './usage-report.js';
+import { json, handleConfigBridge } from './http-bridge.js';
 import { createRotate } from './rotate.js';
+import { initializePoolState } from './pool-state.js';
 import { nowMono } from './clock.js';
 import { CircuitBreaker } from './circuit-breaker.js';
 import { NotifyQueue } from './notify-queue.js';
-import { BoundedMap } from './bounded-map.js';
 import { classifyFailure } from './error-taxonomy.js';
-import { safeParseJson } from './atomic-io.js';
 import { registerOpsRoutes } from './routes-ops.js';
 import { registerPluginUpdater } from './plugin-updater.js';
 import { notifySwitch, notifyExhaustion, pushEvent } from './notify-events.js';
@@ -224,6 +220,11 @@ export function apply(ctx, config = {}) {
     lastConfigRef = cfg;
     lastProfilesRef = profilesSnapshot;
 
+    moduleBreaker.configure({
+      threshold: cfg.circuitBreakerThreshold ?? 5,
+      openMs: cfg.circuitBreakerOpenMs ?? 30000,
+      halfOpenProbes: cfg.circuitBreakerHalfOpenProbes ?? 1,
+    });
     const switchCodes = cfg.switchCodes ?? DEFAULT_SWITCH_CODES;
     const cooldownMs = cfg.cooldownMs ?? 60000;
     const maxCooldownMs = cfg.maxCooldownMs;
@@ -244,25 +245,10 @@ export function apply(ctx, config = {}) {
     const makeState = (base) => {
       let st = poolState.get(base);
       if (!st) {
-        st = {
-          failedUntil: new Map(),
-          failCounts: new Map(),
-          authFailCounts: new Map(),
-          brokenUntil: new Map(),
-          costPerKey: new Map(),
-          lastUsedAt: new Map(),
-          usageCounts: new Map(),
-          byModel: new Map(),
-          usageDays: new Map(),
-          quotaWindows: new Map(),
-          costDays: new Map(),
-          events: [],
-          pointer: 0,
-          lastUsed: null,
-        };
+        st = initializePoolState();
         poolState.set(base, st);
       }
-      return st;
+      return initializePoolState(st);
     };
 
     const buildPool = (base, keys, weights, poolCooldown, poolMax, expiresAt, poolStrategy, poolGuard) =>
@@ -472,6 +458,7 @@ export function apply(ctx, config = {}) {
     ctx,
     dispatchStorage,
     buildRuntime,
+    schedulePersist,
     pushEvent,
     notifySwitch: (runtime, pool, info) => notifySwitch(runtime, pool, info, { webhookSender, notifyQueue: moduleNotifyQueue, now: () => Date.now() }),
     notifyExhaustion,
@@ -533,7 +520,7 @@ export function apply(ctx, config = {}) {
     const cls = classifyFailure(payload);
     const switchable = isSwitchableError(payload, effectiveSwitchCodes) || cls.action === 'switch';
     if (!switchable) return next();
-    if (moduleBreaker) moduleBreaker.onFailure(provider);
+    if (buildRuntime().circuitBreakerEnabled && moduleBreaker) moduleBreaker.onFailure(provider);
     const ref = pool.state.lastUsed;
     if (ref) {
       const backoff = recordFailure(pool, ref, Date.now(), pool.cooldownMs ?? 60000, undefined, cls.soft, true);

package/lib/lifecycle.js

@@ -2,6 +2,7 @@
 import { healIdleCooldowns, autoUnbreakBrokenKeys } from './heal.js';
 import { StatePersistence, resolveStatePath } from './persistence.js';
 import path from 'node:path';
+import { envValue } from './pool.js';
 
 export function setupIdleHealEffect(ctx, getConfig, buildRuntime) {
   return ctx.effect(() => {
@@ -30,21 +31,33 @@ export function setupAutoUnbreakEffect(ctx, getConfig, buildRuntime, ensureSandb
     const intervalMin = cfg?.selfHealingIntervalMinutes ?? 30;
     if (!intervalMin || intervalMin <= 0) return () => {};
     const intervalMs = intervalMin * 60 * 1000;
+    let running = false;
+    let disposed = false;
+    const isActive = () => !disposed && (getConfig()?.selfHealingIntervalMinutes ?? 0) > 0;
     const timer = setInterval(async () => {
+      if (running || disposed) return;
+      running = true;
       try {
         const c = getConfig();
         if (!c || !c.selfHealingIntervalMinutes || c.selfHealingIntervalMinutes <= 0) return;
         const { pools } = buildRuntime();
         const runner = ensureSandboxRunner(ctx);
         await autoUnbreakBrokenKeys(pools, async (ref) => {
-          let val = (await ctx.credentials?.resolve?.(ref))?.value;
-          if (!val) return { ok: false };
-          return runner.probeModels(ref, val);
-        });
+          // A rotating resolver can substitute a healthy neighbor for this
+          // quarantined ref, falsely proving that the broken key recovered.
+          const credentials = ctx.get?.('credentials') ?? ctx.credentials;
+          const resolve = credentials?.__dshKeyRotationOriginalResolve ?? credentials?.resolve;
+          const hit = await resolve?.call(credentials, ref);
+          const value = typeof hit?.value === 'string' && hit.value.length > 0
+            ? hit.value : envValue(ref);
+          if (!value || !isActive()) return { ok: false };
+          return runner.probeModels(ref, value);
+        }, Date.now(), { isActive });
       } catch (e) { logger?.warn?.('[dsh-key-rotation] auto-unbreak failed', e); }
+      finally { running = false; }
     }, intervalMs);
     if (typeof timer.unref === 'function') timer.unref();
-    return () => clearInterval(timer);
+    return () => { disposed = true; clearInterval(timer); };
   }, 'dsh-key-rotation: auto-unbreak');
 }
 
@@ -65,7 +78,7 @@ export function setupPersistence(ctx, { cfg0, poolState, moduleBreaker, verboseL
         if (!snap) return;
         try {
           StatePersistence.restorePools(poolState, snap);
-          if (moduleBreaker && snap.circuit) moduleBreaker.restore(snap.circuit);
+          if (moduleBreaker && snap.circuit) moduleBreaker.restore(snap.circuit, { preserveExisting: true });
           if (verboseLoggingOn?.()) {
             logger?.warn?.(`[dsh-key-rotation] restored ${Object.keys(snap.pools ?? {}).length} pool state(s) from ${path.basename(resolvedPath)}`);
           }

package/lib/notify-events.js

@@ -32,6 +32,8 @@ export function notifySwitch(runtime, pool, info, hooks = { webhookSender: defau
 
 export function pushEvent(pool, ref, reason, cooldownMs, type) {
   const ev = { at: Date.now(), ref, reason: String(reason ?? 'UNKNOWN'), cooldownMs, type: type ?? 'fail' };
+  // Legacy/partially restored state must not replace the provider's error.
+  if (!Array.isArray(pool.state.events)) pool.state.events = [];
   pool.state.events.push(ev);
   if (pool.state.events.length > MAX_EVENTS) pool.state.events.shift();
 }

package/lib/ops-keys.js

@@ -167,7 +167,7 @@ ctx.effect(() => ctx.webServer.register({
     const url = typeof body?.url === 'string' ? body.url.trim() : '';
     if (!url || !url.startsWith('https://')) { json(res, 400, { error: { code: 'bad-url', message: 'dsh-key-rotation: only HTTPS URLs are allowed' } }); return; }
     try {
-      const resp = await fetch(url);
+      const resp = await fetch(url, { signal: AbortSignal.timeout(10000) });
       if (!resp.ok) { json(res, 400, { error: { code: 'fetch-failed', message: 'dsh-key-rotation: fetch returned ' + resp.status } }); return; }
       const data = await resp.json();
       if (!Array.isArray(data)) { json(res, 400, { error: { code: 'bad-format', message: 'dsh-key-rotation: expected JSON array of providers' } }); return; }
@@ -181,7 +181,10 @@ ctx.effect(() => ctx.webServer.register({
       const mergedArr = [...merged.values()];
       await settings.replace(NS, { ...(desc?.value ?? {}), providers: mergedArr }, desc?.revision);
       json(res, 200, { ok: true, providersImported: data.length, total: mergedArr.length });
-    } catch (e) { json(res, 400, { error: { code: 'import-failed', message: String(e?.message ?? e) } }); }
+    } catch (e) {
+      const isTimeout = e?.name === 'TimeoutError' || e?.name === 'AbortError';
+      json(res, isTimeout ? 504 : 400, { error: { code: isTimeout ? 'timeout' : 'import-failed', message: String(e?.message ?? e) } });
+    }
   },
 }), 'dsh-key-rotation: import route');
 

package/lib/persistence.js

@@ -3,6 +3,7 @@
 // reload would otherwise forget cooldowns and stampede recovered keys.
 
 import path from 'node:path';
+import { initializePoolState, hasPoolActivity } from './pool-state.js';
 import { atomicWriteJson, safeReadJson } from './atomic-io.js';
 
 const DEFAULT_FILE = 'dsh-key-rotation-state.json';
@@ -100,21 +101,24 @@ export class StatePersistence {
 
   /** Restore poolState Map from a loaded snapshot. */
   static restorePools(poolState, snapshot) {
-    if (!poolState || !snapshot || !snapshot.pools || typeof snapshot.pools !== 'object') return 0;
+    if (!poolState || !snapshot?.pools || typeof snapshot.pools !== 'object' || Array.isArray(snapshot.pools)) return 0;
     let n = 0;
-    for (const [base, st] of Object.entries(snapshot.pools)) {
-      if (!st || typeof st !== 'object') continue;
-      const failedUntil = new Map();
-      if (st.failedUntil && typeof st.failedUntil === 'object') {
-        for (const [ref, until] of Object.entries(st.failedUntil)) {
-          if (Number.isFinite(until)) failedUntil.set(ref, until);
+    for (const [base, saved] of Object.entries(snapshot.pools)) {
+      if (!saved || typeof saved !== 'object' || Array.isArray(saved)) continue;
+      const existing = poolState.get(base);
+      const state = initializePoolState(existing);
+      // Never replace objects retained by buildRuntime(), nor roll back a pool
+      // already used while the asynchronous load was in flight.
+      if (!existing || !hasPoolActivity(state)) {
+        if (saved.failedUntil && typeof saved.failedUntil === 'object' && !Array.isArray(saved.failedUntil)) {
+          for (const [ref, until] of Object.entries(saved.failedUntil)) {
+            if (Number.isFinite(until)) state.failedUntil.set(ref, until);
+          }
         }
+        state.pointer = Number.isSafeInteger(saved.pointer) && saved.pointer >= 0 ? saved.pointer : 0;
+        state.lastUsed = typeof saved.lastUsed === 'string' ? saved.lastUsed : null;
       }
-      poolState.set(base, {
-        failedUntil,
-        pointer: Number.isFinite(st.pointer) ? st.pointer : 0,
-        lastUsed: typeof st.lastUsed === 'string' ? st.lastUsed : undefined,
-      });
+      poolState.set(base, state);
       n += 1;
     }
     return n;

package/lib/pool-builder.js

@@ -1,5 +1,6 @@
 // lib/pool-builder.js — provider & per-model pool assembly and cleanup
 import { bucketSweep } from './bucket.js';
+import { isValidRef } from './pool.js';
 
 export function parseExpiry(v) {
   if (typeof v === 'number' && Number.isFinite(v) && v > 0) return v;
@@ -22,7 +23,7 @@ export function buildPoolItem({
   rpmLimit,
   makeState,
 }) {
-  const refs = (keys ?? []).filter((ref) => typeof ref === 'string' && ref.length > 0);
+  const refs = (keys ?? []).filter((ref) => typeof ref === 'string' && isValidRef(ref.trim()));
   if (refs.length === 0) return null;
   const w = Array.isArray(weights) ? weights : [];
   const weightedRefs = [];

package/lib/pool-state.js

@@ -0,0 +1,24 @@
+// Shared runtime shape for fresh and persisted pools. Hydrate in place so cached
+// runtime pools, persistence and timers continue to observe the same object.
+const MAP_FIELDS = [
+  'failedUntil', 'failCounts', 'authFailCounts', 'brokenUntil', 'costPerKey',
+  'lastUsedAt', 'usageCounts', 'byModel', 'usageDays', 'quotaWindows', 'costDays',
+];
+
+export function initializePoolState(state = {}) {
+  for (const key of MAP_FIELDS) {
+    if (!(state[key] instanceof Map)) state[key] = new Map();
+  }
+  if (!Array.isArray(state.events)) state.events = [];
+  if (!Number.isSafeInteger(state.pointer) || state.pointer < 0) state.pointer = 0;
+  if (typeof state.lastUsed !== 'string') state.lastUsed = null;
+  return state;
+}
+
+// Startup disk I/O can finish after requests have already used a pool. In that
+// case the live state wins: replaying an older cooldown/cursor can undo success.
+export function hasPoolActivity(state) {
+  return state.pointer !== 0 || state.lastUsed !== null || state.events.length > 0 ||
+    MAP_FIELDS.some((key) => state[key].size > 0) ||
+    (state.lastSuccessAt?.size ?? 0) > 0;
+}

package/lib/pool.js

@@ -122,7 +122,9 @@ export function computeBackoff(baseMs, failCount, maxMs, isSoft = false) {
   }
   const cap = maxMs ?? baseMs * 8;
   if (failCount <= 1) return Math.min(baseMs, cap);
-  const backoff = baseMs * (1 << (failCount - 1)); // 2^(n-1)
+  // Bit shifts wrap/sign-flip at 32 failures. Floating-point exponentiation
+  // saturates at Infinity, which the configured cap safely bounds.
+  const backoff = baseMs === 0 ? 0 : baseMs * (2 ** (failCount - 1));
   return Math.min(backoff, cap);
 }
 
@@ -185,10 +187,13 @@ export function sweepExpired(poolState, now = Date.now(), activeRefs = null) {
   let cleared = 0;
   const activeSet = activeRefs ? new Set(activeRefs) : null;
   for (const st of poolState.values()) {
+    // #338: decay penalty failCounts for stable keys in maintenance sweep
+    decayPenalties({ state: st }, now);
     for (const [ref, until] of [...st.failedUntil.entries()]) {
       if (until <= now) {
         st.failedUntil.delete(ref);
         st.failCounts?.delete(ref);
+        st.quotaWindows?.delete(ref);
         cleared++;
       }
     }
@@ -372,7 +377,6 @@ export function isSwitchableError(failureOrPayload, switchCodes = new Set(DEFAUL
 
   // 1. Direct HTTP status codes (#267: also 408/425 transient)
   if ((status === 429 || status === 408 || status === 425) && (switchCodes.has('RATE_LIMIT') || switchCodes.has('QUOTA') || switchCodes.has('429') || switchCodes.has('TIMEOUT') || switchCodes.has(String(status)))) return true;
-  if (status === 429 && (switchCodes.has('RATE_LIMIT') || switchCodes.has('QUOTA') || switchCodes.has('429'))) return true;
   if ((status === 401 || status === 403) && (switchCodes.has('AUTH') || switchCodes.has('401'))) return true;
   if ((status >= 500 && status <= 504) && (switchCodes.has('SERVER') || switchCodes.has(String(status)))) return true;
 

package/lib/quota-window.js

@@ -4,9 +4,10 @@ export const QUOTA_WINDOW_TYPES = ['midnight_utc', 'midnight_pst', 'rolling_24h'
 
 export function nextQuotaReset(quotaResetWindow, now) {
   now = now || Date.now();
-  if (!quotaResetWindow || typeof quotaResetWindow !== 'object') return null;
-  const type = quotaResetWindow.type;
-  const hour = Number.isFinite(quotaResetWindow.hour) ? quotaResetWindow.hour : 0;
+  const win = typeof quotaResetWindow === 'string' ? { type: quotaResetWindow === 'daily' ? 'midnight_utc' : quotaResetWindow } : quotaResetWindow;
+  if (!win || typeof win !== 'object') return null;
+  const type = win.type === 'daily' ? 'midnight_utc' : win.type;
+  const hour = Number.isFinite(win.hour) ? win.hour : 0;
 
   if (type === 'rolling_24h') {
     const d = new Date(now);

package/lib/rotate.js

@@ -5,6 +5,7 @@ import { classifyFailure } from './error-taxonomy.js';
 import {
   isSwitchableError,
   recordFailure,
+  recordSuccess,
   parseRetryAfter,
   extractRateLimit,
   isRateLimited,
@@ -12,6 +13,7 @@ import {
   sortAttemptList,
 } from './pool.js';
 import { pickCascadeFallback } from './cascade.js';
+import { nextQuotaReset } from './quota-window.js';
 
 /**
  * Create the rotate(options, pool) async generator used by llm/stream.
@@ -34,6 +36,7 @@ export function createRotate(deps) {
     quotaStore,
     circuitBreaker,
     logger,
+    schedulePersist,
     now = nowMono,
   } = deps;
 
@@ -44,6 +47,8 @@ export function createRotate(deps) {
       const runtime0 = buildRuntime();
       const { switchCodes, cooldownMs, maxCooldownMs, switchNotify, rateLimitThreshold } = runtime0;
       let lastFailure = null;
+      const breaker = runtime0.circuitBreakerEnabled === false ? null : circuitBreaker;
+      let activePermit = null;
       const reqStore = { pool, pickedRef: undefined, startMs: now() };
       setRotateStartMs(reqStore.startMs);
       let attemptList = (pool.weightedRefs ?? pool.refs).slice();
@@ -77,8 +82,27 @@ export function createRotate(deps) {
         const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
         const cls = classifyFailure({ code: errCode, message: errMsg });
         const _b = recordFailure(pool, targetRef, now(), _effBase, _max, cls.soft, true);
-        if (circuitBreaker) circuitBreaker.onFailure(pool.base ?? options?.provider);
+
+        // #337: connect quota-window calendar reset for quota exhaustion
+        // "rate limit exceeded" and "deadline exceeded" are not evidence
+        // of an exhausted billing quota. Keep those on ordinary cooldowns.
+        const isQuota = errCode === 'QUOTA' || cls.code === 'QUOTA' ||
+          /\b(?:insufficient[_\s-]+quota|quota(?:\s+is)?\s+(?:exhausted|exceeded)|exceeded\s+(?:your\s+)?(?:current\s+)?quota)\b/i.test(errMsg);
+        const qWindow = pool.quotaResetWindow ?? runtime0.quotaResetWindow;
+        if (isQuota && qWindow) {
+          const quotaResetAt = nextQuotaReset(qWindow, now());
+          if (quotaResetAt && quotaResetAt > now()) {
+            if (!pool.state.quotaWindows) pool.state.quotaWindows = new Map();
+            pool.state.quotaWindows.set(targetRef, quotaResetAt);
+            pool.state.failedUntil.set(targetRef, quotaResetAt);
+          }
+        }
+
+        activePermit?.failure();
         pushEvent(pool, targetRef, errCode ?? 'UNKNOWN', _b);
+        if (typeof schedulePersist === 'function') {
+          try { schedulePersist(); } catch (_) { /* best-effort persistence */ }
+        }
         if (!pool.state.authFailCounts) pool.state.authFailCounts = new Map();
         if (!pool.state.brokenUntil) pool.state.brokenUntil = new Map();
         const _cStr = String(errCode ?? '');
@@ -94,165 +118,192 @@ export function createRotate(deps) {
         }
       };
 
-      // #260: fail fast when provider circuit is open
-      if (circuitBreaker && !circuitBreaker.canRequest(options.provider)) {
-        logWarn(`[dsh-key-rotation] ${options.provider}: circuit open — skipping dispatch`);
-        yield finishError('CIRCUIT_OPEN', `[dsh-key-rotation] provider '${options.provider}' circuit is open`);
-        return;
-      }
-
       for (let attempt = 0; attempt < attemptList.length; attempt++) {
-        let yielded = false;
-        let switching = false;
-        let inner;
-        try {
-          // mark the internal dispatch so the interceptor does not re-rotate
-          inner = dispatchStorage.run(reqStore, () => {
-            const llm = ctx.get('llm');
-            if (!llm || typeof llm.stream !== 'function') {
-              throw new Error('dsh-key-rotation: llm service unavailable');
-            }
-            return llm.stream({ ...options, [MARKER]: true });
-          });
-        } catch (e) {
-          const curRef = reqStore.pickedRef ?? pool.state.lastUsed;
-          penalizeRef(curRef, e?.code ?? 'TRANSPORT', String(e?.message ?? ''));
-          lastFailure = finishError(e?.code ?? 'TRANSPORT',
-            `dsh-key-rotation: dispatch failed: ${String(e?.message ?? e)}`);
-          logWarn(`[dsh-key-rotation] ${options.provider}: key ${String(curRef ?? '?')} threw ${String(e?.code ?? e?.message ?? e)}`);
-          continue;
+        activePermit = breaker ? breaker.acquire(options.provider) : null;
+        if (breaker && !activePermit) {
+          logWarn(`[dsh-key-rotation] ${options.provider}: circuit open — skipping dispatch`);
+          yield finishError('CIRCUIT_OPEN', `[dsh-key-rotation] provider '${options.provider}' circuit is open`);
+          return;
         }
-
-        const _pickedRef = reqStore.pickedRef ?? pool.state.lastUsed;
-        const acquired = (_pickedRef && runtime0.concurrencyLimit > 0) ? concurrencyTracker.acquire(_pickedRef) : false;
         try {
+          let yielded = false;
+          let switching = false;
+          let inner;
           try {
-            for await (const chunk of inner) {
-              // Only actual content deltas lock the stream (no more rotation).
-              // Structural/metadata chunks (block-start/end, usage) do not.
-              if (chunk && (chunk.type === 'text-delta' || chunk.type === 'reasoning-delta' || chunk.type === 'tool-call-delta')) {
-                yielded = true;
-                yield chunk;
-                continue;
+            // mark the internal dispatch so the interceptor does not re-rotate
+            inner = dispatchStorage.run(reqStore, () => {
+              const llm = ctx.get('llm');
+              if (!llm || typeof llm.stream !== 'function') {
+                throw new Error('dsh-key-rotation: llm service unavailable');
               }
-              if (chunk && chunk.type === 'finish') {
-                const kind = chunk.reason?.kind;
-                const failure = chunk.reason?.failure;
-                const code = failure?.code;
-                const message = failure?.message ?? '';
-                const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
-                const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
-                const activeRef = reqStore.pickedRef ?? pool.state.lastUsed;
-                if (switchable) {
-                  penalizeRef(activeRef, code ?? 'UNKNOWN', message);
-                  pool.state.switches = (pool.state.switches ?? 0) + 1;
-                  pool.state.lastReason = String(code ?? 'UNKNOWN');
-                  pool.state.lastSwitchAt = now();
-                  lastFailure = chunk;
-                  logWarn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
-                  // #216: per-switch webhook (opt-in switchNotify), deduped per provider
-                  if (switchNotify && activeRef) {
-                    notifySwitch(runtime0, pool, {
-                      provider: options.provider,
-                      from: activeRef,
-                      code: String(code ?? 'UNKNOWN'),
-                      at: pool.state.lastSwitchAt,
-                    });
-                  }
-                  switching = true;
-                  break;
+              return llm.stream({ ...options, [MARKER]: true });
+            });
+          } catch (e) {
+            const curRef = reqStore.pickedRef ?? pool.state.lastUsed;
+            penalizeRef(curRef, e?.code ?? 'TRANSPORT', String(e?.message ?? ''));
+            lastFailure = finishError(e?.code ?? 'TRANSPORT',
+              `dsh-key-rotation: dispatch failed: ${String(e?.message ?? e)}`);
+            logWarn(`[dsh-key-rotation] ${options.provider}: key ${String(curRef ?? '?')} threw ${String(e?.code ?? e?.message ?? e)}`);
+            continue;
+          }
+
+          const _pickedRef = reqStore.pickedRef ?? pool.state.lastUsed;
+          const acquired = (_pickedRef && runtime0.concurrencyLimit > 0) ? concurrencyTracker.acquire(_pickedRef) : false;
+          try {
+            try {
+              for await (const chunk of inner) {
+                // Only actual content deltas lock the stream (no more rotation).
+                // Structural/metadata chunks (block-start/end, usage) do not.
+                if (chunk && (chunk.type === 'text-delta' || chunk.type === 'reasoning-delta' || chunk.type === 'tool-call-delta')) {
+                  yielded = true;
+                  yield chunk;
+                  continue;
                 }
-                // cost tracking if provider returns usage.cost
-                const todayIso = activeRef ? new Date().toISOString().slice(0, 10) : undefined;
-                if (chunk.usage?.cost != null && activeRef) {
-                  const c = Number(chunk.usage.cost);
-                  if (!isNaN(c)) {
-                    if (!pool.state.costPerKey) pool.state.costPerKey = new Map();
-                    pool.state.costPerKey.set(activeRef, (pool.state.costPerKey.get(activeRef) ?? 0) + c);
-                    // #208: cost per day per key (mirrors usageDays) for budget checks
-                    if (!pool.state.costDays) pool.state.costDays = new Map();
-                    const cMap = pool.state.costDays.get(activeRef) || new Map();
-                    cMap.set(todayIso, (cMap.get(todayIso) ?? 0) + c);
-                    pool.state.costDays.set(activeRef, cMap);
+                if (chunk && chunk.type === 'finish') {
+                  const kind = chunk.reason?.kind;
+                  const failure = chunk.reason?.failure;
+                  const code = failure?.code;
+                  const message = failure?.message ?? '';
+                  const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
+                  const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
+                  const activeRef = reqStore.pickedRef ?? pool.state.lastUsed;
+                  if (switchable) {
+                    penalizeRef(activeRef, code ?? 'UNKNOWN', message);
+                    pool.state.switches = (pool.state.switches ?? 0) + 1;
+                    pool.state.lastReason = String(code ?? 'UNKNOWN');
+                    pool.state.lastSwitchAt = now();
+                    lastFailure = chunk;
+                    logWarn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
+                    // #216: per-switch webhook (opt-in switchNotify), deduped per provider
+                    if (switchNotify && activeRef) {
+                      notifySwitch(runtime0, pool, {
+                        provider: options.provider,
+                        from: activeRef,
+                        code: String(code ?? 'UNKNOWN'),
+                        at: pool.state.lastSwitchAt,
+                      });
+                    }
+                    switching = true;
+                    break;
                   }
-                }
-                // Usage by day (#119)
-                if (activeRef) {
-                  if (!pool.state.usageDays) pool.state.usageDays = new Map();
-                  const dayMap = pool.state.usageDays.get(activeRef) || new Map();
-                  dayMap.set(todayIso, (dayMap.get(todayIso) ?? 0) + 1);
-                  pool.state.usageDays.set(activeRef, dayMap);
-                }
-                // Per-model request detail (#121)
-                if (activeRef && options.model) {
-                  if (!pool.state.byModel) pool.state.byModel = new Map();
-                  let byRef = pool.state.byModel.get(activeRef);
-                  if (!byRef) { byRef = new Map(); pool.state.byModel.set(activeRef, byRef); }
-                  byRef.set(options.model, (byRef.get(options.model) ?? 0) + 1);
-                }
-                // Proactive rate-limit (#115, #303): if response headers say this key is near
-                // its quota or has retry-after, cool it down so the NEXT request starts on a different key.
-                // We do NOT re-run this (already successful) request — that would double-send.
-                const guardEnabled = pool.proactiveRateLimitGuard ?? runtime0.proactiveRateLimitGuard ?? true;
-                const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
-                if (guardEnabled && rate && activeRef) {
-                  if (isRateLimited(rate, rateLimitThreshold ?? 0.1)) {
-                    let cool;
-                    if (typeof rate.retryAfter === 'number' && rate.retryAfter > 0) {
-                      cool = rate.retryAfter * 1000;
-                    } else if (rate.reset && rate.reset > now()) {
-                      cool = rate.reset - now();
-                    } else {
-                      cool = pool.cooldownMs ?? cooldownMs;
+                  // Error/abort terminals are not successful responses. After a
+                  // content delta we may penalize, but must never replay the request.
+                  if (kind === 'error' || kind === 'aborted') {
+                    if (kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes)) {
+                      penalizeRef(activeRef, code ?? 'UNKNOWN', message);
                     }
-                    const maxCool = pool.maxCooldownMs ?? maxCooldownMs;
-                    const effCool = Math.min(cool, maxCool ?? cool);
-                    recordFailure(pool, activeRef, now(), effCool, maxCool);
-                    pushEvent(pool, activeRef, 'RATE_LIMIT', effCool);
-                    logWarn(`[dsh-key-rotation] ${options.provider}: key ${activeRef} proactive pause (remaining ${String(rate.remaining ?? '?')}/${String(rate.limit ?? '?')}, cool ${Math.round(effCool / 1000)}s) — next request will rotate`);
+                    yield chunk;
+                    return;
                   }
-                }
-                // #7: persist quota snapshot regardless of threshold (so dashboard widget can show it).
-                if (rate && activeRef && Number.isFinite(rate.remaining) && quotaStore) {
-                  quotaStore.set(activeRef, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: now() });
+                  // cost tracking if provider returns usage.cost
+                  const todayIso = activeRef ? new Date().toISOString().slice(0, 10) : undefined;
+                  if (chunk.usage?.cost != null && activeRef) {
+                    const c = Number(chunk.usage.cost);
+                    if (!isNaN(c)) {
+                      if (!pool.state.costPerKey) pool.state.costPerKey = new Map();
+                      pool.state.costPerKey.set(activeRef, (pool.state.costPerKey.get(activeRef) ?? 0) + c);
+                      // #208: cost per day per key (mirrors usageDays) for budget checks
+                      if (!pool.state.costDays) pool.state.costDays = new Map();
+                      const cMap = pool.state.costDays.get(activeRef) || new Map();
+                      cMap.set(todayIso, (cMap.get(todayIso) ?? 0) + c);
+                      pool.state.costDays.set(activeRef, cMap);
+                    }
+                  }
+                  // Usage by day (#119)
+                  if (activeRef) {
+                    if (!pool.state.usageDays) pool.state.usageDays = new Map();
+                    const dayMap = pool.state.usageDays.get(activeRef) || new Map();
+                    dayMap.set(todayIso, (dayMap.get(todayIso) ?? 0) + 1);
+                    pool.state.usageDays.set(activeRef, dayMap);
+                  }
+                  // Per-model request detail (#121)
+                  if (activeRef && options.model) {
+                    if (!pool.state.byModel) pool.state.byModel = new Map();
+                    let byRef = pool.state.byModel.get(activeRef);
+                    if (!byRef) { byRef = new Map(); pool.state.byModel.set(activeRef, byRef); }
+                    byRef.set(options.model, (byRef.get(options.model) ?? 0) + 1);
+                  }
+                  // #336: record success for the active key upon successful stream finish
+                  if (activeRef) {
+                    recordSuccess(pool, activeRef, now());
+                  }
+
+                  // Proactive rate-limit (#115, #303): if response headers say this key is near
+                  // its quota or has retry-after, cool it down so the NEXT request starts on a different key.
+                  // We do NOT re-run this (already successful) request — that would double-send.
+                  const guardEnabled = pool.proactiveRateLimitGuard ?? runtime0.proactiveRateLimitGuard ?? true;
+                  const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
+                  if (guardEnabled && rate && activeRef) {
+                    if (isRateLimited(rate, rateLimitThreshold ?? 0.1)) {
+                      let cool;
+                      if (typeof rate.retryAfter === 'number' && rate.retryAfter > 0) {
+                        cool = rate.retryAfter * 1000;
+                      } else if (rate.reset && rate.reset > now()) {
+                        cool = rate.reset - now();
+                      } else {
+                        cool = pool.cooldownMs ?? cooldownMs;
+                      }
+                      const maxCool = pool.maxCooldownMs ?? maxCooldownMs;
+                      const effCool = Math.min(cool, maxCool ?? cool);
+                      recordFailure(pool, activeRef, now(), effCool, maxCool);
+                      pushEvent(pool, activeRef, 'RATE_LIMIT', effCool);
+                      logWarn(`[dsh-key-rotation] ${options.provider}: key ${activeRef} proactive pause (remaining ${String(rate.remaining ?? '?')}/${String(rate.limit ?? '?')}, cool ${Math.round(effCool / 1000)}s) — next request will rotate`);
+                    }
+                  }
+                  // #7: persist quota snapshot regardless of threshold (so dashboard widget can show it).
+                  if (rate && activeRef && Number.isFinite(rate.remaining) && quotaStore) {
+                    quotaStore.set(activeRef, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: now() });
+                  }
+                  recordLatency(pool, reqStore);
+                  activePermit?.success();
+                  yield chunk;
+                  return;
                 }
                 yield chunk;
-                recordLatency(pool, reqStore);
-                if (circuitBreaker) circuitBreaker.onSuccess(options.provider);
-                return;
               }
-              yield chunk;
-            }
-          } catch (e) {
-            const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
-            const activeRef = _pickedRef ?? reqStore.pickedRef ?? pool.state.lastUsed;
-            if (!yielded && isSwitchableError(e, effectiveSwitchCodes)) {
-              penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
-              pool.state.switches = (pool.state.switches ?? 0) + 1;
-              pool.state.lastReason = String(e?.code ?? 'TRANSPORT');
-              pool.state.lastSwitchAt = now();
-              lastFailure = finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
-              logWarn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} stream threw ${String(e?.code ?? e?.message ?? e)} - failover to next key`);
-              if (switchNotify && activeRef) {
-                notifySwitch(runtime0, pool, {
-                  provider: options.provider,
-                  from: activeRef,
-                  code: String(e?.code ?? 'TRANSPORT'),
-                  at: pool.state.lastSwitchAt,
-                });
+            } catch (e) {
+              const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
+              const activeRef = _pickedRef ?? reqStore.pickedRef ?? pool.state.lastUsed;
+              if (!yielded && isSwitchableError(e, effectiveSwitchCodes)) {
+                penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+                pool.state.switches = (pool.state.switches ?? 0) + 1;
+                pool.state.lastReason = String(e?.code ?? 'TRANSPORT');
+                pool.state.lastSwitchAt = now();
+                lastFailure = finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+                logWarn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} stream threw ${String(e?.code ?? e?.message ?? e)} - failover to next key`);
+                if (switchNotify && activeRef) {
+                  notifySwitch(runtime0, pool, {
+                    provider: options.provider,
+                    from: activeRef,
+                    code: String(e?.code ?? 'TRANSPORT'),
+                    at: pool.state.lastSwitchAt,
+                  });
+                }
+                continue; // Failover to next key!
+              }
+              if (yielded && isSwitchableError(e, effectiveSwitchCodes)) {
+                penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
               }
-              continue; // Failover to next key!
+              yield finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
+              return;
             }
-            yield finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
-            return;
-          }
 
-          if (switching) continue; // try the next key
-          return; // clean end — served
-        } finally {
-          if (acquired && _pickedRef) {
-            concurrencyTracker.release(_pickedRef);
+            if (switching) continue; // try the next key
+            if (yielded) {
+              const activeRef = reqStore.pickedRef ?? _pickedRef ?? pool.state.lastUsed;
+              if (activeRef) recordSuccess(pool, activeRef, now());
+              recordLatency(pool, reqStore);
+              activePermit?.success();
+            }
+            return; // clean end — served
+          } finally {
+            if (acquired && _pickedRef) {
+              concurrencyTracker.release(_pickedRef);
+            }
           }
+        } finally {
+          // Also runs on consumer return(), abort, empty EOF and local errors.
+          activePermit?.release();
         }
       }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@goodandready/dsh-key-rotation",
-  "version": "0.8.17",
+  "version": "0.8.19",
   "packageManager": "pnpm@10.33.2",
   "description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
   "keywords": [