@goodandready/dsh-key-rotation 0.7.36 → 0.7.37

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -253,6 +253,14 @@ All management routes require loopback authentication (`127.0.0.1` / `::1`) with
253
253
 
254
254
  MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
255
255
 
256
+ ### v0.7.37
257
+ - **Request Context Isolation via AsyncLocalStorage**: Scoped active key resolution (`pickedRef`), start timestamps, and retry counts strictly to each async dispatch context using Node's `node:async_hooks`. Eliminates race conditions where concurrent streaming requests could penalize healthy keys.
258
+ - **Failover on Pre-Yield Stream Exceptions**: Fixed fatal stream termination where transport errors (e.g. HTTP 429 thrown before headers, socket hang-up) aborted the generator. The catch block now inspects `isSwitchableError` and cascades seamlessly to the next key if no content tokens have been yielded.
259
+ - **Pure Local Candidate List in rotate()**: Generator uses an isolated local slice of keys (`attemptList`), eliminating shared mutations on `pool.weightedRefs`.
260
+ - **Automatic Quarantine Release on Probe Success**: Successful sandbox model tests via Settings UI (`/dsh-key-rotation/test`) automatically lift `failedUntil` and `brokenUntil` quarantine flags.
261
+ - **Long-Term Memory Compaction**: Wired `compactUsage(pool, 30, now)` into the periodic 30-second maintenance sweep to prevent memory growth on high-uptime servers.
262
+ - **Request-Scoped Latency Recording**: Replaced module-global start timestamp with context-scoped `startMs` for accurate p50/p95 latency metrics under concurrent load.
263
+
256
264
  ### v0.7.36
257
265
  - **Architecture de-bloat & hardening**: Removed 6 unused/overengineered modules (`shadow`, `incident`, `agent-budget`, `region`, `canary`, `maintenance`) and dead route registrations.
258
266
  - **High-throughput buildRuntime memoization**: Eliminates per-token deep-cloning and schema validation on every streaming chunk.
package/README.ru.md CHANGED
@@ -253,6 +253,14 @@ dsh-key-rotation:
253
253
 
254
254
  MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
255
255
 
256
+ ### v0.7.37
257
+ - **Изоляция контекста запросов через AsyncLocalStorage**: Полная привязка активного ключа (`pickedRef`), времени старта и попыток к асинхронному контексту вызова через `node:async_hooks`. Устранена гонка, при которой параллельные запросы могли ошибочно штрафовать здоровый ключ соседа.
258
+ - **Failover при исключениях в потоке до первого чанка**: Устранено аварийное прерывание потока при транспортных сбоях (например, выброс HTTP 429 до отправки заголовков). Теперь блок перехвата проверяет `isSwitchableError` и прозрачно переключает поток на запасной ключ, если клиенту ещё не было отдано полезных данных.
259
+ - **Чистый локальный список кандидатов в rotate()**: Устранена прямая мутация разделяемого массива `pool.weightedRefs` во время выполнения ротации.
260
+ - **Автоматический вывод из карантина при успехе проверки**: Успешная проверка ключа через песочницу в настройках (`/dsh-key-rotation/test`) моментально сбрасывает флаги карантина (`failedUntil` и `brokenUntil`).
261
+ - **Регулярная компактизация памяти**: Функция `compactUsage(pool, 30, now)` подключена в 30-секундный фоновый таймер пула, предотвращая утечки памяти при непрерывной многомесячной работе.
262
+ - **Точные метрики задержки при параллельных вызовах**: Переход с глобальной переменной времени старта на контекстный замер `startMs` гарантирует достоверность p50/p95 latency под нагрузкой.
263
+
256
264
  ### v0.7.36
257
265
  - **Глубокий рефакторинг и деблоатинг**: Удалены 6 неиспользуемых модулей оверинжиниринга (`shadow`, `incident`, `agent-budget`, `region`, `canary`, `maintenance`) и сопутствующие устаревшие эндпоинты.
258
266
  - **Мемоизация buildRuntime**: Устранено глубокое клонирование и повторный парсинг схемы на каждом токене/чанге стрима.
package/README.zh.md CHANGED
@@ -214,6 +214,14 @@ dsh-key-rotation:
214
214
 
215
215
  MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
216
216
 
217
+ ### v0.7.37
218
+ - **通过 AsyncLocalStorage 隔离请求上下文**:使用 Node.js 的 `node:async_hooks` 将解析后的密钥 (`pickedRef`)、启动时间和重试严格限定在单个请求上下文内,彻底消除并发请求间的竞态条件与误罚。
219
+ - **流异常自动故障转移**:修复流在首个 token 返回前抛出传输异常(如 HTTP 429)直接终止的问题。若未发送内容块,现在会自动触发 `isSwitchableError` 并顺畅切换到备用密钥。
220
+ - **避免在 rotate() 中直接修改共享状态**:遍历候选列表改用纯净的局部切片,不再直接覆盖修改 `pool.weightedRefs`。
221
+ - **测试成功后自动解除隔离**:在设置界面通过沙箱成功验证密钥有效性后,自动清除 `failedUntil` 和 `brokenUntil` 惩罚标记。
222
+ - **定期内存压缩清理**:将 `compactUsage(pool, 30, now)` 接入 30 秒后台巡检定时器,杜绝超长运行环境下的内存增长。
223
+ - **并发环境下的精确延迟统计**:为每个请求独立计时,避免全局变量被并发请求覆盖导致 p50/p95 延迟失真。
224
+
217
225
  ### v0.7.36
218
226
  - **架构精简与稳定性加固**:移除 6 个过度设计的模块(`shadow`、`incident`、`agent-budget`、`region`、`canary`、`maintenance`)与废弃端点。
219
227
  - **buildRuntime 高性能记忆化**:消除每个流式 token/chunk 上的深拷贝与模式重解析开销。
package/lib/index.js CHANGED
@@ -32,6 +32,7 @@
32
32
  // cooldownMs: number key cooldown after a switchable failure
33
33
  // providers: array [{ provider, keys: [envName, ...] }]
34
34
  // ─────────────────────────────────────────────────────────────────────────────
35
+ import { AsyncLocalStorage } from 'node:async_hooks';
35
36
  import Schema from '@deepseek-ai/schemastery';
36
37
  import { keyTail, isLoopbackAddress, isTrustedBridgeRequest, SWITCHABLE_MESSAGE_PATTERN, DEFAULT_SWITCH_CODES, isValidRef, pickNext, applyCooldown, recordFailure, recordSuccess, computeBackoff, envValue, sweepExpired, parseRetryAfter, computeHealthScore, extractRateLimit, isRateLimited, selectPool, isSwitchableError, formatExhaustionMessage, expiringSoon, shouldNotifyDaily, costForDay, costForWeek, budgetVerdict } from './pool.js';
37
38
 
@@ -61,7 +62,9 @@ import { nextQuotaReset } from './quota-window.js';
61
62
  import { QuotaStore } from './quota.js';
62
63
  import { WebhookSender } from './webhook.js';
63
64
  import { bucketAllow, bucketRetryMs, bucketSweep, bucketInfo } from './bucket.js';
64
- import { usageRows, usageCsv } from './usage-report.js';
65
+ import { usageRows, usageCsv, compactUsage } from './usage-report.js';
66
+
67
+ const dispatchStorage = new AsyncLocalStorage();
65
68
  import { findSecrets, looksLikeApiSecret } from './keycheck.js';
66
69
 
67
70
  /** The llm-pi-ai namespace whose provider profiles map providers to pools. */
@@ -449,6 +452,9 @@ export function apply(ctx, config = {}) {
449
452
  }
450
453
  const n = sweepExpired(poolState, now);
451
454
  if (n > 0) console.warn(`[dsh-key-rotation] sweep: cleared ${n} expired cooldown(s)`);
455
+ for (const pool of buildRuntime().poolByRef.values()) {
456
+ compactUsage(pool, 30, now);
457
+ }
452
458
  // #207 expiry pre-warning + #208 cost budget - piggybacked on this timer,
453
459
  // deduped to one notification per key/window per day (shouldNotifyDaily).
454
460
  try {
@@ -761,6 +767,8 @@ export function apply(ctx, config = {}) {
761
767
  let hit = await original(candidate);
762
768
  if (hit && typeof hit.value === 'string' && hit.value.length > 0) {
763
769
  pool.state.lastUsed = candidate;
770
+ const store = dispatchStorage.getStore();
771
+ if (store && store.pool === pool) store.pickedRef = candidate;
764
772
  if (pool.state.failCounts) pool.state.failCounts.delete(candidate);
765
773
  pool.state.failedUntil.delete(candidate);
766
774
  if (pool.state.authFailCounts) pool.state.authFailCounts.delete(candidate);
@@ -780,6 +788,8 @@ export function apply(ctx, config = {}) {
780
788
  const envVal = envValue(candidate);
781
789
  if (envVal !== undefined) {
782
790
  pool.state.lastUsed = candidate;
791
+ const store = dispatchStorage.getStore();
792
+ if (store && store.pool === pool) store.pickedRef = candidate;
783
793
  if (pool.state.failCounts) pool.state.failCounts.delete(candidate);
784
794
  pool.state.failedUntil.delete(candidate);
785
795
  if (pool.state.authFailCounts) pool.state.authFailCounts.delete(candidate);
@@ -817,13 +827,14 @@ export function apply(ctx, config = {}) {
817
827
  // Latency recording (#6): record successful llm/stream latency per ref.
818
828
  // ponytail: only the true success path (finish-chunk). Failures are not recorded.
819
829
  let _rotateStartMs = Date.now();
820
- function recordLatency(pool) {
830
+ function recordLatency(pool, reqStore) {
821
831
  try {
822
832
  const cfg = getConfig();
823
833
  if (!cfg || cfg.latencyEnabled === false) return;
824
- const ref = pool && pool.state && pool.state.lastUsed;
834
+ const ref = reqStore?.pickedRef ?? pool?.state?.lastUsed;
825
835
  if (!ref) return;
826
- const elapsed = Date.now() - _rotateStartMs;
836
+ const startMs = reqStore?.startMs ?? _rotateStartMs;
837
+ const elapsed = Date.now() - startMs;
827
838
  if (!Number.isFinite(elapsed) || elapsed < 0) return;
828
839
  latencyHistogram.record(ref, elapsed);
829
840
  } catch (_) { /* ponytail: never crash */ }
@@ -836,12 +847,14 @@ export function apply(ctx, config = {}) {
836
847
  return (async function* () {
837
848
  const { switchCodes, cooldownMs, maxCooldownMs } = buildRuntime();
838
849
  let lastFailure = null;
839
- _rotateStartMs = Date.now();
850
+ const reqStore = { pool, pickedRef: undefined, startMs: Date.now() };
851
+ _rotateStartMs = reqStore.startMs;
840
852
 
841
853
  const runtime0 = buildRuntime();
854
+ let attemptList = (pool.weightedRefs ?? pool.refs).slice();
842
855
  if (runtime0.concurrencyLimit > 0 && concurrencyTracker.isEnabled()) {
843
856
  // #193: prefer least-loaded key within limit
844
- const available = (pool.weightedRefs ?? pool.refs).filter((r) => {
857
+ const available = attemptList.filter((r) => {
845
858
  const fu = pool.state.failedUntil.get(r) ?? 0;
846
859
  if (fu > Date.now()) return false;
847
860
  const exp = pool.expiresAt ? pool.expiresAt[r] : undefined;
@@ -849,30 +862,54 @@ export function apply(ctx, config = {}) {
849
862
  return true;
850
863
  });
851
864
  const preferred = concurrencyTracker.pickLeastLoaded(available);
852
- if (preferred && (pool.weightedRefs ?? pool.refs)[0] !== preferred) {
853
- // Move preferred to front of the attempt list
854
- const list = (pool.weightedRefs ?? pool.refs).slice();
865
+ if (preferred && attemptList[0] !== preferred) {
866
+ const list = attemptList.slice();
855
867
  const i = list.indexOf(preferred);
856
868
  if (i > 0) { list.splice(i, 1); list.unshift(preferred); }
857
- pool.weightedRefs = list;
869
+ attemptList = list;
858
870
  }
859
871
  }
860
- for (let attempt = 0; attempt < (pool.weightedRefs ?? pool.refs).length; attempt++) {
872
+
873
+ const penalizeRef = (targetRef, errCode, errMsg) => {
874
+ if (!targetRef) return;
875
+ const _retry = parseRetryAfter(errMsg);
876
+ const _base = pool.cooldownMs ?? cooldownMs;
877
+ const _max = pool.maxCooldownMs ?? maxCooldownMs;
878
+ const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
879
+ const _b = recordFailure(pool, targetRef, Date.now(), _effBase, _max);
880
+ pushEvent(pool, targetRef, errCode ?? 'UNKNOWN', _b);
881
+ if (!pool.state.authFailCounts) pool.state.authFailCounts = new Map();
882
+ if (!pool.state.brokenUntil) pool.state.brokenUntil = new Map();
883
+ const _cStr = String(errCode ?? '');
884
+ if (_cStr === 'AUTH' || /auth/i.test(errMsg)) {
885
+ const _c2 = (pool.state.authFailCounts.get(targetRef) ?? 0) + 1;
886
+ pool.state.authFailCounts.set(targetRef, _c2);
887
+ if (_c2 >= 3) {
888
+ pool.state.brokenUntil.set(targetRef, Date.now() + 86400000 * 30);
889
+ pool.state.failedUntil.set(targetRef, Date.now() + 86400000 * 30);
890
+ }
891
+ } else {
892
+ pool.state.authFailCounts.delete(targetRef);
893
+ }
894
+ };
895
+
896
+ for (let attempt = 0; attempt < attemptList.length; attempt++) {
861
897
  let yielded = false;
862
898
  let switching = false;
863
899
  let inner;
864
900
  try {
865
901
  // mark the internal dispatch so the interceptor does not re-rotate
866
- inner = ctx.llm.stream({ ...options, [MARKER]: true });
902
+ inner = dispatchStorage.run(reqStore, () => ctx.llm.stream({ ...options, [MARKER]: true }));
867
903
  } catch (e) {
868
- if (pool.state.lastUsed) { const _retry = parseRetryAfter(String(e?.message ?? '')); const _base = pool.cooldownMs ?? cooldownMs; const _max = pool.maxCooldownMs ?? maxCooldownMs; const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base; const _b = recordFailure(pool, pool.state.lastUsed, Date.now(), _effBase, _max); pushEvent(pool, pool.state.lastUsed, e?.code ?? 'TRANSPORT', _b); const _code = String(e?.code ?? ''); if (_code === 'AUTH' || /auth/i.test(String(e?.message ?? ''))) { const _c = (pool.state.authFailCounts.get(pool.state.lastUsed) ?? 0) + 1; pool.state.authFailCounts.set(pool.state.lastUsed, _c); if (_c >= 3) { pool.state.brokenUntil.set(pool.state.lastUsed, Date.now() + 86400000*30); pool.state.failedUntil.set(pool.state.lastUsed, Date.now() + 86400000*30); } } else { pool.state.authFailCounts.delete(pool.state.lastUsed); } }
904
+ const curRef = reqStore.pickedRef ?? pool.state.lastUsed;
905
+ penalizeRef(curRef, e?.code ?? 'TRANSPORT', String(e?.message ?? ''));
869
906
  lastFailure = finishError(e?.code ?? 'TRANSPORT',
870
907
  `dsh-key-rotation: dispatch failed: ${String(e?.message ?? e)}`);
871
- console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(pool.state.lastUsed ?? '?')} threw ${String(e?.code ?? e?.message ?? e)}`);
908
+ console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(curRef ?? '?')} threw ${String(e?.code ?? e?.message ?? e)}`);
872
909
  continue;
873
910
  }
874
911
 
875
- const _pickedRef = pool.state.lastUsed;
912
+ const _pickedRef = reqStore.pickedRef ?? pool.state.lastUsed;
876
913
  if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.acquire(_pickedRef);
877
914
  try {
878
915
  for await (const chunk of inner) {
@@ -890,39 +927,19 @@ export function apply(ctx, config = {}) {
890
927
  const message = failure?.message ?? '';
891
928
  const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
892
929
  const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
930
+ const activeRef = reqStore.pickedRef ?? pool.state.lastUsed;
893
931
  if (switchable) {
894
- if (pool.state.lastUsed) {
895
- const _retry = parseRetryAfter(message);
896
- const _base = pool.cooldownMs ?? cooldownMs;
897
- const _max = pool.maxCooldownMs ?? maxCooldownMs;
898
- const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
899
- const _b = recordFailure(pool, pool.state.lastUsed, Date.now(), _effBase, _max);
900
- pushEvent(pool, pool.state.lastUsed, code ?? 'UNKNOWN', _b);
901
- // authFailCounts/brokenUntil: lazy-init if state was created by an older plugin version
902
- if (!pool.state.authFailCounts) pool.state.authFailCounts = new Map();
903
- if (!pool.state.brokenUntil) pool.state.brokenUntil = new Map();
904
- const _code2 = String(code ?? '');
905
- if (_code2 === 'AUTH' || /auth/i.test(message)) {
906
- const _c2 = (pool.state.authFailCounts.get(pool.state.lastUsed) ?? 0) + 1;
907
- pool.state.authFailCounts.set(pool.state.lastUsed, _c2);
908
- if (_c2 >= 3) {
909
- pool.state.brokenUntil.set(pool.state.lastUsed, Date.now() + 86400000*30);
910
- pool.state.failedUntil.set(pool.state.lastUsed, Date.now() + 86400000*30);
911
- }
912
- } else {
913
- pool.state.authFailCounts.delete(pool.state.lastUsed);
914
- }
915
- }
932
+ penalizeRef(activeRef, code ?? 'UNKNOWN', message);
916
933
  pool.state.switches = (pool.state.switches ?? 0) + 1;
917
934
  pool.state.lastReason = String(code ?? 'UNKNOWN');
918
935
  pool.state.lastSwitchAt = Date.now();
919
936
  lastFailure = chunk;
920
- console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(pool.state.lastUsed ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
937
+ console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
921
938
  // #216: per-switch webhook (opt-in switchNotify), deduped per provider
922
- if (buildRuntime().switchNotify && pool.state.lastUsed) {
939
+ if (buildRuntime().switchNotify && activeRef) {
923
940
  notifySwitch(buildRuntime(), pool, {
924
941
  provider: options.provider,
925
- from: pool.state.lastUsed,
942
+ from: activeRef,
926
943
  code: String(code ?? 'UNKNOWN'),
927
944
  at: pool.state.lastSwitchAt,
928
945
  });
@@ -931,59 +948,78 @@ export function apply(ctx, config = {}) {
931
948
  break;
932
949
  }
933
950
  // cost tracking if provider returns usage.cost
934
- if (chunk.usage?.cost != null && pool.state.lastUsed) {
951
+ if (chunk.usage?.cost != null && activeRef) {
935
952
  const c = Number(chunk.usage.cost);
936
953
  if (!isNaN(c)) {
937
954
  if (!pool.state.costPerKey) pool.state.costPerKey = new Map();
938
- pool.state.costPerKey.set(pool.state.lastUsed, (pool.state.costPerKey.get(pool.state.lastUsed) ?? 0) + c);
955
+ pool.state.costPerKey.set(activeRef, (pool.state.costPerKey.get(activeRef) ?? 0) + c);
939
956
  // #208: cost per day per key (mirrors usageDays) for budget checks
940
957
  if (!pool.state.costDays) pool.state.costDays = new Map();
941
958
  const cday = new Date().toISOString().slice(0, 10);
942
- const cMap = pool.state.costDays.get(pool.state.lastUsed) || new Map();
959
+ const cMap = pool.state.costDays.get(activeRef) || new Map();
943
960
  cMap.set(cday, (cMap.get(cday) ?? 0) + c);
944
- pool.state.costDays.set(pool.state.lastUsed, cMap);
961
+ pool.state.costDays.set(activeRef, cMap);
945
962
  }
946
963
  }
947
964
  // Usage by day (#119)
948
- if (pool.state.lastUsed) {
965
+ if (activeRef) {
949
966
  if (!pool.state.usageDays) pool.state.usageDays = new Map();
950
967
  const day = new Date().toISOString().slice(0, 10);
951
- const dayMap = pool.state.usageDays.get(pool.state.lastUsed) || new Map();
968
+ const dayMap = pool.state.usageDays.get(activeRef) || new Map();
952
969
  dayMap.set(day, (dayMap.get(day) ?? 0) + 1);
953
- pool.state.usageDays.set(pool.state.lastUsed, dayMap);
970
+ pool.state.usageDays.set(activeRef, dayMap);
954
971
  }
955
972
  // Per-model request detail (#121)
956
- if (pool.state.lastUsed && options.model) {
973
+ if (activeRef && options.model) {
957
974
  if (!pool.state.byModel) pool.state.byModel = new Map();
958
- let byRef = pool.state.byModel.get(pool.state.lastUsed);
959
- if (!byRef) { byRef = new Map(); pool.state.byModel.set(pool.state.lastUsed, byRef); }
975
+ let byRef = pool.state.byModel.get(activeRef);
976
+ if (!byRef) { byRef = new Map(); pool.state.byModel.set(activeRef, byRef); }
960
977
  byRef.set(options.model, (byRef.get(options.model) ?? 0) + 1);
961
978
  }
962
979
  // Proactive rate-limit (#115): if response headers say this key is near
963
980
  // its quota, cool it down so the NEXT request starts on a different key.
964
981
  // We do NOT re-run this (already successful) request — that would double-send.
965
982
  const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
966
- if (rate && pool.state.lastUsed) {
983
+ if (rate && activeRef) {
967
984
  const { rateLimitThreshold } = buildRuntime();
968
985
  if (isRateLimited(rate, rateLimitThreshold ?? 0.1)) {
969
986
  const cool = rate.reset && rate.reset > Date.now() ? (rate.reset - Date.now()) : pool.cooldownMs;
970
- recordFailure(pool, pool.state.lastUsed, Date.now(), cool, pool.maxCooldownMs);
971
- pushEvent(pool, pool.state.lastUsed, 'RATE_LIMIT', cool);
972
- console.warn(`[dsh-key-rotation] ${options.provider}: key ${pool.state.lastUsed} near quota (remaining ${String(rate.remaining)}/${String(rate.limit)}) — next request will rotate`);
987
+ recordFailure(pool, activeRef, Date.now(), cool, pool.maxCooldownMs);
988
+ pushEvent(pool, activeRef, 'RATE_LIMIT', cool);
989
+ console.warn(`[dsh-key-rotation] ${options.provider}: key ${activeRef} near quota (remaining ${String(rate.remaining)}/${String(rate.limit)}) — next request will rotate`);
973
990
  }
974
991
  }
975
992
  // #7: persist quota snapshot regardless of threshold (so dashboard widget can show it).
976
- if (rate && pool.state.lastUsed && Number.isFinite(rate.remaining)) {
977
- quotaStore.set(pool.state.lastUsed, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: Date.now() });
993
+ if (rate && activeRef && Number.isFinite(rate.remaining)) {
994
+ quotaStore.set(activeRef, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: Date.now() });
978
995
  }
979
996
  yield chunk;
980
- recordLatency(pool);
997
+ recordLatency(pool, reqStore);
981
998
  return;
982
999
  }
983
1000
  yield chunk;
984
1001
  }
985
1002
  } catch (e) {
986
1003
  if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.release(_pickedRef);
1004
+ const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
1005
+ const activeRef = _pickedRef ?? reqStore.pickedRef ?? pool.state.lastUsed;
1006
+ if (!yielded && isSwitchableError(e, effectiveSwitchCodes)) {
1007
+ penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
1008
+ pool.state.switches = (pool.state.switches ?? 0) + 1;
1009
+ pool.state.lastReason = String(e?.code ?? 'TRANSPORT');
1010
+ pool.state.lastSwitchAt = Date.now();
1011
+ lastFailure = finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
1012
+ console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} stream threw ${String(e?.code ?? e?.message ?? e)} - failover to next key`);
1013
+ if (buildRuntime().switchNotify && activeRef) {
1014
+ notifySwitch(buildRuntime(), pool, {
1015
+ provider: options.provider,
1016
+ from: activeRef,
1017
+ code: String(e?.code ?? 'TRANSPORT'),
1018
+ at: pool.state.lastSwitchAt,
1019
+ });
1020
+ }
1021
+ continue; // Failover to next key!
1022
+ }
987
1023
  yield finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
988
1024
  return;
989
1025
  }
@@ -1435,6 +1471,16 @@ export function apply(ctx, config = {}) {
1435
1471
  const result = probe === 'chat' ? await runner.probeChat(ref, keyForProbe) : await runner.probeModels(ref, keyForProbe);
1436
1472
  const cached = { ...result, at: Date.now() };
1437
1473
  lastTestCache.set(ref, cached);
1474
+ if (cached.ok) {
1475
+ for (const st of poolState.values()) {
1476
+ if (st.failedUntil?.has(ref) || st.failCounts?.has(ref) || st.brokenUntil?.has(ref)) {
1477
+ st.failedUntil?.delete(ref);
1478
+ st.failCounts?.delete(ref);
1479
+ st.authFailCounts?.delete(ref);
1480
+ st.brokenUntil?.delete(ref);
1481
+ }
1482
+ }
1483
+ }
1438
1484
  json(res, 200, { ok: cached.ok, ref, tail, source, probe, code: cached.code, latencyMs: cached.latencyMs, modelsCount: cached.modelsCount });
1439
1485
  return;
1440
1486
  }
package/package.json CHANGED
@@ -1,6 +1,7 @@
1
1
  {
2
2
  "name": "@goodandready/dsh-key-rotation",
3
- "version": "0.7.36",
3
+ "version": "0.7.37",
4
+ "packageManager": "pnpm@10.33.2",
4
5
  "description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
5
6
  "keywords": [
6
7
  "deepseek-harness",
@@ -55,4 +56,4 @@
55
56
  "scripts": {
56
57
  "test": "node --test test/*.test.js test/*.test.mjs"
57
58
  }
58
- }
59
+ }