@goodandready/dsh-key-rotation 0.7.36 → 0.7.37
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -0
- package/README.ru.md +8 -0
- package/README.zh.md +8 -0
- package/lib/index.js +103 -57
- package/package.json +3 -2
package/README.md
CHANGED
|
@@ -253,6 +253,14 @@ All management routes require loopback authentication (`127.0.0.1` / `::1`) with
|
|
|
253
253
|
|
|
254
254
|
MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
|
|
255
255
|
|
|
256
|
+
### v0.7.37
|
|
257
|
+
- **Request Context Isolation via AsyncLocalStorage**: Scoped active key resolution (`pickedRef`), start timestamps, and retry counts strictly to each async dispatch context using Node's `node:async_hooks`. Eliminates race conditions where concurrent streaming requests could penalize healthy keys.
|
|
258
|
+
- **Failover on Pre-Yield Stream Exceptions**: Fixed fatal stream termination where transport errors (e.g. HTTP 429 thrown before headers, socket hang-up) aborted the generator. The catch block now inspects `isSwitchableError` and cascades seamlessly to the next key if no content tokens have been yielded.
|
|
259
|
+
- **Pure Local Candidate List in rotate()**: Generator uses an isolated local slice of keys (`attemptList`), eliminating shared mutations on `pool.weightedRefs`.
|
|
260
|
+
- **Automatic Quarantine Release on Probe Success**: Successful sandbox model tests via Settings UI (`/dsh-key-rotation/test`) automatically lift `failedUntil` and `brokenUntil` quarantine flags.
|
|
261
|
+
- **Long-Term Memory Compaction**: Wired `compactUsage(pool, 30, now)` into the periodic 30-second maintenance sweep to prevent memory growth on high-uptime servers.
|
|
262
|
+
- **Request-Scoped Latency Recording**: Replaced module-global start timestamp with context-scoped `startMs` for accurate p50/p95 latency metrics under concurrent load.
|
|
263
|
+
|
|
256
264
|
### v0.7.36
|
|
257
265
|
- **Architecture de-bloat & hardening**: Removed 6 unused/overengineered modules (`shadow`, `incident`, `agent-budget`, `region`, `canary`, `maintenance`) and dead route registrations.
|
|
258
266
|
- **High-throughput buildRuntime memoization**: Eliminates per-token deep-cloning and schema validation on every streaming chunk.
|
package/README.ru.md
CHANGED
|
@@ -253,6 +253,14 @@ dsh-key-rotation:
|
|
|
253
253
|
|
|
254
254
|
MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
|
|
255
255
|
|
|
256
|
+
### v0.7.37
|
|
257
|
+
- **Изоляция контекста запросов через AsyncLocalStorage**: Полная привязка активного ключа (`pickedRef`), времени старта и попыток к асинхронному контексту вызова через `node:async_hooks`. Устранена гонка, при которой параллельные запросы могли ошибочно штрафовать здоровый ключ соседа.
|
|
258
|
+
- **Failover при исключениях в потоке до первого чанка**: Устранено аварийное прерывание потока при транспортных сбоях (например, выброс HTTP 429 до отправки заголовков). Теперь блок перехвата проверяет `isSwitchableError` и прозрачно переключает поток на запасной ключ, если клиенту ещё не было отдано полезных данных.
|
|
259
|
+
- **Чистый локальный список кандидатов в rotate()**: Устранена прямая мутация разделяемого массива `pool.weightedRefs` во время выполнения ротации.
|
|
260
|
+
- **Автоматический вывод из карантина при успехе проверки**: Успешная проверка ключа через песочницу в настройках (`/dsh-key-rotation/test`) моментально сбрасывает флаги карантина (`failedUntil` и `brokenUntil`).
|
|
261
|
+
- **Регулярная компактизация памяти**: Функция `compactUsage(pool, 30, now)` подключена в 30-секундный фоновый таймер пула, предотвращая утечки памяти при непрерывной многомесячной работе.
|
|
262
|
+
- **Точные метрики задержки при параллельных вызовах**: Переход с глобальной переменной времени старта на контекстный замер `startMs` гарантирует достоверность p50/p95 latency под нагрузкой.
|
|
263
|
+
|
|
256
264
|
### v0.7.36
|
|
257
265
|
- **Глубокий рефакторинг и деблоатинг**: Удалены 6 неиспользуемых модулей оверинжиниринга (`shadow`, `incident`, `agent-budget`, `region`, `canary`, `maintenance`) и сопутствующие устаревшие эндпоинты.
|
|
258
266
|
- **Мемоизация buildRuntime**: Устранено глубокое клонирование и повторный парсинг схемы на каждом токене/чанге стрима.
|
package/README.zh.md
CHANGED
|
@@ -214,6 +214,14 @@ dsh-key-rotation:
|
|
|
214
214
|
|
|
215
215
|
MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)
|
|
216
216
|
|
|
217
|
+
### v0.7.37
|
|
218
|
+
- **通过 AsyncLocalStorage 隔离请求上下文**:使用 Node.js 的 `node:async_hooks` 将解析后的密钥 (`pickedRef`)、启动时间和重试严格限定在单个请求上下文内,彻底消除并发请求间的竞态条件与误罚。
|
|
219
|
+
- **流异常自动故障转移**:修复流在首个 token 返回前抛出传输异常(如 HTTP 429)直接终止的问题。若未发送内容块,现在会自动触发 `isSwitchableError` 并顺畅切换到备用密钥。
|
|
220
|
+
- **避免在 rotate() 中直接修改共享状态**:遍历候选列表改用纯净的局部切片,不再直接覆盖修改 `pool.weightedRefs`。
|
|
221
|
+
- **测试成功后自动解除隔离**:在设置界面通过沙箱成功验证密钥有效性后,自动清除 `failedUntil` 和 `brokenUntil` 惩罚标记。
|
|
222
|
+
- **定期内存压缩清理**:将 `compactUsage(pool, 30, now)` 接入 30 秒后台巡检定时器,杜绝超长运行环境下的内存增长。
|
|
223
|
+
- **并发环境下的精确延迟统计**:为每个请求独立计时,避免全局变量被并发请求覆盖导致 p50/p95 延迟失真。
|
|
224
|
+
|
|
217
225
|
### v0.7.36
|
|
218
226
|
- **架构精简与稳定性加固**:移除 6 个过度设计的模块(`shadow`、`incident`、`agent-budget`、`region`、`canary`、`maintenance`)与废弃端点。
|
|
219
227
|
- **buildRuntime 高性能记忆化**:消除每个流式 token/chunk 上的深拷贝与模式重解析开销。
|
package/lib/index.js
CHANGED
|
@@ -32,6 +32,7 @@
|
|
|
32
32
|
// cooldownMs: number key cooldown after a switchable failure
|
|
33
33
|
// providers: array [{ provider, keys: [envName, ...] }]
|
|
34
34
|
// ─────────────────────────────────────────────────────────────────────────────
|
|
35
|
+
import { AsyncLocalStorage } from 'node:async_hooks';
|
|
35
36
|
import Schema from '@deepseek-ai/schemastery';
|
|
36
37
|
import { keyTail, isLoopbackAddress, isTrustedBridgeRequest, SWITCHABLE_MESSAGE_PATTERN, DEFAULT_SWITCH_CODES, isValidRef, pickNext, applyCooldown, recordFailure, recordSuccess, computeBackoff, envValue, sweepExpired, parseRetryAfter, computeHealthScore, extractRateLimit, isRateLimited, selectPool, isSwitchableError, formatExhaustionMessage, expiringSoon, shouldNotifyDaily, costForDay, costForWeek, budgetVerdict } from './pool.js';
|
|
37
38
|
|
|
@@ -61,7 +62,9 @@ import { nextQuotaReset } from './quota-window.js';
|
|
|
61
62
|
import { QuotaStore } from './quota.js';
|
|
62
63
|
import { WebhookSender } from './webhook.js';
|
|
63
64
|
import { bucketAllow, bucketRetryMs, bucketSweep, bucketInfo } from './bucket.js';
|
|
64
|
-
import { usageRows, usageCsv } from './usage-report.js';
|
|
65
|
+
import { usageRows, usageCsv, compactUsage } from './usage-report.js';
|
|
66
|
+
|
|
67
|
+
const dispatchStorage = new AsyncLocalStorage();
|
|
65
68
|
import { findSecrets, looksLikeApiSecret } from './keycheck.js';
|
|
66
69
|
|
|
67
70
|
/** The llm-pi-ai namespace whose provider profiles map providers to pools. */
|
|
@@ -449,6 +452,9 @@ export function apply(ctx, config = {}) {
|
|
|
449
452
|
}
|
|
450
453
|
const n = sweepExpired(poolState, now);
|
|
451
454
|
if (n > 0) console.warn(`[dsh-key-rotation] sweep: cleared ${n} expired cooldown(s)`);
|
|
455
|
+
for (const pool of buildRuntime().poolByRef.values()) {
|
|
456
|
+
compactUsage(pool, 30, now);
|
|
457
|
+
}
|
|
452
458
|
// #207 expiry pre-warning + #208 cost budget - piggybacked on this timer,
|
|
453
459
|
// deduped to one notification per key/window per day (shouldNotifyDaily).
|
|
454
460
|
try {
|
|
@@ -761,6 +767,8 @@ export function apply(ctx, config = {}) {
|
|
|
761
767
|
let hit = await original(candidate);
|
|
762
768
|
if (hit && typeof hit.value === 'string' && hit.value.length > 0) {
|
|
763
769
|
pool.state.lastUsed = candidate;
|
|
770
|
+
const store = dispatchStorage.getStore();
|
|
771
|
+
if (store && store.pool === pool) store.pickedRef = candidate;
|
|
764
772
|
if (pool.state.failCounts) pool.state.failCounts.delete(candidate);
|
|
765
773
|
pool.state.failedUntil.delete(candidate);
|
|
766
774
|
if (pool.state.authFailCounts) pool.state.authFailCounts.delete(candidate);
|
|
@@ -780,6 +788,8 @@ export function apply(ctx, config = {}) {
|
|
|
780
788
|
const envVal = envValue(candidate);
|
|
781
789
|
if (envVal !== undefined) {
|
|
782
790
|
pool.state.lastUsed = candidate;
|
|
791
|
+
const store = dispatchStorage.getStore();
|
|
792
|
+
if (store && store.pool === pool) store.pickedRef = candidate;
|
|
783
793
|
if (pool.state.failCounts) pool.state.failCounts.delete(candidate);
|
|
784
794
|
pool.state.failedUntil.delete(candidate);
|
|
785
795
|
if (pool.state.authFailCounts) pool.state.authFailCounts.delete(candidate);
|
|
@@ -817,13 +827,14 @@ export function apply(ctx, config = {}) {
|
|
|
817
827
|
// Latency recording (#6): record successful llm/stream latency per ref.
|
|
818
828
|
// ponytail: only the true success path (finish-chunk). Failures are not recorded.
|
|
819
829
|
let _rotateStartMs = Date.now();
|
|
820
|
-
function recordLatency(pool) {
|
|
830
|
+
function recordLatency(pool, reqStore) {
|
|
821
831
|
try {
|
|
822
832
|
const cfg = getConfig();
|
|
823
833
|
if (!cfg || cfg.latencyEnabled === false) return;
|
|
824
|
-
const ref =
|
|
834
|
+
const ref = reqStore?.pickedRef ?? pool?.state?.lastUsed;
|
|
825
835
|
if (!ref) return;
|
|
826
|
-
const
|
|
836
|
+
const startMs = reqStore?.startMs ?? _rotateStartMs;
|
|
837
|
+
const elapsed = Date.now() - startMs;
|
|
827
838
|
if (!Number.isFinite(elapsed) || elapsed < 0) return;
|
|
828
839
|
latencyHistogram.record(ref, elapsed);
|
|
829
840
|
} catch (_) { /* ponytail: never crash */ }
|
|
@@ -836,12 +847,14 @@ export function apply(ctx, config = {}) {
|
|
|
836
847
|
return (async function* () {
|
|
837
848
|
const { switchCodes, cooldownMs, maxCooldownMs } = buildRuntime();
|
|
838
849
|
let lastFailure = null;
|
|
839
|
-
|
|
850
|
+
const reqStore = { pool, pickedRef: undefined, startMs: Date.now() };
|
|
851
|
+
_rotateStartMs = reqStore.startMs;
|
|
840
852
|
|
|
841
853
|
const runtime0 = buildRuntime();
|
|
854
|
+
let attemptList = (pool.weightedRefs ?? pool.refs).slice();
|
|
842
855
|
if (runtime0.concurrencyLimit > 0 && concurrencyTracker.isEnabled()) {
|
|
843
856
|
// #193: prefer least-loaded key within limit
|
|
844
|
-
const available =
|
|
857
|
+
const available = attemptList.filter((r) => {
|
|
845
858
|
const fu = pool.state.failedUntil.get(r) ?? 0;
|
|
846
859
|
if (fu > Date.now()) return false;
|
|
847
860
|
const exp = pool.expiresAt ? pool.expiresAt[r] : undefined;
|
|
@@ -849,30 +862,54 @@ export function apply(ctx, config = {}) {
|
|
|
849
862
|
return true;
|
|
850
863
|
});
|
|
851
864
|
const preferred = concurrencyTracker.pickLeastLoaded(available);
|
|
852
|
-
if (preferred &&
|
|
853
|
-
|
|
854
|
-
const list = (pool.weightedRefs ?? pool.refs).slice();
|
|
865
|
+
if (preferred && attemptList[0] !== preferred) {
|
|
866
|
+
const list = attemptList.slice();
|
|
855
867
|
const i = list.indexOf(preferred);
|
|
856
868
|
if (i > 0) { list.splice(i, 1); list.unshift(preferred); }
|
|
857
|
-
|
|
869
|
+
attemptList = list;
|
|
858
870
|
}
|
|
859
871
|
}
|
|
860
|
-
|
|
872
|
+
|
|
873
|
+
const penalizeRef = (targetRef, errCode, errMsg) => {
|
|
874
|
+
if (!targetRef) return;
|
|
875
|
+
const _retry = parseRetryAfter(errMsg);
|
|
876
|
+
const _base = pool.cooldownMs ?? cooldownMs;
|
|
877
|
+
const _max = pool.maxCooldownMs ?? maxCooldownMs;
|
|
878
|
+
const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
|
|
879
|
+
const _b = recordFailure(pool, targetRef, Date.now(), _effBase, _max);
|
|
880
|
+
pushEvent(pool, targetRef, errCode ?? 'UNKNOWN', _b);
|
|
881
|
+
if (!pool.state.authFailCounts) pool.state.authFailCounts = new Map();
|
|
882
|
+
if (!pool.state.brokenUntil) pool.state.brokenUntil = new Map();
|
|
883
|
+
const _cStr = String(errCode ?? '');
|
|
884
|
+
if (_cStr === 'AUTH' || /auth/i.test(errMsg)) {
|
|
885
|
+
const _c2 = (pool.state.authFailCounts.get(targetRef) ?? 0) + 1;
|
|
886
|
+
pool.state.authFailCounts.set(targetRef, _c2);
|
|
887
|
+
if (_c2 >= 3) {
|
|
888
|
+
pool.state.brokenUntil.set(targetRef, Date.now() + 86400000 * 30);
|
|
889
|
+
pool.state.failedUntil.set(targetRef, Date.now() + 86400000 * 30);
|
|
890
|
+
}
|
|
891
|
+
} else {
|
|
892
|
+
pool.state.authFailCounts.delete(targetRef);
|
|
893
|
+
}
|
|
894
|
+
};
|
|
895
|
+
|
|
896
|
+
for (let attempt = 0; attempt < attemptList.length; attempt++) {
|
|
861
897
|
let yielded = false;
|
|
862
898
|
let switching = false;
|
|
863
899
|
let inner;
|
|
864
900
|
try {
|
|
865
901
|
// mark the internal dispatch so the interceptor does not re-rotate
|
|
866
|
-
inner = ctx.llm.stream({ ...options, [MARKER]: true });
|
|
902
|
+
inner = dispatchStorage.run(reqStore, () => ctx.llm.stream({ ...options, [MARKER]: true }));
|
|
867
903
|
} catch (e) {
|
|
868
|
-
|
|
904
|
+
const curRef = reqStore.pickedRef ?? pool.state.lastUsed;
|
|
905
|
+
penalizeRef(curRef, e?.code ?? 'TRANSPORT', String(e?.message ?? ''));
|
|
869
906
|
lastFailure = finishError(e?.code ?? 'TRANSPORT',
|
|
870
907
|
`dsh-key-rotation: dispatch failed: ${String(e?.message ?? e)}`);
|
|
871
|
-
console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(
|
|
908
|
+
console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(curRef ?? '?')} threw ${String(e?.code ?? e?.message ?? e)}`);
|
|
872
909
|
continue;
|
|
873
910
|
}
|
|
874
911
|
|
|
875
|
-
const _pickedRef = pool.state.lastUsed;
|
|
912
|
+
const _pickedRef = reqStore.pickedRef ?? pool.state.lastUsed;
|
|
876
913
|
if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.acquire(_pickedRef);
|
|
877
914
|
try {
|
|
878
915
|
for await (const chunk of inner) {
|
|
@@ -890,39 +927,19 @@ export function apply(ctx, config = {}) {
|
|
|
890
927
|
const message = failure?.message ?? '';
|
|
891
928
|
const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
|
|
892
929
|
const switchable = !yielded && kind === 'error' && isSwitchableError(failure, effectiveSwitchCodes);
|
|
930
|
+
const activeRef = reqStore.pickedRef ?? pool.state.lastUsed;
|
|
893
931
|
if (switchable) {
|
|
894
|
-
|
|
895
|
-
const _retry = parseRetryAfter(message);
|
|
896
|
-
const _base = pool.cooldownMs ?? cooldownMs;
|
|
897
|
-
const _max = pool.maxCooldownMs ?? maxCooldownMs;
|
|
898
|
-
const _effBase = _retry !== undefined ? Math.max(_base, Math.min(_retry, _max ?? _base * 8)) : _base;
|
|
899
|
-
const _b = recordFailure(pool, pool.state.lastUsed, Date.now(), _effBase, _max);
|
|
900
|
-
pushEvent(pool, pool.state.lastUsed, code ?? 'UNKNOWN', _b);
|
|
901
|
-
// authFailCounts/brokenUntil: lazy-init if state was created by an older plugin version
|
|
902
|
-
if (!pool.state.authFailCounts) pool.state.authFailCounts = new Map();
|
|
903
|
-
if (!pool.state.brokenUntil) pool.state.brokenUntil = new Map();
|
|
904
|
-
const _code2 = String(code ?? '');
|
|
905
|
-
if (_code2 === 'AUTH' || /auth/i.test(message)) {
|
|
906
|
-
const _c2 = (pool.state.authFailCounts.get(pool.state.lastUsed) ?? 0) + 1;
|
|
907
|
-
pool.state.authFailCounts.set(pool.state.lastUsed, _c2);
|
|
908
|
-
if (_c2 >= 3) {
|
|
909
|
-
pool.state.brokenUntil.set(pool.state.lastUsed, Date.now() + 86400000*30);
|
|
910
|
-
pool.state.failedUntil.set(pool.state.lastUsed, Date.now() + 86400000*30);
|
|
911
|
-
}
|
|
912
|
-
} else {
|
|
913
|
-
pool.state.authFailCounts.delete(pool.state.lastUsed);
|
|
914
|
-
}
|
|
915
|
-
}
|
|
932
|
+
penalizeRef(activeRef, code ?? 'UNKNOWN', message);
|
|
916
933
|
pool.state.switches = (pool.state.switches ?? 0) + 1;
|
|
917
934
|
pool.state.lastReason = String(code ?? 'UNKNOWN');
|
|
918
935
|
pool.state.lastSwitchAt = Date.now();
|
|
919
936
|
lastFailure = chunk;
|
|
920
|
-
console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(
|
|
937
|
+
console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
|
|
921
938
|
// #216: per-switch webhook (opt-in switchNotify), deduped per provider
|
|
922
|
-
if (buildRuntime().switchNotify &&
|
|
939
|
+
if (buildRuntime().switchNotify && activeRef) {
|
|
923
940
|
notifySwitch(buildRuntime(), pool, {
|
|
924
941
|
provider: options.provider,
|
|
925
|
-
from:
|
|
942
|
+
from: activeRef,
|
|
926
943
|
code: String(code ?? 'UNKNOWN'),
|
|
927
944
|
at: pool.state.lastSwitchAt,
|
|
928
945
|
});
|
|
@@ -931,59 +948,78 @@ export function apply(ctx, config = {}) {
|
|
|
931
948
|
break;
|
|
932
949
|
}
|
|
933
950
|
// cost tracking if provider returns usage.cost
|
|
934
|
-
if (chunk.usage?.cost != null &&
|
|
951
|
+
if (chunk.usage?.cost != null && activeRef) {
|
|
935
952
|
const c = Number(chunk.usage.cost);
|
|
936
953
|
if (!isNaN(c)) {
|
|
937
954
|
if (!pool.state.costPerKey) pool.state.costPerKey = new Map();
|
|
938
|
-
pool.state.costPerKey.set(
|
|
955
|
+
pool.state.costPerKey.set(activeRef, (pool.state.costPerKey.get(activeRef) ?? 0) + c);
|
|
939
956
|
// #208: cost per day per key (mirrors usageDays) for budget checks
|
|
940
957
|
if (!pool.state.costDays) pool.state.costDays = new Map();
|
|
941
958
|
const cday = new Date().toISOString().slice(0, 10);
|
|
942
|
-
const cMap = pool.state.costDays.get(
|
|
959
|
+
const cMap = pool.state.costDays.get(activeRef) || new Map();
|
|
943
960
|
cMap.set(cday, (cMap.get(cday) ?? 0) + c);
|
|
944
|
-
pool.state.costDays.set(
|
|
961
|
+
pool.state.costDays.set(activeRef, cMap);
|
|
945
962
|
}
|
|
946
963
|
}
|
|
947
964
|
// Usage by day (#119)
|
|
948
|
-
if (
|
|
965
|
+
if (activeRef) {
|
|
949
966
|
if (!pool.state.usageDays) pool.state.usageDays = new Map();
|
|
950
967
|
const day = new Date().toISOString().slice(0, 10);
|
|
951
|
-
const dayMap = pool.state.usageDays.get(
|
|
968
|
+
const dayMap = pool.state.usageDays.get(activeRef) || new Map();
|
|
952
969
|
dayMap.set(day, (dayMap.get(day) ?? 0) + 1);
|
|
953
|
-
pool.state.usageDays.set(
|
|
970
|
+
pool.state.usageDays.set(activeRef, dayMap);
|
|
954
971
|
}
|
|
955
972
|
// Per-model request detail (#121)
|
|
956
|
-
if (
|
|
973
|
+
if (activeRef && options.model) {
|
|
957
974
|
if (!pool.state.byModel) pool.state.byModel = new Map();
|
|
958
|
-
let byRef = pool.state.byModel.get(
|
|
959
|
-
if (!byRef) { byRef = new Map(); pool.state.byModel.set(
|
|
975
|
+
let byRef = pool.state.byModel.get(activeRef);
|
|
976
|
+
if (!byRef) { byRef = new Map(); pool.state.byModel.set(activeRef, byRef); }
|
|
960
977
|
byRef.set(options.model, (byRef.get(options.model) ?? 0) + 1);
|
|
961
978
|
}
|
|
962
979
|
// Proactive rate-limit (#115): if response headers say this key is near
|
|
963
980
|
// its quota, cool it down so the NEXT request starts on a different key.
|
|
964
981
|
// We do NOT re-run this (already successful) request — that would double-send.
|
|
965
982
|
const rate = extractRateLimit(chunk?.metadata?.headers ?? chunk?.headers);
|
|
966
|
-
if (rate &&
|
|
983
|
+
if (rate && activeRef) {
|
|
967
984
|
const { rateLimitThreshold } = buildRuntime();
|
|
968
985
|
if (isRateLimited(rate, rateLimitThreshold ?? 0.1)) {
|
|
969
986
|
const cool = rate.reset && rate.reset > Date.now() ? (rate.reset - Date.now()) : pool.cooldownMs;
|
|
970
|
-
recordFailure(pool,
|
|
971
|
-
pushEvent(pool,
|
|
972
|
-
console.warn(`[dsh-key-rotation] ${options.provider}: key ${
|
|
987
|
+
recordFailure(pool, activeRef, Date.now(), cool, pool.maxCooldownMs);
|
|
988
|
+
pushEvent(pool, activeRef, 'RATE_LIMIT', cool);
|
|
989
|
+
console.warn(`[dsh-key-rotation] ${options.provider}: key ${activeRef} near quota (remaining ${String(rate.remaining)}/${String(rate.limit)}) — next request will rotate`);
|
|
973
990
|
}
|
|
974
991
|
}
|
|
975
992
|
// #7: persist quota snapshot regardless of threshold (so dashboard widget can show it).
|
|
976
|
-
if (rate &&
|
|
977
|
-
quotaStore.set(
|
|
993
|
+
if (rate && activeRef && Number.isFinite(rate.remaining)) {
|
|
994
|
+
quotaStore.set(activeRef, { remaining: rate.remaining, limit: rate.limit, reset: rate.reset, at: Date.now() });
|
|
978
995
|
}
|
|
979
996
|
yield chunk;
|
|
980
|
-
recordLatency(pool);
|
|
997
|
+
recordLatency(pool, reqStore);
|
|
981
998
|
return;
|
|
982
999
|
}
|
|
983
1000
|
yield chunk;
|
|
984
1001
|
}
|
|
985
1002
|
} catch (e) {
|
|
986
1003
|
if (_pickedRef && runtime0.concurrencyLimit > 0) concurrencyTracker.release(_pickedRef);
|
|
1004
|
+
const effectiveSwitchCodes = pool.switchCodes ?? switchCodes;
|
|
1005
|
+
const activeRef = _pickedRef ?? reqStore.pickedRef ?? pool.state.lastUsed;
|
|
1006
|
+
if (!yielded && isSwitchableError(e, effectiveSwitchCodes)) {
|
|
1007
|
+
penalizeRef(activeRef, e?.code ?? 'TRANSPORT', String(e?.message ?? e));
|
|
1008
|
+
pool.state.switches = (pool.state.switches ?? 0) + 1;
|
|
1009
|
+
pool.state.lastReason = String(e?.code ?? 'TRANSPORT');
|
|
1010
|
+
pool.state.lastSwitchAt = Date.now();
|
|
1011
|
+
lastFailure = finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
|
|
1012
|
+
console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(activeRef ?? '?')} stream threw ${String(e?.code ?? e?.message ?? e)} - failover to next key`);
|
|
1013
|
+
if (buildRuntime().switchNotify && activeRef) {
|
|
1014
|
+
notifySwitch(buildRuntime(), pool, {
|
|
1015
|
+
provider: options.provider,
|
|
1016
|
+
from: activeRef,
|
|
1017
|
+
code: String(e?.code ?? 'TRANSPORT'),
|
|
1018
|
+
at: pool.state.lastSwitchAt,
|
|
1019
|
+
});
|
|
1020
|
+
}
|
|
1021
|
+
continue; // Failover to next key!
|
|
1022
|
+
}
|
|
987
1023
|
yield finishError(e?.code ?? 'TRANSPORT', String(e?.message ?? e));
|
|
988
1024
|
return;
|
|
989
1025
|
}
|
|
@@ -1435,6 +1471,16 @@ export function apply(ctx, config = {}) {
|
|
|
1435
1471
|
const result = probe === 'chat' ? await runner.probeChat(ref, keyForProbe) : await runner.probeModels(ref, keyForProbe);
|
|
1436
1472
|
const cached = { ...result, at: Date.now() };
|
|
1437
1473
|
lastTestCache.set(ref, cached);
|
|
1474
|
+
if (cached.ok) {
|
|
1475
|
+
for (const st of poolState.values()) {
|
|
1476
|
+
if (st.failedUntil?.has(ref) || st.failCounts?.has(ref) || st.brokenUntil?.has(ref)) {
|
|
1477
|
+
st.failedUntil?.delete(ref);
|
|
1478
|
+
st.failCounts?.delete(ref);
|
|
1479
|
+
st.authFailCounts?.delete(ref);
|
|
1480
|
+
st.brokenUntil?.delete(ref);
|
|
1481
|
+
}
|
|
1482
|
+
}
|
|
1483
|
+
}
|
|
1438
1484
|
json(res, 200, { ok: cached.ok, ref, tail, source, probe, code: cached.code, latencyMs: cached.latencyMs, modelsCount: cached.modelsCount });
|
|
1439
1485
|
return;
|
|
1440
1486
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@goodandready/dsh-key-rotation",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.37",
|
|
4
|
+
"packageManager": "pnpm@10.33.2",
|
|
4
5
|
"description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
|
|
5
6
|
"keywords": [
|
|
6
7
|
"deepseek-harness",
|
|
@@ -55,4 +56,4 @@
|
|
|
55
56
|
"scripts": {
|
|
56
57
|
"test": "node --test test/*.test.js test/*.test.mjs"
|
|
57
58
|
}
|
|
58
|
-
}
|
|
59
|
+
}
|