@goodandready/dsh-key-rotation 0.7.27 → 0.7.29
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -0
- package/lib/bucket.js +11 -0
- package/lib/client.js +136 -4
- package/lib/index.js +223 -5
- package/lib/maintenance.js +64 -0
- package/lib/usage-report.js +49 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -39,6 +39,16 @@
|
|
|
39
39
|
- **Interactive webhooks** (added in 0.7.27) — set `webhookActionToken` and exhaustion notifications gain action buttons on Telegram (`inline_keyboard`), Discord (buttons) and Slack (actions block): *Reset cooldown* and *Pause 1h* per pool. Buttons call back to `POST /dsh-key-rotation/webhook-action` with a bearer token; actions `reset-<provider>`, `pause-<provider>` (1 h), `disable-rotation`, `enable-rotation`.
|
|
40
40
|
- **Token leak detector** (added in 0.7.27) — recognizes live key shapes (OpenAI `sk-`, Anthropic `sk-ant-`, Google `AIza…`, GitHub `ghp_…`, AWS `AKIA…`, Slack `xox…`, Telegram bot tokens, Stripe, PEM blocks). Saving the config section with a real key pasted into a wrong field is rejected (`400 secret-in-config`); the key-save response carries `looksLikeSecret`, and the card hints when the value does not look like an API key.
|
|
41
41
|
- **Header status chip** (added in 0.7.27) — a small chip in the session header shows one dot for all pools (green = all keys healthy, amber = some cooling, red = a pool fully exhausted) plus a `healthy/total` counter; polls every 4 s.
|
|
42
|
+
- **Key expiry pre-warning** (added in 0.7.28) — `expiryWarnDays` (default 7): keys expiring within that horizon send one webhook notification per key per day and show a yellow *expires in Nd* badge in the card.
|
|
43
|
+
- **Provider cost budget** (added in 0.7.28) — per-provider `costBudgetDaily` / `costBudgetWeekly` (0 = off): a warn webhook fires from 80% spend, and with `pauseOnBudget: true` the whole pool is paused for 24 h at 100%. The card shows a live budget line with warn/red coloring.
|
|
44
|
+
- **Usage report + CSV export** (added in 0.7.28) — `GET /dsh-key-rotation/usage?days=N[&provider=…][&format=csv]` returns per-key requests/cost over the window; an *Export CSV* button in the card downloads the same data for one provider.
|
|
45
|
+
- **RPM capacity indicator** (added in 0.7.28) — `/status` carries `rpm: {used, remaining, resetMs}` per key; the card shows a ⏱ counter next to the active key.
|
|
46
|
+
- **Real key test probe** (added in 0.7.28) — the per-key *Test* button now sends `probe=models` through the existing `/test` route, so it validates the key against the live API (models list + latency), not just credential presence.
|
|
47
|
+
- **Per-key weights in the GUI** (added in 0.7.29) — a small number input per key edits its round-robin weight (1 = equal share); reorder/remove/add keep the positional `weights` array in sync.
|
|
48
|
+
- **Switch notifications** (added in 0.7.29) — opt-in `switchNotify: true` sends a webhook on every key switch (who failed, why, when), deduped to at most one message per provider per `switchNotifyThrottleMs` (default 60 s).
|
|
49
|
+
- **Budget action buttons** (added in 0.7.29) — when `webhookActionToken` is set, budget notifications carry *Pause 1h* / *Reset cooldown* buttons (same callback route as exhaustion alerts).
|
|
50
|
+
- **Config snapshot export/restore** (added in 0.7.29) — *Snapshot ⬇* downloads the whole config as one JSON (token fields exported empty, keys are credential names only); *Restore* imports it back — empty token fields never wipe existing secrets, and a live-looking credential in the file is rejected.
|
|
51
|
+
- **Last probe result per key** (added in 0.7.29) — the card polls `/sandbox-cache` and shows the most recent probe outcome (✓/✕ + latency) next to each key, greyed out when older than 24 h.
|
|
42
52
|
|
|
43
53
|
## Install
|
|
44
54
|
|
|
@@ -83,6 +93,12 @@ dsh-key-rotation:
|
|
|
83
93
|
| `providers` | — | `[{ provider, keys: [envName, ...] }]`. `keys` are credential/env **names**, not the key values themselves. |
|
|
84
94
|
| `rpmLimit` | `0` | Requests-per-minute cap **per key** (0 = off). A capped key is skipped pre-emptively until its 60 s window frees up. |
|
|
85
95
|
| `webhookActionToken` | `''` | Bearer token for the interactive webhook callback route. When set, exhaustion webhooks carry action buttons; empty disables them. |
|
|
96
|
+
| `expiryWarnDays` | `7` | Pre-warning horizon (days) for keys with `expiresAt`: webhook + card badge. |
|
|
97
|
+
| `switchNotify` | `false` | Send a webhook on every key switch (opt-in — can be chatty). |
|
|
98
|
+
| `switchNotifyThrottleMs` | `60000` | Minimum gap between switch notifications for the same provider. |
|
|
99
|
+
| `providers[].weights` | `[]` | Positional round-robin weights per key (editable in the card, 0.7.29). |
|
|
100
|
+
| `providers[].costBudgetDaily` / `.costBudgetWeekly` | `0` | Daily / weekly spend budget per provider (0 = off). Warn webhook from 80%. |
|
|
101
|
+
| `providers[].pauseOnBudget` | `false` | Pause the whole pool for 24 h when a budget is exceeded. |
|
|
86
102
|
| `providers[].tags` | `[]` | Free-form labels for a provider pool, surfaced in `GET /status`. |
|
|
87
103
|
|
|
88
104
|
### Interactive webhook actions (added in 0.7.27)
|
|
@@ -98,6 +114,15 @@ curl -X POST http://127.0.0.1:3080/dsh-key-rotation/webhook-action \
|
|
|
98
114
|
|
|
99
115
|
Actions: `reset-<provider>` (clear cooldowns), `pause-<provider>` (pause the whole pool for 1 hour), `disable-rotation` / `enable-rotation` (global). Platform callback payloads (`callback_data`, Discord `custom_id`, Slack button `value`) are accepted too. Without the correct bearer token the route answers `401`; without a configured token — `503`.
|
|
100
116
|
|
|
117
|
+
### Usage report (added in 0.7.28)
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
curl "http://127.0.0.1:3080/dsh-key-rotation/usage?days=7" # JSON
|
|
121
|
+
curl "http://127.0.0.1:3080/dsh-key-rotation/usage?format=csv&days=30&provider=my-provider" > usage.csv
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Per-key rows: `requests`, `cost`, `active`, and per-day counts over the window (1–90 days, default 7).
|
|
125
|
+
|
|
101
126
|
### How keys are stored
|
|
102
127
|
|
|
103
128
|
The plugin config only ever references keys by **name** (e.g. `MY_PROVIDER_API_KEY`). The values live in the dsh **Credentials** service or `$DSH_HOME/.credentials.yaml` — never in the plugin config.
|
package/lib/bucket.js
CHANGED
|
@@ -39,3 +39,14 @@ export function bucketSweep(windows, liveRefs) {
|
|
|
39
39
|
if (!liveRefs.has(ref)) windows.delete(ref);
|
|
40
40
|
}
|
|
41
41
|
}
|
|
42
|
+
|
|
43
|
+
/** #210: snapshot for /status - used/remaining/resetMs for one ref. */
|
|
44
|
+
export function bucketInfo(windows, ref, limit, now = Date.now()) {
|
|
45
|
+
if (!limit || limit <= 0) return null;
|
|
46
|
+
const hits = (windows?.get(ref) ?? []).filter((t) => t > now - WINDOW_MS);
|
|
47
|
+
return {
|
|
48
|
+
used: hits.length,
|
|
49
|
+
remaining: Math.max(0, limit - hits.length),
|
|
50
|
+
resetMs: hits.length ? Math.max(0, hits[0] + WINDOW_MS - now) : 0,
|
|
51
|
+
};
|
|
52
|
+
}
|
package/lib/client.js
CHANGED
|
@@ -64,6 +64,12 @@ window.__ModuleLoader__.load({
|
|
|
64
64
|
keyFromEnv: 'from the environment, read-only here',
|
|
65
65
|
keyWriteFailed: 'could not store the key: {msg}',
|
|
66
66
|
notSecretShape: 'saved, but the value does not look like an API key - check for a typo',
|
|
67
|
+
rpmTitle: 'requests/min: {u} used, {r} remaining',
|
|
68
|
+
budgetLabel: 'budget:',
|
|
69
|
+
exportCsv: 'Export CSV',
|
|
70
|
+
weightHint: 'round-robin weight: how many times this key joins the cycle (1 = equal share)',
|
|
71
|
+
snapshotExport: 'Snapshot ⬇',
|
|
72
|
+
snapshotImport: 'Restore',
|
|
67
73
|
keyHint: 'The value is stored in DSH credentials and never sent back to the browser — only its last 5 characters are shown. Names are generated for you; hover a key to see the one it uses.',
|
|
68
74
|
brokenKey: 'broken (3× AUTH)',
|
|
69
75
|
keyExpired: 'expired',
|
|
@@ -121,6 +127,12 @@ window.__ModuleLoader__.load({
|
|
|
121
127
|
keyFromEnv: 'задан в окружении, отсюда не меняется',
|
|
122
128
|
keyWriteFailed: 'не удалось сохранить ключ: {msg}',
|
|
123
129
|
notSecretShape: 'сохранено, но значение не похоже на API-ключ — проверьте опечатки',
|
|
130
|
+
rpmTitle: 'запросов/мин: {u} использовано, {r} осталось',
|
|
131
|
+
budgetLabel: 'бюджет:',
|
|
132
|
+
exportCsv: 'CSV',
|
|
133
|
+
weightHint: 'вес в круге: сколько раз ключ участвует в ротации (1 = поровну)',
|
|
134
|
+
snapshotExport: 'Снапшот ⬇',
|
|
135
|
+
snapshotImport: 'Восстановить',
|
|
124
136
|
keyHint: 'Значение хранится в учётных данных DSH и обратно в браузер не отдаётся — показываются только последние 5 символов. Имена переменных создаются автоматически; наведите на ключ, чтобы увидеть используемое имя.',
|
|
125
137
|
brokenKey: 'сломан (3× AUTH)',
|
|
126
138
|
keyExpired: 'истёк',
|
|
@@ -169,6 +181,24 @@ window.__ModuleLoader__.load({
|
|
|
169
181
|
return byProvider;
|
|
170
182
|
}
|
|
171
183
|
|
|
184
|
+
/** Последний probe-результат по каждому ключу (#219): /sandbox-cache. */
|
|
185
|
+
function useProbeCache() {
|
|
186
|
+
const [cache, setCache] = React.useState({});
|
|
187
|
+
React.useEffect(() => {
|
|
188
|
+
let alive = true;
|
|
189
|
+
const pull = () => {
|
|
190
|
+
fetch('/dsh-key-rotation/sandbox-cache', { headers: { accept: 'application/json' } })
|
|
191
|
+
.then((r) => (r.ok ? r.json() : null))
|
|
192
|
+
.then((data) => { if (alive && data) setCache(data); })
|
|
193
|
+
.catch(() => { /* кэш не критичен: карточка работает и без него */ });
|
|
194
|
+
};
|
|
195
|
+
pull();
|
|
196
|
+
const id = setInterval(pull, 4000);
|
|
197
|
+
return () => { alive = false; clearInterval(id); };
|
|
198
|
+
}, []);
|
|
199
|
+
return cache;
|
|
200
|
+
}
|
|
201
|
+
|
|
172
202
|
// formatAgo moved to lib/client-helpers.js for testability — keep local alias for bundle self-containment
|
|
173
203
|
function formatAgo(t, at) {
|
|
174
204
|
if (!at) return '';
|
|
@@ -294,7 +324,8 @@ window.__ModuleLoader__.load({
|
|
|
294
324
|
}); };
|
|
295
325
|
const doTest = (ref) => {
|
|
296
326
|
setTesting(ref); setTestResult((m) => ({ ...m, [ref]: null }));
|
|
297
|
-
|
|
327
|
+
// #212: real API probe (models is free on most providers), not just presence
|
|
328
|
+
fetch('/dsh-key-rotation/test', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ ref, probe: 'models' }) })
|
|
298
329
|
.then((r) => r.json())
|
|
299
330
|
.then((data) => setTestResult((m) => ({ ...m, [ref]: data })))
|
|
300
331
|
.catch((e) => setTestResult((m) => ({ ...m, [ref]: { ok: false, message: String(e?.message ?? e) } })))
|
|
@@ -338,6 +369,7 @@ window.__ModuleLoader__.load({
|
|
|
338
369
|
|
|
339
370
|
const val = draft ?? state.value;
|
|
340
371
|
const status = useRotationStatus();
|
|
372
|
+
const probeCache = useProbeCache();
|
|
341
373
|
const [resetting, setResetting] = React.useState('');
|
|
342
374
|
const doReset = (providerId) => {
|
|
343
375
|
setResetting(providerId);
|
|
@@ -425,6 +457,8 @@ window.__ModuleLoader__.load({
|
|
|
425
457
|
const entry = { ...(providers[pIndex] ?? {}) };
|
|
426
458
|
const allRefs = providers.flatMap((prov) => prov?.keys ?? []);
|
|
427
459
|
entry.keys = [...(entry.keys ?? []), nextKeyRef(entry.provider, entry.keys, allRefs)];
|
|
460
|
+
// keep weights aligned with keys (#215): new key gets default weight 1
|
|
461
|
+
if (Array.isArray(entry.weights) && entry.weights.length > 0) entry.weights = [...entry.weights, 1];
|
|
428
462
|
providers[pIndex] = entry;
|
|
429
463
|
return { ...cur, providers };
|
|
430
464
|
});
|
|
@@ -432,6 +466,10 @@ window.__ModuleLoader__.load({
|
|
|
432
466
|
const next = [...(Array.isArray(cur.providers) ? cur.providers : [])];
|
|
433
467
|
stashUndo({ type: 'key', index: pIndex, kIndex, key: next[pIndex]?.keys?.[kIndex] });
|
|
434
468
|
next[pIndex] = { ...next[pIndex], keys: (next[pIndex].keys ?? []).filter((_, i) => i !== kIndex) };
|
|
469
|
+
// weights are positional - drop along with the key (#215)
|
|
470
|
+
if (Array.isArray(next[pIndex].weights) && next[pIndex].weights.length > 0) {
|
|
471
|
+
next[pIndex] = { ...next[pIndex], weights: next[pIndex].weights.filter((_, i) => i !== kIndex) };
|
|
472
|
+
}
|
|
435
473
|
return { ...cur, providers: next };
|
|
436
474
|
}); };
|
|
437
475
|
const removeProvider = (pIndex) => setField((cur) => {
|
|
@@ -451,6 +489,26 @@ window.__ModuleLoader__.load({
|
|
|
451
489
|
keys[kIndex] = keys[target];
|
|
452
490
|
keys[target] = moved;
|
|
453
491
|
entry.keys = keys;
|
|
492
|
+
// weights are positional - swap along with the keys (#215)
|
|
493
|
+
const weights = [...(entry.weights ?? [])];
|
|
494
|
+
if (weights.length > 0) {
|
|
495
|
+
const w = weights[kIndex];
|
|
496
|
+
weights[kIndex] = weights[target];
|
|
497
|
+
weights[target] = w;
|
|
498
|
+
entry.weights = weights;
|
|
499
|
+
}
|
|
500
|
+
providers[pIndex] = entry;
|
|
501
|
+
return { ...cur, providers };
|
|
502
|
+
});
|
|
503
|
+
// #215: set a single key's round-robin weight (integer >= 1)
|
|
504
|
+
const setKeyWeight = (pIndex, kIndex, weight) => setField((cur) => {
|
|
505
|
+
const n = Math.max(1, Math.min(1000, Math.floor(Number(weight) || 1)));
|
|
506
|
+
const providers = [...(cur.providers ?? [])];
|
|
507
|
+
const entry = { ...(providers[pIndex] ?? {}) };
|
|
508
|
+
const weights = [...(entry.weights ?? [])];
|
|
509
|
+
while (weights.length < (entry.keys ?? []).length) weights.push(1);
|
|
510
|
+
weights[kIndex] = n;
|
|
511
|
+
entry.weights = weights;
|
|
454
512
|
providers[pIndex] = entry;
|
|
455
513
|
return { ...cur, providers };
|
|
456
514
|
});
|
|
@@ -517,7 +575,8 @@ window.__ModuleLoader__.load({
|
|
|
517
575
|
if (hit.expired) return { color: 'var(--dsw-alias-state-error-primary)', text: t('keyExpired') };
|
|
518
576
|
if (hit.expiresAt && !hit.expired) {
|
|
519
577
|
const days = Math.ceil((hit.expiresAt - Date.now()) / 86400000);
|
|
520
|
-
|
|
578
|
+
const warnDays = Number(val?.expiryWarnDays) || 7; // #207: configurable horizon
|
|
579
|
+
if (days <= warnDays) return { color: 'var(--dsw-alias-state-warning-primary)', text: t('keyExpiringSoon').replace('{n}', String(days)) };
|
|
521
580
|
}
|
|
522
581
|
if (hit.broken) return { color: 'var(--dsw-alias-state-error-primary)', text: t('brokenKey') };
|
|
523
582
|
if (!hit.present) return { color: 'var(--dsw-alias-state-error-primary)', text: t('keyMissing') };
|
|
@@ -541,6 +600,7 @@ window.__ModuleLoader__.load({
|
|
|
541
600
|
h('option', { key: prov.id, value: prov.id }, prov.name + (prov.id !== prov.name ? ' — ' + prov.id : ''))));
|
|
542
601
|
|
|
543
602
|
const keys = entry.keys ?? [];
|
|
603
|
+
const entryWeights = entry.weights ?? [];
|
|
544
604
|
const keyRows = keys.map((key, kIndex) => {
|
|
545
605
|
const st = keyStatus(entry.provider, key);
|
|
546
606
|
const info = keyInfo(entry.provider, key);
|
|
@@ -593,12 +653,34 @@ window.__ModuleLoader__.load({
|
|
|
593
653
|
}
|
|
594
654
|
}
|
|
595
655
|
if (info && info.lastUsedAt) meta.push(h('span', { key: 'lu', className: 'krot-tail', title: 'last used' }, formatAgo((k)=>t(k), info.lastUsedAt)));
|
|
656
|
+
// #215: per-key weight input (default 1)
|
|
657
|
+
meta.push(h('input', { key: 'w', type: 'number', min: 1, max: 1000, className: 'krot-in krot-weight',
|
|
658
|
+
value: (entryWeights[kIndex] ?? info?.weight ?? 1),
|
|
659
|
+
title: t('weightHint'),
|
|
660
|
+
onChange: (e) => setKeyWeight(pIndex, kIndex, e.target.value),
|
|
661
|
+
style: { width: '52px', padding: '2px 6px', fontSize: '12px' } }));
|
|
662
|
+
// #210: RPM capacity indicator (only when rpmLimit is active)
|
|
663
|
+
if (info && info.rpm) meta.push(h('span', { key: 'rpm', className: 'krot-tail',
|
|
664
|
+
title: t('rpmTitle').replace('{u}', String(info.rpm.used)).replace('{r}', String(info.rpm.remaining)),
|
|
665
|
+
style: info.rpm.remaining === 0 ? { color: 'var(--dsw-alias-state-error-primary)', fontWeight: 700 } : undefined },
|
|
666
|
+
'⏱' + info.rpm.remaining));
|
|
596
667
|
if (info && typeof info.cost === 'number' && info.cost > 0) meta.push(h('span', { key: 'c', className: 'krot-tail', title: 'cost' }, '$' + info.cost.toFixed(2)));
|
|
597
668
|
const tr = testResult[key];
|
|
598
669
|
if (tr) meta.push(h('span', { key: 'tr', className: 'krot-tail',
|
|
599
|
-
title: tr.message || (tr.ok ? t('testOk') : t('testFail'))
|
|
670
|
+
title: (tr.message || (tr.ok ? t('testOk') : t('testFail')))
|
|
671
|
+
+ (tr.ok && tr.modelsCount ? ' · ' + tr.modelsCount + ' models' : '')
|
|
672
|
+
+ (tr.ok && tr.latencyMs ? ' · ' + tr.latencyMs + 'ms' : ''),
|
|
600
673
|
style: { color: tr.ok ? 'var(--dsw-alias-state-success-primary)' : 'var(--dsw-alias-state-error-primary)', fontWeight: 700 } },
|
|
601
|
-
tr.ok ? '✓' : '✕'));
|
|
674
|
+
tr.ok ? (tr.modelsCount ? tr.modelsCount + 'm' : '✓') : '✕'));
|
|
675
|
+
// #219: last probe from the sandbox cache, greyed when older than 24h
|
|
676
|
+
else if (probeCache && probeCache[key]) {
|
|
677
|
+
const pc = probeCache[key];
|
|
678
|
+
const stale = Date.now() - (pc.at ?? 0) > 86400000;
|
|
679
|
+
meta.push(h('span', { key: 'pc', className: 'krot-tail',
|
|
680
|
+
title: 'last probe ' + (pc.at ? new Date(pc.at).toLocaleTimeString() : '') + (pc.ok ? ' ok' : ' ' + (pc.code ?? 'fail')),
|
|
681
|
+
style: { opacity: stale ? 0.4 : 0.7, color: pc.ok ? 'var(--dsw-alias-state-success-primary)' : 'var(--dsw-alias-state-error-primary)' } },
|
|
682
|
+
(pc.ok ? '✓' : '✕') + (pc.latencyMs ? ' ' + pc.latencyMs + 'ms' : '')));
|
|
683
|
+
}
|
|
602
684
|
meta.push(h('button', { key: 't', className: 'krot-btn', onClick: () => doTest(key), disabled: testing === key, title: t('testKey') }, testing === key ? '…' : t('testKey')));
|
|
603
685
|
meta.push(h('span', { key: 'a', className: 'krot-acts' },
|
|
604
686
|
btn('↑', () => moveKey(pIndex, kIndex, -1), { disabled: kIndex === 0, title: t('moveUp') }),
|
|
@@ -623,6 +705,28 @@ window.__ModuleLoader__.load({
|
|
|
623
705
|
const exhaustionWarning = providerStatus && providerStatus.lastExhaustionAt && (Date.now() - providerStatus.lastExhaustionAt) < 3600000
|
|
624
706
|
? h('p', { className: 'krot-err' }, t('poolExhausted') + ' (' + formatAgo(t, providerStatus.lastExhaustionAt) + ')')
|
|
625
707
|
: null;
|
|
708
|
+
// #208: budget line (warn color at >=80%, red at 100%)
|
|
709
|
+
const budgetLine = providerStatus && (providerStatus.budgetDaily > 0 || providerStatus.budgetWeekly > 0)
|
|
710
|
+
? (() => {
|
|
711
|
+
const dayRatio = providerStatus.budgetDaily > 0 ? providerStatus.todayCost / providerStatus.budgetDaily : 0;
|
|
712
|
+
const weekRatio = providerStatus.budgetWeekly > 0 ? providerStatus.weeklyCost / providerStatus.budgetWeekly : 0;
|
|
713
|
+
const worst = Math.max(dayRatio, weekRatio);
|
|
714
|
+
const color = worst >= 1 ? 'var(--dsw-alias-state-error-primary)' : worst >= 0.8 ? 'var(--dsw-alias-state-warning-primary)' : 'var(--dsw-alias-label-tertiary)';
|
|
715
|
+
const parts = [];
|
|
716
|
+
if (providerStatus.budgetDaily > 0) parts.push('$' + (providerStatus.todayCost ?? 0).toFixed(2) + '/' + '$' + providerStatus.budgetDaily);
|
|
717
|
+
if (providerStatus.budgetWeekly > 0) parts.push('week $' + (providerStatus.weeklyCost ?? 0).toFixed(2) + '/' + '$' + providerStatus.budgetWeekly);
|
|
718
|
+
if (worst >= 1 && providerStatus.pauseOnBudget) parts.push('· paused');
|
|
719
|
+
return h('p', { className: 'krot-hint', style: { color } }, t('budgetLabel') + ' ' + parts.join(' · '));
|
|
720
|
+
})()
|
|
721
|
+
: null;
|
|
722
|
+
// #209: CSV export for this provider's usage (last 7 days)
|
|
723
|
+
const exportCsv = h('button', { className: 'krot-btn', title: t('exportCsv'),
|
|
724
|
+
onClick: () => {
|
|
725
|
+
const url = '/dsh-key-rotation/usage?format=csv&days=7&provider=' + encodeURIComponent(entry.provider);
|
|
726
|
+
const a = document.createElement('a');
|
|
727
|
+
a.href = url; a.download = 'usage-' + entry.provider + '.csv';
|
|
728
|
+
document.body.appendChild(a); a.click(); a.remove();
|
|
729
|
+
} }, t('exportCsv'));
|
|
626
730
|
|
|
627
731
|
return h('div', { key: pIndex, className: 'krot-prov' },
|
|
628
732
|
h('div', { className: 'krot-prov-head' },
|
|
@@ -660,11 +764,13 @@ window.__ModuleLoader__.load({
|
|
|
660
764
|
h('div', { className: 'krot-foot' },
|
|
661
765
|
btn(t('addKey'), () => addKey(pIndex), { title: t('addKeyTitle') }),
|
|
662
766
|
switchesLine,
|
|
767
|
+
budgetLine,
|
|
663
768
|
exhaustionWarning,
|
|
664
769
|
(providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('div', { style: { display: 'flex', gap: '2px', alignItems: 'end', height: '24px', marginTop: '4px' } }, (() => { const now = Date.now(); const buckets = Array(24).fill(0); for (const ev of providerStatus.events) { const h = Math.floor((now - ev.at) / 3600000); if (h >= 0 && h < 24) buckets[23 - h]++; } const max = Math.max(1, ...buckets); return buckets.map((c, i) => h('div', { key: i, title: c + ' switches', style: { flex: 1, background: c ? 'var(--dsw-alias-state-warning-primary)' : 'var(--dsw-alias-border-l2)', height: (c / max * 24) + 'px', minHeight: '2px', borderRadius: '2px' } })); })()) : null),
|
|
665
770
|
(providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('details', { style: { fontSize: '11px', marginTop: '6px' } }, h('summary', null, 'Recent failures ('+providerStatus.events.length+')'), h('ul', { style: { margin: '4px 0 0', paddingLeft: '16px' } }, providerStatus.events.slice().reverse().map((ev, i) => h('li', { key: i, style: ev.type === 'probe' ? { opacity: .5 } : null }, new Date(ev.at).toLocaleTimeString() + ' ' + (ev.type === 'probe' ? '[probe] ' : '') + ev.ref + ' ' + ev.reason + ' cd=' + ev.cooldownMs)) )) : null),
|
|
666
771
|
btn(t('resetCooldown'), () => doReset(entry.provider), { disabled: !(providerStatus && providerStatus.switches > 0) || resetting === entry.provider, title: t('resetCooldown') }),
|
|
667
772
|
btn(testAllProvider === entry.provider ? t('testing') : t('testAll'), () => doTestAll(entry.provider), { disabled: testAllProvider === entry.provider, title: t('testAll') }),
|
|
773
|
+
exportCsv,
|
|
668
774
|
),
|
|
669
775
|
);
|
|
670
776
|
});
|
|
@@ -714,6 +820,32 @@ window.__ModuleLoader__.load({
|
|
|
714
820
|
}); } catch (err) { setSecretError(String(err.message || err)); } };
|
|
715
821
|
reader.readAsText(f);
|
|
716
822
|
} }))),
|
|
823
|
+
// #218: full snapshot export/import - one file moves the whole config
|
|
824
|
+
btn(t('snapshotExport'), () => {
|
|
825
|
+
fetch('/dsh-key-rotation/snapshot', { headers: { accept: 'application/json' } })
|
|
826
|
+
.then((r) => r.json())
|
|
827
|
+
.then((data) => {
|
|
828
|
+
const blob = new Blob([JSON.stringify(data.snapshot ?? {}, null, 2)], { type: 'application/json' });
|
|
829
|
+
const url = URL.createObjectURL(blob);
|
|
830
|
+
const a = document.createElement('a'); a.href = url; a.download = 'dsh-key-rotation-snapshot.json'; a.click(); URL.revokeObjectURL(url);
|
|
831
|
+
})
|
|
832
|
+
.catch((e) => setSecretError(String(e?.message ?? e)));
|
|
833
|
+
}, {}),
|
|
834
|
+
h('label', { className: 'krot-btn', style: { cursor: 'pointer' } }, t('snapshotImport'), h('input', { type: 'file', accept: '.json', style: { display: 'none' }, onChange: (e) => {
|
|
835
|
+
const f = e.target.files[0]; if (!f) return;
|
|
836
|
+
const reader2 = new FileReader();
|
|
837
|
+
reader2.onload = () => {
|
|
838
|
+
try {
|
|
839
|
+
const snap = JSON.parse(String(reader2.result));
|
|
840
|
+
if (!snap || typeof snap !== 'object' || Array.isArray(snap)) throw new Error('expected snapshot object');
|
|
841
|
+
fetch('/dsh-key-rotation/snapshot', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ snapshot: snap }) })
|
|
842
|
+
.then((r) => r.json().then((data) => ({ ok: r.ok, data })))
|
|
843
|
+
.then(({ ok, data }) => { if (!ok) throw new Error(data?.error?.message ?? 'import failed'); load(); })
|
|
844
|
+
.catch((err) => setSecretError(t('keyWriteFailed').replace('{msg}', String(err?.message ?? err))));
|
|
845
|
+
} catch (err) { setSecretError(String(err.message || err)); }
|
|
846
|
+
};
|
|
847
|
+
reader2.readAsText(f);
|
|
848
|
+
} })),
|
|
717
849
|
h('p', { className: 'krot-hint' }, t('keyHint')),
|
|
718
850
|
secretError ? h('p', { className: 'krot-err' }, secretError) : null,
|
|
719
851
|
state.error ? h('p', { className: 'krot-err' }, state.error) : null,
|
package/lib/index.js
CHANGED
|
@@ -44,10 +44,12 @@ const NS = 'dsh-key-rotation';
|
|
|
44
44
|
/** Config bridge route (GET / PUT / DELETE), loopback-fenced like llm-fallback. */
|
|
45
45
|
const CONFIG_PATH = '/dsh-key-rotation/config';
|
|
46
46
|
const STATUS_PATH = '/dsh-key-rotation/status';
|
|
47
|
+
const SNAPSHOT_PATH = '/dsh-key-rotation/snapshot';
|
|
47
48
|
const KEY_PATH = '/dsh-key-rotation/key';
|
|
48
49
|
const RESET_PATH = '/dsh-key-rotation/reset';
|
|
49
50
|
const IMPORT_PATH = '/dsh-key-rotation/import';
|
|
50
51
|
const HEALTH_PATH = '/dsh-key-rotation/health';
|
|
52
|
+
const USAGE_PATH = '/dsh-key-rotation/usage';
|
|
51
53
|
const TEST_PATH = '/dsh-key-rotation/test';
|
|
52
54
|
const SANDBOX_CACHE_PATH = '/dsh-key-rotation/sandbox-cache';
|
|
53
55
|
const AGENT_BUDGET_PATH = '/dsh-key-rotation/agent-budget';
|
|
@@ -69,7 +71,9 @@ import { RegionMap } from './region.js';
|
|
|
69
71
|
import { IncidentReporter } from './incident.js';
|
|
70
72
|
import { ShadowRouter } from './shadow.js';
|
|
71
73
|
import { WebhookSender } from './webhook.js';
|
|
72
|
-
import { bucketAllow, bucketRetryMs, bucketSweep } from './bucket.js';
|
|
74
|
+
import { bucketAllow, bucketRetryMs, bucketSweep, bucketInfo } from './bucket.js';
|
|
75
|
+
import { expiringSoon, shouldNotifyDaily, costForDay, budgetVerdict, costForWeek } from './maintenance.js';
|
|
76
|
+
import { usageRows, usageCsv } from './usage-report.js';
|
|
73
77
|
import { findSecrets, looksLikeApiSecret } from './keycheck.js';
|
|
74
78
|
|
|
75
79
|
/** The llm-pi-ai namespace whose provider profiles map providers to pools. */
|
|
@@ -78,6 +82,31 @@ const PIAI_NS = 'llm-pi-ai';
|
|
|
78
82
|
const MARKER = '__dshKeyRotation';
|
|
79
83
|
/** #199: set true via webhook action; checked in the llm/stream interceptor. */
|
|
80
84
|
let rotationDisabled = false;
|
|
85
|
+
// #207/#208 dedupe maps: one notification per key/window per day.
|
|
86
|
+
const expiryNotifiedAt = new Map();
|
|
87
|
+
const budgetNotifiedAt = new Map();
|
|
88
|
+
const switchNotifiedAt = new Map();
|
|
89
|
+
const DAY_MS = 86400000;
|
|
90
|
+
|
|
91
|
+
// #216: one webhook per switch, deduped to at most one message per provider
|
|
92
|
+
// per switchNotifyThrottleMs. Extracted for testability.
|
|
93
|
+
export function notifySwitch(runtime, pool, info, hooks = { webhookSender, now: () => Date.now() }) {
|
|
94
|
+
if (!runtime?.notifyWebhook) return;
|
|
95
|
+
const throttle = Math.max(0, runtime.switchNotifyThrottleMs ?? 60000);
|
|
96
|
+
const last = switchNotifiedAt.get(info.provider) ?? 0;
|
|
97
|
+
const now = hooks.now();
|
|
98
|
+
if (now - last < throttle) return;
|
|
99
|
+
switchNotifiedAt.set(info.provider, now);
|
|
100
|
+
hooks.webhookSender.send(runtime.notifyWebhook, {
|
|
101
|
+
title: `Key switched: ${info.provider}`,
|
|
102
|
+
text: `${info.from} failed (${info.code}) - next key in pool`,
|
|
103
|
+
provider: info.provider,
|
|
104
|
+
kind: 'switch',
|
|
105
|
+
from: info.from,
|
|
106
|
+
code: info.code,
|
|
107
|
+
at: info.at,
|
|
108
|
+
});
|
|
109
|
+
}
|
|
81
110
|
const MAX_EVENTS = 50;
|
|
82
111
|
function pushEvent(pool, ref, reason, cooldownMs, type) {
|
|
83
112
|
const ev = { at: Date.now(), ref, reason: String(reason ?? 'UNKNOWN'), cooldownMs, type: type ?? 'fail' };
|
|
@@ -183,12 +212,18 @@ export const Config = Schema.object({
|
|
|
183
212
|
rateLimitThreshold: Schema.number().default(0.1),
|
|
184
213
|
rpmLimit: Schema.number().default(0),
|
|
185
214
|
webhookActionToken: Schema.string().default(''),
|
|
215
|
+
expiryWarnDays: Schema.number().default(7),
|
|
216
|
+
switchNotify: Schema.boolean().default(false),
|
|
217
|
+
switchNotifyThrottleMs: Schema.number().default(60000),
|
|
186
218
|
providers: Schema.array(Schema.object({
|
|
187
219
|
provider: Schema.string().required(),
|
|
188
220
|
keys: Schema.array(Schema.string()).default([]),
|
|
189
221
|
weights: Schema.array(Schema.number()).default([]),
|
|
190
222
|
expiresAt: Schema.array(Schema.union([Schema.number(), Schema.string()])).default([]),
|
|
191
223
|
tags: Schema.array(Schema.string()).default([]),
|
|
224
|
+
costBudgetDaily: Schema.number(),
|
|
225
|
+
costBudgetWeekly: Schema.number(),
|
|
226
|
+
pauseOnBudget: Schema.boolean().default(false),
|
|
192
227
|
models: Schema.dict(Schema.object({
|
|
193
228
|
keys: Schema.array(Schema.string()).default([]),
|
|
194
229
|
weights: Schema.array(Schema.number()).default([]),
|
|
@@ -542,6 +577,64 @@ export function apply(ctx, config = {}) {
|
|
|
542
577
|
}
|
|
543
578
|
const n = sweepExpired(poolState, now);
|
|
544
579
|
if (n > 0) console.warn(`[dsh-key-rotation] sweep: cleared ${n} expired cooldown(s)`);
|
|
580
|
+
// #207 expiry pre-warning + #208 cost budget - piggybacked on this timer,
|
|
581
|
+
// deduped to one notification per key/window per day (shouldNotifyDaily).
|
|
582
|
+
try {
|
|
583
|
+
const runtime = buildRuntime();
|
|
584
|
+
const seen = new Set();
|
|
585
|
+
for (const pool of runtime.poolByRef.values()) {
|
|
586
|
+
if (seen.has(pool.base)) continue;
|
|
587
|
+
seen.add(pool.base);
|
|
588
|
+
// #207: keys expiring within expiryWarnDays -> one webhook per key/day
|
|
589
|
+
for (const { ref, expiresInDays } of expiringSoon(pool, runtime.expiryWarnDays, now)) {
|
|
590
|
+
if (!shouldNotifyDaily(expiryNotifiedAt, pool.base + ':' + ref, now)) continue;
|
|
591
|
+
console.warn(`[dsh-key-rotation] ${pool.base}: key ${ref} expires in ~${expiresInDays}d`);
|
|
592
|
+
if (runtime.notifyWebhook) {
|
|
593
|
+
webhookSender.send(runtime.notifyWebhook, {
|
|
594
|
+
title: `Key expiring soon: ${pool.base}`,
|
|
595
|
+
text: `${ref} expires in ~${expiresInDays} day(s)`,
|
|
596
|
+
provider: pool.base,
|
|
597
|
+
kind: 'expiry',
|
|
598
|
+
keys: [ref],
|
|
599
|
+
});
|
|
600
|
+
}
|
|
601
|
+
}
|
|
602
|
+
// #208: daily/weekly budget -> warn webhook, optional 1-day pause at 100%
|
|
603
|
+
const budget = runtime.providerBudgets.get(pool.base);
|
|
604
|
+
if (!budget) continue;
|
|
605
|
+
const daily = costForDay(pool.state.costDays);
|
|
606
|
+
const weekly = costForWeek(pool.state.costDays, now);
|
|
607
|
+
const verdict = budgetVerdict(daily, budget.costBudgetDaily);
|
|
608
|
+
const wVerdict = budgetVerdict(weekly, budget.costBudgetWeekly);
|
|
609
|
+
const hit = verdict.warn || wVerdict.warn;
|
|
610
|
+
if (hit && shouldNotifyDaily(budgetNotifiedAt, pool.base + ':budget', now)) {
|
|
611
|
+
console.warn(`[dsh-key-rotation] ${pool.base}: cost budget - day $${daily.toFixed(2)}/$${budget.costBudgetDaily} week $${weekly.toFixed(2)}/$${budget.costBudgetWeekly}`);
|
|
612
|
+
if (runtime.notifyWebhook) {
|
|
613
|
+
// #217: budget webhook gains action buttons when a callback token
|
|
614
|
+
// is configured (the /webhook-action route already knows these ids)
|
|
615
|
+
const token = runtime.webhookActionToken ?? '';
|
|
616
|
+
webhookSender.send(runtime.notifyWebhook, {
|
|
617
|
+
title: `Cost budget: ${pool.base}`,
|
|
618
|
+
text: `day $${daily.toFixed(2)} of $${budget.costBudgetDaily} · week $${weekly.toFixed(2)} of $${budget.costBudgetWeekly}` + (verdict.exceeded || wVerdict.exceeded ? ' · EXCEEDED' : ''),
|
|
619
|
+
provider: pool.base,
|
|
620
|
+
kind: 'budget',
|
|
621
|
+
spend: { daily, weekly },
|
|
622
|
+
actionToken: token || undefined,
|
|
623
|
+
actions: token ? [
|
|
624
|
+
{ id: `pause-${pool.base}`, label: 'Pause 1h' },
|
|
625
|
+
{ id: `reset-${pool.base}`, label: 'Reset cooldown' },
|
|
626
|
+
] : undefined,
|
|
627
|
+
});
|
|
628
|
+
}
|
|
629
|
+
}
|
|
630
|
+
if ((verdict.exceeded || wVerdict.exceeded) && budget.pauseOnBudget) {
|
|
631
|
+
const until = now + DAY_MS;
|
|
632
|
+
for (const ref of pool.refs) {
|
|
633
|
+
if ((pool.state.failedUntil.get(ref) ?? 0) < until) pool.state.failedUntil.set(ref, until);
|
|
634
|
+
}
|
|
635
|
+
}
|
|
636
|
+
}
|
|
637
|
+
} catch (_) { /* maintenance must never crash the sweep */ }
|
|
545
638
|
}, 30000);
|
|
546
639
|
return () => clearInterval(id);
|
|
547
640
|
}, 'dsh-key-rotation: sweep expired cooldowns');
|
|
@@ -627,7 +720,8 @@ export function apply(ctx, config = {}) {
|
|
|
627
720
|
if (exp !== undefined) parsedExpiry[refs[i]] = exp;
|
|
628
721
|
}
|
|
629
722
|
}
|
|
630
|
-
return { base, refs,
|
|
723
|
+
return { base, refs, weights: refs.map((_, i) => (typeof w[i] === 'number' && w[i] > 0 ? Math.floor(w[i]) : 1)),
|
|
724
|
+
weightedRefs: weightedRefs.length > 0 ? weightedRefs : refs,
|
|
631
725
|
state: makeState(base), cooldownMs: poolCooldown, maxCooldownMs: poolMax, expiresAt: parsedExpiry, rpmLimit };
|
|
632
726
|
};
|
|
633
727
|
for (const p of cfg.providers ?? []) {
|
|
@@ -674,10 +768,15 @@ export function apply(ctx, config = {}) {
|
|
|
674
768
|
}
|
|
675
769
|
// #195: provider -> tags (metadata, surfaced in status)
|
|
676
770
|
const providerTags = new Map();
|
|
771
|
+
// #208: provider -> { costBudgetDaily, costBudgetWeekly, pauseOnBudget }
|
|
772
|
+
const providerBudgets = new Map();
|
|
677
773
|
for (const p of cfg.providers ?? []) {
|
|
678
774
|
if (Array.isArray(p.tags) && p.tags.length > 0) providerTags.set(p.provider, p.tags);
|
|
775
|
+
const daily = typeof p.costBudgetDaily === 'number' ? p.costBudgetDaily : 0;
|
|
776
|
+
const weekly = typeof p.costBudgetWeekly === 'number' ? p.costBudgetWeekly : 0;
|
|
777
|
+
if (daily > 0 || weekly > 0) providerBudgets.set(p.provider, { costBudgetDaily: daily, costBudgetWeekly: weekly, pauseOnBudget: p.pauseOnBudget ?? false });
|
|
679
778
|
}
|
|
680
|
-
return { switchCodes, cooldownMs, maxCooldownMs, notifyWebhook, notifyThreshold, incidentThreshold, concurrencyLimit, canaryProbingEnabled, canaryIntervalMs, cascade, quotaResetWindow, backupDir, backupIntervalMs, backupKeep, rotationScheduleDays, rateLimitThreshold, rpmLimit, webhookActionToken, providerTags, poolByRef, providerToPool, modelPoolByProvider, cloneIds };
|
|
779
|
+
return { switchCodes, cooldownMs, maxCooldownMs, notifyWebhook, notifyThreshold, incidentThreshold, concurrencyLimit, canaryProbingEnabled, canaryIntervalMs, cascade, quotaResetWindow, backupDir, backupIntervalMs, backupKeep, rotationScheduleDays, rateLimitThreshold, rpmLimit, webhookActionToken, expiryWarnDays: cfg.expiryWarnDays ?? 7, switchNotify: cfg.switchNotify ?? false, switchNotifyThrottleMs: cfg.switchNotifyThrottleMs ?? 60000, providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider, cloneIds };
|
|
681
780
|
}
|
|
682
781
|
|
|
683
782
|
// ── patch credentials.resolve: pool refs resolve to the next healthy key ──
|
|
@@ -878,7 +977,16 @@ export function apply(ctx, config = {}) {
|
|
|
878
977
|
pool.state.lastReason = String(code ?? 'UNKNOWN');
|
|
879
978
|
pool.state.lastSwitchAt = Date.now();
|
|
880
979
|
lastFailure = chunk;
|
|
881
|
-
console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(pool.state.lastUsed ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)})
|
|
980
|
+
console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(pool.state.lastUsed ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
|
|
981
|
+
// #216: per-switch webhook (opt-in switchNotify), deduped per provider
|
|
982
|
+
if (buildRuntime().switchNotify && pool.state.lastUsed) {
|
|
983
|
+
notifySwitch(buildRuntime(), pool, {
|
|
984
|
+
provider: options.provider,
|
|
985
|
+
from: pool.state.lastUsed,
|
|
986
|
+
code: String(code ?? 'UNKNOWN'),
|
|
987
|
+
at: pool.state.lastSwitchAt,
|
|
988
|
+
});
|
|
989
|
+
}
|
|
882
990
|
switching = true;
|
|
883
991
|
break;
|
|
884
992
|
}
|
|
@@ -888,6 +996,12 @@ export function apply(ctx, config = {}) {
|
|
|
888
996
|
if (!isNaN(c)) {
|
|
889
997
|
if (!pool.state.costPerKey) pool.state.costPerKey = new Map();
|
|
890
998
|
pool.state.costPerKey.set(pool.state.lastUsed, (pool.state.costPerKey.get(pool.state.lastUsed) ?? 0) + c);
|
|
999
|
+
// #208: cost per day per key (mirrors usageDays) for budget checks
|
|
1000
|
+
if (!pool.state.costDays) pool.state.costDays = new Map();
|
|
1001
|
+
const cday = new Date().toISOString().slice(0, 10);
|
|
1002
|
+
const cMap = pool.state.costDays.get(pool.state.lastUsed) || new Map();
|
|
1003
|
+
cMap.set(cday, (cMap.get(cday) ?? 0) + c);
|
|
1004
|
+
pool.state.costDays.set(pool.state.lastUsed, cMap);
|
|
891
1005
|
}
|
|
892
1006
|
}
|
|
893
1007
|
// Usage by day (#119)
|
|
@@ -987,7 +1101,7 @@ export function apply(ctx, config = {}) {
|
|
|
987
1101
|
json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: status is local-only' } });
|
|
988
1102
|
return;
|
|
989
1103
|
}
|
|
990
|
-
const { poolByRef, providerTags } = buildRuntime();
|
|
1104
|
+
const { poolByRef, providerTags, providerBudgets } = buildRuntime();
|
|
991
1105
|
const base = ctx.get('credentials');
|
|
992
1106
|
const now = Date.now();
|
|
993
1107
|
const seen = new Set();
|
|
@@ -995,6 +1109,7 @@ export function apply(ctx, config = {}) {
|
|
|
995
1109
|
for (const pool of poolByRef.values()) {
|
|
996
1110
|
if (seen.has(pool.base)) continue;
|
|
997
1111
|
seen.add(pool.base);
|
|
1112
|
+
try {
|
|
998
1113
|
const keys = [];
|
|
999
1114
|
for (const ref of pool.refs) {
|
|
1000
1115
|
let present = false;
|
|
@@ -1032,6 +1147,10 @@ export function apply(ctx, config = {}) {
|
|
|
1032
1147
|
writable,
|
|
1033
1148
|
active: pool.state.lastUsed === ref,
|
|
1034
1149
|
cooldownMsLeft: until !== undefined && until > now ? until - now : 0,
|
|
1150
|
+
// #210: RPM capacity snapshot (null when rpmLimit is off)
|
|
1151
|
+
rpm: bucketInfo(pool.state.rpmWindows, ref, pool.rpmLimit, now),
|
|
1152
|
+
// #215: effective round-robin weight of this key
|
|
1153
|
+
weight: pool.weights?.[pool.refs.indexOf(ref)] ?? 1,
|
|
1035
1154
|
usage: pool.state.usageCounts?.get(ref) ?? 0,
|
|
1036
1155
|
byModel: pool.state.byModel?.get(ref) ? Object.fromEntries(pool.state.byModel.get(ref)) : {},
|
|
1037
1156
|
usageDays: pool.state.usageDays?.get(ref) ? Object.fromEntries(pool.state.usageDays.get(ref)) : {},
|
|
@@ -1054,12 +1173,111 @@ export function apply(ctx, config = {}) {
|
|
|
1054
1173
|
totalUsage: [...(pool.state.usageCounts?.values() ?? [])].reduce((a, b) => a + b, 0),
|
|
1055
1174
|
events: (pool.state.events ?? []).slice(-50),
|
|
1056
1175
|
healthScore: computeHealthScore(pool.state),
|
|
1176
|
+
// #208: today/week spend + configured budget for the card
|
|
1177
|
+
todayCost: costForDay(pool.state.costDays),
|
|
1178
|
+
weeklyCost: costForWeek(pool.state.costDays, now),
|
|
1179
|
+
budgetDaily: providerBudgets.get(pool.base)?.costBudgetDaily ?? 0,
|
|
1180
|
+
budgetWeekly: providerBudgets.get(pool.base)?.costBudgetWeekly ?? 0,
|
|
1181
|
+
pauseOnBudget: providerBudgets.get(pool.base)?.pauseOnBudget ?? false,
|
|
1057
1182
|
});
|
|
1183
|
+
} catch (e) {
|
|
1184
|
+
console.warn(`[dsh-key-rotation] status: pool ${pool.base} failed: ${String(e?.message ?? e)} ${e?.stack ?? ''}`);
|
|
1185
|
+
providers.push({ provider: pool.base, keys: [], statusError: String(e?.message ?? e) });
|
|
1186
|
+
}
|
|
1058
1187
|
}
|
|
1059
1188
|
json(res, 200, { providers });
|
|
1060
1189
|
},
|
|
1061
1190
|
}), 'dsh-key-rotation: status route');
|
|
1062
1191
|
|
|
1192
|
+
// #209: usage report - per-key requests/cost over the last N days.
|
|
1193
|
+
// ?format=csv returns text/csv; ?days=N window (1..90, default 7).
|
|
1194
|
+
ctx.effect(() => ctx.webServer.register({
|
|
1195
|
+
kind: 'exact',
|
|
1196
|
+
path: USAGE_PATH,
|
|
1197
|
+
handler: (req, res) => {
|
|
1198
|
+
if (req.method !== 'GET') { json(res, 405, { error: { code: 'method', message: 'GET only' } }); return; }
|
|
1199
|
+
if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: usage is local-only' } }); return; }
|
|
1200
|
+
const url = new URL(req.url ?? USAGE_PATH, 'http://localhost');
|
|
1201
|
+
const days = Math.min(90, Math.max(1, Number(url.searchParams.get('days')) || 7));
|
|
1202
|
+
const csv = url.searchParams.get('format') === 'csv';
|
|
1203
|
+
const provider = url.searchParams.get('provider') ?? '';
|
|
1204
|
+
const runtime = buildRuntime();
|
|
1205
|
+
const now = Date.now();
|
|
1206
|
+
const seen = new Set();
|
|
1207
|
+
const report = [];
|
|
1208
|
+
for (const pool of runtime.poolByRef.values()) {
|
|
1209
|
+
if (seen.has(pool.base)) continue;
|
|
1210
|
+
seen.add(pool.base);
|
|
1211
|
+
if (provider && pool.base !== provider) continue;
|
|
1212
|
+
report.push({ provider: pool.base, rows: usageRows(pool, days, now) });
|
|
1213
|
+
}
|
|
1214
|
+
if (csv) {
|
|
1215
|
+
res.writeHead(200, { 'content-type': 'text/csv; charset=utf-8', 'content-disposition': 'attachment; filename="dsh-key-rotation-usage.csv"' });
|
|
1216
|
+
const parts = [];
|
|
1217
|
+
for (const p of report) {
|
|
1218
|
+
if (parts.length > 0) parts.push('');
|
|
1219
|
+
parts.push('# ' + p.provider);
|
|
1220
|
+
parts.push(usageCsv(p.rows));
|
|
1221
|
+
}
|
|
1222
|
+
res.end(parts.join('\n') + '\n');
|
|
1223
|
+
return;
|
|
1224
|
+
}
|
|
1225
|
+
json(res, 200, { at: now, days, providers: report });
|
|
1226
|
+
},
|
|
1227
|
+
}), 'dsh-key-rotation: usage route');
|
|
1228
|
+
|
|
1229
|
+
// #218: full config snapshot - one JSON file to move between machines.
|
|
1230
|
+
// Secret values never travel: only credential/env names. Token fields are
|
|
1231
|
+
// exported as empty strings; on import they keep existing values when empty.
|
|
1232
|
+
ctx.effect(() => ctx.webServer.register({
|
|
1233
|
+
kind: 'exact',
|
|
1234
|
+
path: SNAPSHOT_PATH,
|
|
1235
|
+
handler: async (req, res) => {
|
|
1236
|
+
if (req.method !== 'GET' && req.method !== 'POST') { json(res, 405, { error: { code: 'method', message: 'GET (export) or POST (import) only' } }); return; }
|
|
1237
|
+
if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: snapshot is local-only' } }); return; }
|
|
1238
|
+
if (req.method === 'GET') {
|
|
1239
|
+
const descriptor = descriptorOf(ctx, NS);
|
|
1240
|
+
const value = descriptor?.value ?? {};
|
|
1241
|
+
const exportable = { ...value };
|
|
1242
|
+
// token-shaped fields stay empty in the file; refs are names, not secrets
|
|
1243
|
+
exportable.webhookActionToken = '';
|
|
1244
|
+
if (exportable.incidentGitHubToken) exportable.incidentGitHubToken = '';
|
|
1245
|
+
json(res, 200, { at: Date.now(), version: 1, snapshot: exportable });
|
|
1246
|
+
return;
|
|
1247
|
+
}
|
|
1248
|
+
// POST = import: { snapshot } -> merge with current section, PUT semantics
|
|
1249
|
+
let body;
|
|
1250
|
+
try { body = await readJson(req); } catch (e) { json(res, 400, { error: { code: 'bad-request', message: String(e?.message ?? e) } }); return; }
|
|
1251
|
+
const snap = body?.snapshot;
|
|
1252
|
+
if (!snap || typeof snap !== 'object' || Array.isArray(snap)) { json(res, 400, { error: { code: 'bad-format', message: 'dsh-key-rotation: POST requires {"snapshot": {...}}' } }); return; }
|
|
1253
|
+
// #200 leak guard applies to imported content too
|
|
1254
|
+
try {
|
|
1255
|
+
const masked = structuredClone(snap);
|
|
1256
|
+
if (masked.incidentGitHubToken) masked.incidentGitHubToken = '***';
|
|
1257
|
+
if (masked.webhookActionToken) masked.webhookActionToken = '***';
|
|
1258
|
+
if (masked.notifyWebhook) masked.notifyWebhook = '***';
|
|
1259
|
+
const findings = findSecrets(JSON.stringify(masked));
|
|
1260
|
+
if (findings.length > 0) { json(res, 400, { error: { code: 'secret-in-snapshot', message: 'dsh-key-rotation: snapshot carries a live-looking credential', findings } }); return; }
|
|
1261
|
+
} catch { /* scanning must never block a valid import */ }
|
|
1262
|
+
const settings = ctx.get('settings');
|
|
1263
|
+
if (!settings) { json(res, 503, { error: { code: 'settings-rejected', message: 'dsh-key-rotation: no settings provider' } }); return; }
|
|
1264
|
+
const desc = descriptorOf(ctx, NS);
|
|
1265
|
+
if (desc === void 0) { json(res, 500, { error: { code: 'settings-rejected', message: 'dsh-key-rotation: namespace missing' } }); return; }
|
|
1266
|
+
const cur = desc.value ?? {};
|
|
1267
|
+
// empty token fields in the file keep the current values (never wipe a secret)
|
|
1268
|
+
const merged = { ...cur, ...snap };
|
|
1269
|
+
if (!snap.webhookActionToken) merged.webhookActionToken = cur.webhookActionToken ?? '';
|
|
1270
|
+
if (!snap.incidentGitHubToken) merged.incidentGitHubToken = cur.incidentGitHubToken ?? '';
|
|
1271
|
+
try {
|
|
1272
|
+
await settings.replace(NS, merged, desc.revision);
|
|
1273
|
+
const after = descriptorOf(ctx, NS);
|
|
1274
|
+
json(res, 200, { ok: true, revision: after?.revision });
|
|
1275
|
+
} catch (e) {
|
|
1276
|
+
json(res, e?.code === 'SETTINGS_CONFLICT' ? 409 : 400, { error: { code: 'settings-rejected', message: String(e?.message ?? e) } });
|
|
1277
|
+
}
|
|
1278
|
+
},
|
|
1279
|
+
}), 'dsh-key-rotation: snapshot route');
|
|
1280
|
+
|
|
1063
1281
|
// ── key route: store a key value without leaving the rotation card ──
|
|
1064
1282
|
//
|
|
1065
1283
|
// Adding a key used to mean two screens: create the credential elsewhere,
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
// lib/maintenance.js - pure helpers for #207 (expiry pre-warning) and
|
|
2
|
+
// #208 (cost budget). No I/O; the timers in index.js call these and decide
|
|
3
|
+
// whether to send webhooks.
|
|
4
|
+
|
|
5
|
+
const DAY_MS = 86400000;
|
|
6
|
+
|
|
7
|
+
/**
|
|
8
|
+
* #207: keys of `pool` whose expiresAt falls within the next `warnDays`.
|
|
9
|
+
* Returns [{ ref, expiresInDays, expiresAt }], soonest first.
|
|
10
|
+
*/
|
|
11
|
+
export function expiringSoon(pool, warnDays, now = Date.now()) {
|
|
12
|
+
if (!pool || !pool.expiresAt) return [];
|
|
13
|
+
const horizon = now + Math.max(1, warnDays ?? 7) * DAY_MS;
|
|
14
|
+
return Object.entries(pool.expiresAt)
|
|
15
|
+
.filter(([, at]) => at > now && at <= horizon)
|
|
16
|
+
.map(([ref, at]) => ({ ref, expiresAt: at, expiresInDays: Math.max(0, Math.floor((at - now) / DAY_MS)) }))
|
|
17
|
+
.sort((a, b) => a.expiresAt - b.expiresAt);
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
/** #207 dedupe: true when a day-level notification for `key` is due. */
|
|
21
|
+
export function shouldNotifyDaily(lastNotified, key, now = Date.now()) {
|
|
22
|
+
if (!lastNotified.has(key)) {
|
|
23
|
+
lastNotified.set(key, now);
|
|
24
|
+
return true;
|
|
25
|
+
}
|
|
26
|
+
const last = lastNotified.get(key);
|
|
27
|
+
if (now - last < DAY_MS) return false;
|
|
28
|
+
lastNotified.set(key, now);
|
|
29
|
+
return true;
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* #208: total spend of a pool on ISO day `day` (defaults to today)
|
|
34
|
+
* across costDays Map<ref, Map<day, cost>>.
|
|
35
|
+
*/
|
|
36
|
+
export function costForDay(costDays, day) {
|
|
37
|
+
const d = day ?? new Date().toISOString().slice(0, 10);
|
|
38
|
+
let total = 0;
|
|
39
|
+
for (const perRef of (costDays?.values() ?? [])) {
|
|
40
|
+
total += perRef.get(d) ?? 0;
|
|
41
|
+
}
|
|
42
|
+
return total;
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
/** #208: total spend over the last 7 ISO days ending today. */
|
|
46
|
+
export function costForWeek(costDays, now = Date.now()) {
|
|
47
|
+
let total = 0;
|
|
48
|
+
for (let i = 0; i < 7; i++) {
|
|
49
|
+
const d = new Date(now - i * 86400000).toISOString().slice(0, 10);
|
|
50
|
+
total += costForDay(costDays, d);
|
|
51
|
+
}
|
|
52
|
+
return total;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* #208 budget verdict for a pool: { spend, budget, ratio, warn, exceeded }.
|
|
57
|
+
* budget <= 0 -> never warn.
|
|
58
|
+
*/
|
|
59
|
+
export function budgetVerdict(spend, budget) {
|
|
60
|
+
if (!budget || budget <= 0) return { spend, budget: 0, ratio: 0, warn: false, exceeded: false };
|
|
61
|
+
const ratio = spend / budget;
|
|
62
|
+
// warn from 80%, exceeded at 100%
|
|
63
|
+
return { spend, budget, ratio, warn: ratio >= 0.8, exceeded: ratio >= 1 };
|
|
64
|
+
}
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
// lib/usage-report.js - #209 usage report rows from pool state. Pure helpers.
|
|
2
|
+
const DAY_MS = 86400000;
|
|
3
|
+
|
|
4
|
+
/**
|
|
5
|
+
* Build per-key usage rows for the last `days` ISO days (default 7).
|
|
6
|
+
* Returns [{ ref, requests, cost, active, usageByDay: {day: n} }].
|
|
7
|
+
*/
|
|
8
|
+
export function usageRows(pool, days = 7, now = Date.now()) {
|
|
9
|
+
const out = [];
|
|
10
|
+
const dayKeys = [];
|
|
11
|
+
for (let i = 0; i < Math.max(1, days); i++) dayKeys.push(new Date(now - i * DAY_MS).toISOString().slice(0, 10));
|
|
12
|
+
const refs = pool?.refs ?? [];
|
|
13
|
+
for (const ref of refs) {
|
|
14
|
+
const daysMap = pool.state.usageDays?.get(ref) ?? new Map();
|
|
15
|
+
const costMap = pool.state.costDays?.get(ref) ?? new Map();
|
|
16
|
+
let requests = 0, cost = 0;
|
|
17
|
+
const usageByDay = {};
|
|
18
|
+
for (const d of dayKeys) {
|
|
19
|
+
const r = daysMap.get(d) ?? 0;
|
|
20
|
+
const c = costMap.get(d) ?? 0;
|
|
21
|
+
requests += r;
|
|
22
|
+
cost += c;
|
|
23
|
+
usageByDay[d] = r;
|
|
24
|
+
}
|
|
25
|
+
out.push({
|
|
26
|
+
ref,
|
|
27
|
+
requests,
|
|
28
|
+
cost: Math.round(cost * 100) / 100,
|
|
29
|
+
active: pool.state.lastUsed === ref,
|
|
30
|
+
usageByDay,
|
|
31
|
+
});
|
|
32
|
+
}
|
|
33
|
+
return out;
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
/** CSV of usage rows: ref,requests,cost,active + per-day columns. */
|
|
37
|
+
export function usageCsv(rows) {
|
|
38
|
+
const dayCols = [...new Set(rows.flatMap((r) => Object.keys(r.usageByDay)))].sort();
|
|
39
|
+
const head = ['ref', 'requests', 'cost', 'active', ...dayCols];
|
|
40
|
+
const esc = (v) => {
|
|
41
|
+
const s = String(v ?? '');
|
|
42
|
+
return /[",\n]/.test(s) ? '"' + s.replace(/"/g, '""') + '"' : s;
|
|
43
|
+
};
|
|
44
|
+
const lines = [head.join(',')];
|
|
45
|
+
for (const r of rows) {
|
|
46
|
+
lines.push([esc(r.ref), r.requests, r.cost, r.active ? 'yes' : 'no', ...dayCols.map((d) => r.usageByDay[d] ?? 0)].join(','));
|
|
47
|
+
}
|
|
48
|
+
return lines.join('\n');
|
|
49
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@goodandready/dsh-key-rotation",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.29",
|
|
4
4
|
"description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"deepseek-harness",
|