@goodandready/dsh-key-rotation 0.7.28 → 0.7.30

package/README.md

@@ -44,6 +44,16 @@
   - **Usage report + CSV export** (added in 0.7.28) — `GET /dsh-key-rotation/usage?days=N[&provider=…][&format=csv]` returns per-key requests/cost over the window; an *Export CSV* button in the card downloads the same data for one provider.
   - **RPM capacity indicator** (added in 0.7.28) — `/status` carries `rpm: {used, remaining, resetMs}` per key; the card shows a ⏱ counter next to the active key.
   - **Real key test probe** (added in 0.7.28) — the per-key *Test* button now sends `probe=models` through the existing `/test` route, so it validates the key against the live API (models list + latency), not just credential presence.
+  - **Per-key weights in the GUI** (added in 0.7.29) — a small number input per key edits its round-robin weight (1 = equal share); reorder/remove/add keep the positional `weights` array in sync.
+  - **Switch notifications** (added in 0.7.29) — opt-in `switchNotify: true` sends a webhook on every key switch (who failed, why, when), deduped to at most one message per provider per `switchNotifyThrottleMs` (default 60 s).
+  - **Budget action buttons** (added in 0.7.29) — when `webhookActionToken` is set, budget notifications carry *Pause 1h* / *Reset cooldown* buttons (same callback route as exhaustion alerts).
+  - **Config snapshot export/restore** (added in 0.7.29) — *Snapshot ⬇* downloads the whole config as one JSON (token fields exported empty, keys are credential names only); *Restore* imports it back — empty token fields never wipe existing secrets, and a live-looking credential in the file is rejected.
+  - **Last probe result per key** (added in 0.7.29) — the card polls `/sandbox-cache` and shows the most recent probe outcome (✓/✕ + latency) next to each key, greyed out when older than 24 h.
+  - **Pre-exhaustion alert** (added in 0.7.30) — `warnBelowHealthy` (0 = off): while fewer keys than the threshold are healthy, a webhook fires (once per day per pool) with a *Reset cooldown* button, so the pool never silently dries up.
+  - **Telegram-native callbacks** (added in 0.7.30) — `/webhook-action` parses Telegram update envelopes (`callback_query.data`), so interactive buttons work natively; `POST {"setWebhook": {"botToken": …}}` registers the bot webhook in one call (the token is not stored).
+  - **Broken-key review** (added in 0.7.30) — keys quarantined for repeated AUTH failures get a *Re-test* button: a live probe (models list) that automatically lifts the 30-day quarantine when the key answers again.
+  - **7-day switches chart** (added in 0.7.30) — a second bar chart under the 24 h sparkline shows switches per day over the last week (client-side, no new server state).
+  - **Latency SLO alert** (added in 0.7.30) — `latencySloMs` (0 = off): when a key's p95 latency exceeds the threshold, a webhook fires (once per day per key); the card shows `p95 / SLO` per provider.
 
 ## Install
 
@@ -89,6 +99,11 @@ dsh-key-rotation:
 | `rpmLimit` | `0` | Requests-per-minute cap **per key** (0 = off). A capped key is skipped pre-emptively until its 60 s window frees up. |
 | `webhookActionToken` | `''` | Bearer token for the interactive webhook callback route. When set, exhaustion webhooks carry action buttons; empty disables them. |
 | `expiryWarnDays` | `7` | Pre-warning horizon (days) for keys with `expiresAt`: webhook + card badge. |
+| `switchNotify` | `false` | Send a webhook on every key switch (opt-in — can be chatty). |
+| `switchNotifyThrottleMs` | `60000` | Minimum gap between switch notifications for the same provider. |
+| `warnBelowHealthy` | `0` | Webhook "pool running low" while healthy keys < N (0 = off). |
+| `latencySloMs` | `0` | Latency SLO per key: webhook when p95 exceeds it (0 = off). |
+| `providers[].weights` | `[]` | Positional round-robin weights per key (editable in the card, 0.7.29). |
 | `providers[].costBudgetDaily` / `.costBudgetWeekly` | `0` | Daily / weekly spend budget per provider (0 = off). Warn webhook from 80%. |
 | `providers[].pauseOnBudget` | `false` | Pause the whole pool for 24 h when a budget is exceeded. |
 | `providers[].tags` | `[]` | Free-form labels for a provider pool, surfaced in `GET /status`. |

package/lib/client.js

@@ -67,6 +67,11 @@ window.__ModuleLoader__.load({
       rpmTitle: 'requests/min: {u} used, {r} remaining',
       budgetLabel: 'budget:',
       exportCsv: 'Export CSV',
+      weightHint: 'round-robin weight: how many times this key joins the cycle (1 = equal share)',
+      retestBroken: 'Re-test',
+      retestFail: 're-test failed - key still down',
+      snapshotExport: 'Snapshot ⬇',
+      snapshotImport: 'Restore',
       keyHint: 'The value is stored in DSH credentials and never sent back to the browser — only its last 5 characters are shown. Names are generated for you; hover a key to see the one it uses.',
       brokenKey: 'broken (3× AUTH)',
       keyExpired: 'expired',
@@ -127,6 +132,11 @@ window.__ModuleLoader__.load({
       rpmTitle: 'запросов/мин: {u} использовано, {r} осталось',
       budgetLabel: 'бюджет:',
       exportCsv: 'CSV',
+      weightHint: 'вес в круге: сколько раз ключ участвует в ротации (1 = поровну)',
+      retestBroken: 'Re-test',
+      retestFail: 'перепроверка не прошла — ключ всё ещё недоступен',
+      snapshotExport: 'Снапшот ⬇',
+      snapshotImport: 'Восстановить',
       keyHint: 'Значение хранится в учётных данных DSH и обратно в браузер не отдаётся — показываются только последние 5 символов. Имена переменных создаются автоматически; наведите на ключ, чтобы увидеть используемое имя.',
       brokenKey: 'сломан (3× AUTH)',
       keyExpired: 'истёк',
@@ -175,6 +185,24 @@ window.__ModuleLoader__.load({
       return byProvider;
     }
 
+    /** Последний probe-результат по каждому ключу (#219): /sandbox-cache. */
+    function useProbeCache() {
+      const [cache, setCache] = React.useState({});
+      React.useEffect(() => {
+        let alive = true;
+        const pull = () => {
+          fetch('/dsh-key-rotation/sandbox-cache', { headers: { accept: 'application/json' } })
+            .then((r) => (r.ok ? r.json() : null))
+            .then((data) => { if (alive && data) setCache(data); })
+            .catch(() => { /* кэш не критичен: карточка работает и без него */ });
+        };
+        pull();
+        const id = setInterval(pull, 4000);
+        return () => { alive = false; clearInterval(id); };
+      }, []);
+      return cache;
+    }
+
     // formatAgo moved to lib/client-helpers.js for testability — keep local alias for bundle self-containment
     function formatAgo(t, at) {
       if (!at) return '';
@@ -345,6 +373,7 @@ window.__ModuleLoader__.load({
 
       const val = draft ?? state.value;
       const status = useRotationStatus();
+      const probeCache = useProbeCache();
       const [resetting, setResetting] = React.useState('');
       const doReset = (providerId) => {
         setResetting(providerId);
@@ -355,9 +384,21 @@ window.__ModuleLoader__.load({
           .catch((e) => setSecretError(t('keyWriteFailed').replace('{msg}', String(e?.message ?? e))))
           .finally(() => setResetting(''));
       };
-
-
-      const keyInfo = (providerId, ref) => {
+      // #223: re-test a broken key; a successful live probe lifts the 30-day broken quarantine
+      const retestBroken = (ref) => {
+        setSecretError('');
+        fetch('/dsh-key-rotation/test', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ ref, probe: 'models' }) })
+          .then((r) => r.json())
+          .then((data) => {
+            if (data && data.ok) {
+              return fetch('/dsh-key-rotation/reset', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ ref }) });
+            }
+            setSecretError(t('retestFail'));
+            return null;
+          })
+          .catch((e) => setSecretError(t('keyWriteFailed').replace('{msg}', String(e?.message ?? e))));
+      };
+      const keyInfo = (providerId, ref) => {
         const entryStatus = status[providerId];
         if (!entryStatus || !ref) return null;
         return (entryStatus.keys ?? []).find((k) => k.ref === ref) ?? null;
@@ -432,6 +473,8 @@ window.__ModuleLoader__.load({
         const entry = { ...(providers[pIndex] ?? {}) };
         const allRefs = providers.flatMap((prov) => prov?.keys ?? []);
         entry.keys = [...(entry.keys ?? []), nextKeyRef(entry.provider, entry.keys, allRefs)];
+        // keep weights aligned with keys (#215): new key gets default weight 1
+        if (Array.isArray(entry.weights) && entry.weights.length > 0) entry.weights = [...entry.weights, 1];
         providers[pIndex] = entry;
         return { ...cur, providers };
       });
@@ -439,6 +482,10 @@ window.__ModuleLoader__.load({
         const next = [...(Array.isArray(cur.providers) ? cur.providers : [])];
         stashUndo({ type: 'key', index: pIndex, kIndex, key: next[pIndex]?.keys?.[kIndex] });
         next[pIndex] = { ...next[pIndex], keys: (next[pIndex].keys ?? []).filter((_, i) => i !== kIndex) };
+        // weights are positional - drop along with the key (#215)
+        if (Array.isArray(next[pIndex].weights) && next[pIndex].weights.length > 0) {
+          next[pIndex] = { ...next[pIndex], weights: next[pIndex].weights.filter((_, i) => i !== kIndex) };
+        }
         return { ...cur, providers: next };
       }); };
       const removeProvider = (pIndex) => setField((cur) => {
@@ -458,6 +505,26 @@ window.__ModuleLoader__.load({
         keys[kIndex] = keys[target];
         keys[target] = moved;
         entry.keys = keys;
+        // weights are positional - swap along with the keys (#215)
+        const weights = [...(entry.weights ?? [])];
+        if (weights.length > 0) {
+          const w = weights[kIndex];
+          weights[kIndex] = weights[target];
+          weights[target] = w;
+          entry.weights = weights;
+        }
+        providers[pIndex] = entry;
+        return { ...cur, providers };
+      });
+      // #215: set a single key's round-robin weight (integer >= 1)
+      const setKeyWeight = (pIndex, kIndex, weight) => setField((cur) => {
+        const n = Math.max(1, Math.min(1000, Math.floor(Number(weight) || 1)));
+        const providers = [...(cur.providers ?? [])];
+        const entry = { ...(providers[pIndex] ?? {}) };
+        const weights = [...(entry.weights ?? [])];
+        while (weights.length < (entry.keys ?? []).length) weights.push(1);
+        weights[kIndex] = n;
+        entry.weights = weights;
         providers[pIndex] = entry;
         return { ...cur, providers };
       });
@@ -549,6 +616,7 @@ window.__ModuleLoader__.load({
           h('option', { key: prov.id, value: prov.id }, prov.name + (prov.id !== prov.name ? ' — ' + prov.id : ''))));
 
         const keys = entry.keys ?? [];
+        const entryWeights = entry.weights ?? [];
         const keyRows = keys.map((key, kIndex) => {
           const st = keyStatus(entry.provider, key);
           const info = keyInfo(entry.provider, key);
@@ -601,6 +669,12 @@ window.__ModuleLoader__.load({
             }
           }
           if (info && info.lastUsedAt) meta.push(h('span', { key: 'lu', className: 'krot-tail', title: 'last used' }, formatAgo((k)=>t(k), info.lastUsedAt)));
+          // #215: per-key weight input (default 1)
+          meta.push(h('input', { key: 'w', type: 'number', min: 1, max: 1000, className: 'krot-in krot-weight',
+            value: (entryWeights[kIndex] ?? info?.weight ?? 1),
+            title: t('weightHint'),
+            onChange: (e) => setKeyWeight(pIndex, kIndex, e.target.value),
+            style: { width: '52px', padding: '2px 6px', fontSize: '12px' } }));
           // #210: RPM capacity indicator (only when rpmLimit is active)
           if (info && info.rpm) meta.push(h('span', { key: 'rpm', className: 'krot-tail',
             title: t('rpmTitle').replace('{u}', String(info.rpm.used)).replace('{r}', String(info.rpm.remaining)),
@@ -614,7 +688,20 @@ window.__ModuleLoader__.load({
               + (tr.ok && tr.latencyMs ? ' · ' + tr.latencyMs + 'ms' : ''),
             style: { color: tr.ok ? 'var(--dsw-alias-state-success-primary)' : 'var(--dsw-alias-state-error-primary)', fontWeight: 700 } },
             tr.ok ? (tr.modelsCount ? tr.modelsCount + 'm' : '✓') : '✕'));
+          // #219: last probe from the sandbox cache, greyed when older than 24h
+          else if (probeCache && probeCache[key]) {
+            const pc = probeCache[key];
+            const stale = Date.now() - (pc.at ?? 0) > 86400000;
+            meta.push(h('span', { key: 'pc', className: 'krot-tail',
+              title: 'last probe ' + (pc.at ? new Date(pc.at).toLocaleTimeString() : '') + (pc.ok ? ' ok' : ' ' + (pc.code ?? 'fail')),
+              style: { opacity: stale ? 0.4 : 0.7, color: pc.ok ? 'var(--dsw-alias-state-success-primary)' : 'var(--dsw-alias-state-error-primary)' } },
+              (pc.ok ? '✓' : '✕') + (pc.latencyMs ? ' ' + pc.latencyMs + 'ms' : '')));
+          }
           meta.push(h('button', { key: 't', className: 'krot-btn', onClick: () => doTest(key), disabled: testing === key, title: t('testKey') }, testing === key ? '…' : t('testKey')));
+          // #223: broken keys get a one-click live re-test + auto-unbreak
+          if (info && info.broken) {
+            meta.push(h('button', { key: 'rt', className: 'krot-btn', onClick: () => retestBroken(key), title: t('retestBroken') }, t('retestBroken')));
+          }
           meta.push(h('span', { key: 'a', className: 'krot-acts' },
             btn('↑', () => moveKey(pIndex, kIndex, -1), { disabled: kIndex === 0, title: t('moveUp') }),
             btn('↓', () => moveKey(pIndex, kIndex, 1), { disabled: kIndex === keys.length - 1, title: t('moveDown') }),
@@ -652,6 +739,14 @@ window.__ModuleLoader__.load({
               return h('p', { className: 'krot-hint', style: { color } }, t('budgetLabel') + ' ' + parts.join(' · '));
             })()
           : null;
+        // #225: provider p95 latency + SLO marker
+        const sloLine = providerStatus && providerStatus.p95 != null
+          ? (() => {
+              const over = providerStatus.latencySloMs && providerStatus.p95 > providerStatus.latencySloMs;
+              return h('p', { className: 'krot-hint', style: over ? { color: 'var(--dsw-alias-state-warning-primary)' } : undefined },
+                'p95 ' + providerStatus.p95 + 'ms' + (providerStatus.latencySloMs ? ' / ' + providerStatus.latencySloMs + 'ms SLO' : ''));
+            })()
+          : null;
         // #209: CSV export for this provider's usage (last 7 days)
         const exportCsv = h('button', { className: 'krot-btn', title: t('exportCsv'),
           onClick: () => {
@@ -698,8 +793,11 @@ window.__ModuleLoader__.load({
             btn(t('addKey'), () => addKey(pIndex), { title: t('addKeyTitle') }),
             switchesLine,
             budgetLine,
+            sloLine,
             exhaustionWarning,
             (providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('div', { style: { display: 'flex', gap: '2px', alignItems: 'end', height: '24px', marginTop: '4px' } }, (() => { const now = Date.now(); const buckets = Array(24).fill(0); for (const ev of providerStatus.events) { const h = Math.floor((now - ev.at) / 3600000); if (h >= 0 && h < 24) buckets[23 - h]++; } const max = Math.max(1, ...buckets); return buckets.map((c, i) => h('div', { key: i, title: c + ' switches', style: { flex: 1, background: c ? 'var(--dsw-alias-state-warning-primary)' : 'var(--dsw-alias-border-l2)', height: (c / max * 24) + 'px', minHeight: '2px', borderRadius: '2px' } })); })()) : null),
+            // #224: 7-day switches per day (client-side, from the same events)
+            (providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('div', { style: { display: 'flex', gap: '2px', alignItems: 'end', height: '16px', marginTop: '2px' } }, (() => { const now = Date.now(); const days = Array(7).fill(0); for (const ev of providerStatus.events) { const d = Math.floor((now - ev.at) / 86400000); if (d >= 0 && d < 7) days[6 - d]++; } const max = Math.max(1, ...days); return days.map((c, i) => h('div', { key: i, title: c + ' switches · day -' + (6 - i), style: { flex: 1, background: c ? 'var(--dsw-alias-state-info-primary, var(--dsw-alias-state-warning-primary))' : 'var(--dsw-alias-border-l2)', height: (c / max * 16) + 'px', minHeight: '2px', borderRadius: '2px' } })); })()) : null),
             (providerStatus && Array.isArray(providerStatus.events) && providerStatus.events.length > 0 ? h('details', { style: { fontSize: '11px', marginTop: '6px' } }, h('summary', null, 'Recent failures ('+providerStatus.events.length+')'), h('ul', { style: { margin: '4px 0 0', paddingLeft: '16px' } }, providerStatus.events.slice().reverse().map((ev, i) => h('li', { key: i, style: ev.type === 'probe' ? { opacity: .5 } : null }, new Date(ev.at).toLocaleTimeString() + ' ' + (ev.type === 'probe' ? '[probe] ' : '') + ev.ref + ' ' + ev.reason + ' cd=' + ev.cooldownMs)) )) : null),
             btn(t('resetCooldown'), () => doReset(entry.provider), { disabled: !(providerStatus && providerStatus.switches > 0) || resetting === entry.provider, title: t('resetCooldown') }),
             btn(testAllProvider === entry.provider ? t('testing') : t('testAll'), () => doTestAll(entry.provider), { disabled: testAllProvider === entry.provider, title: t('testAll') }),
@@ -753,6 +851,32 @@ window.__ModuleLoader__.load({
           }); } catch (err) { setSecretError(String(err.message || err)); } };
           reader.readAsText(f);
         } }))),
+        // #218: full snapshot export/import - one file moves the whole config
+        btn(t('snapshotExport'), () => {
+          fetch('/dsh-key-rotation/snapshot', { headers: { accept: 'application/json' } })
+            .then((r) => r.json())
+            .then((data) => {
+              const blob = new Blob([JSON.stringify(data.snapshot ?? {}, null, 2)], { type: 'application/json' });
+              const url = URL.createObjectURL(blob);
+              const a = document.createElement('a'); a.href = url; a.download = 'dsh-key-rotation-snapshot.json'; a.click(); URL.revokeObjectURL(url);
+            })
+            .catch((e) => setSecretError(String(e?.message ?? e)));
+        }, {}),
+        h('label', { className: 'krot-btn', style: { cursor: 'pointer' } }, t('snapshotImport'), h('input', { type: 'file', accept: '.json', style: { display: 'none' }, onChange: (e) => {
+          const f = e.target.files[0]; if (!f) return;
+          const reader2 = new FileReader();
+          reader2.onload = () => {
+            try {
+              const snap = JSON.parse(String(reader2.result));
+              if (!snap || typeof snap !== 'object' || Array.isArray(snap)) throw new Error('expected snapshot object');
+              fetch('/dsh-key-rotation/snapshot', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ snapshot: snap }) })
+                .then((r) => r.json().then((data) => ({ ok: r.ok, data })))
+                .then(({ ok, data }) => { if (!ok) throw new Error(data?.error?.message ?? 'import failed'); load(); })
+                .catch((err) => setSecretError(t('keyWriteFailed').replace('{msg}', String(err?.message ?? err))));
+            } catch (err) { setSecretError(String(err.message || err)); }
+          };
+          reader2.readAsText(f);
+        } })),
         h('p', { className: 'krot-hint' }, t('keyHint')),
         secretError ? h('p', { className: 'krot-err' }, secretError) : null,
         state.error ? h('p', { className: 'krot-err' }, state.error) : null,

package/lib/index.js

@@ -44,6 +44,7 @@ const NS = 'dsh-key-rotation';
 /** Config bridge route (GET / PUT / DELETE), loopback-fenced like llm-fallback. */
 const CONFIG_PATH = '/dsh-key-rotation/config';
 const STATUS_PATH = '/dsh-key-rotation/status';
+const SNAPSHOT_PATH = '/dsh-key-rotation/snapshot';
 const KEY_PATH = '/dsh-key-rotation/key';
 const RESET_PATH = '/dsh-key-rotation/reset';
 const IMPORT_PATH = '/dsh-key-rotation/import';
@@ -84,7 +85,30 @@ let rotationDisabled = false;
 // #207/#208 dedupe maps: one notification per key/window per day.
 const expiryNotifiedAt = new Map();
 const budgetNotifiedAt = new Map();
+const switchNotifiedAt = new Map();
+const lowHealthNotifiedAt = new Map();
+const sloNotifiedAt = new Map();
 const DAY_MS = 86400000;
+
+// #216: one webhook per switch, deduped to at most one message per provider
+// per switchNotifyThrottleMs. Extracted for testability.
+export function notifySwitch(runtime, pool, info, hooks = { webhookSender, now: () => Date.now() }) {
+  if (!runtime?.notifyWebhook) return;
+  const throttle = Math.max(0, runtime.switchNotifyThrottleMs ?? 60000);
+  const last = switchNotifiedAt.get(info.provider) ?? 0;
+  const now = hooks.now();
+  if (now - last < throttle) return;
+  switchNotifiedAt.set(info.provider, now);
+  hooks.webhookSender.send(runtime.notifyWebhook, {
+    title: `Key switched: ${info.provider}`,
+    text: `${info.from} failed (${info.code}) - next key in pool`,
+    provider: info.provider,
+    kind: 'switch',
+    from: info.from,
+    code: info.code,
+    at: info.at,
+  });
+}
 const MAX_EVENTS = 50;
 function pushEvent(pool, ref, reason, cooldownMs, type) {
   const ev = { at: Date.now(), ref, reason: String(reason ?? 'UNKNOWN'), cooldownMs, type: type ?? 'fail' };
@@ -191,6 +215,10 @@ export const Config = Schema.object({
   rpmLimit: Schema.number().default(0),
   webhookActionToken: Schema.string().default(''),
   expiryWarnDays: Schema.number().default(7),
+  switchNotify: Schema.boolean().default(false),
+  switchNotifyThrottleMs: Schema.number().default(60000),
+  warnBelowHealthy: Schema.number().default(0),
+  latencySloMs: Schema.number().default(0),
   providers: Schema.array(Schema.object({
     provider: Schema.string().required(),
     keys: Schema.array(Schema.string()).default([]),
@@ -586,12 +614,20 @@ export function apply(ctx, config = {}) {
           if (hit && shouldNotifyDaily(budgetNotifiedAt, pool.base + ':budget', now)) {
             console.warn(`[dsh-key-rotation] ${pool.base}: cost budget - day $${daily.toFixed(2)}/$${budget.costBudgetDaily} week $${weekly.toFixed(2)}/$${budget.costBudgetWeekly}`);
             if (runtime.notifyWebhook) {
+              // #217: budget webhook gains action buttons when a callback token
+              // is configured (the /webhook-action route already knows these ids)
+              const token = runtime.webhookActionToken ?? '';
               webhookSender.send(runtime.notifyWebhook, {
                 title: `Cost budget: ${pool.base}`,
                 text: `day $${daily.toFixed(2)} of $${budget.costBudgetDaily} · week $${weekly.toFixed(2)} of $${budget.costBudgetWeekly}` + (verdict.exceeded || wVerdict.exceeded ? ' · EXCEEDED' : ''),
                 provider: pool.base,
                 kind: 'budget',
                 spend: { daily, weekly },
+                actionToken: token || undefined,
+                actions: token ? [
+                  { id: `pause-${pool.base}`, label: 'Pause 1h' },
+                  { id: `reset-${pool.base}`, label: 'Reset cooldown' },
+                ] : undefined,
               });
             }
           }
@@ -601,6 +637,55 @@ export function apply(ctx, config = {}) {
               if ((pool.state.failedUntil.get(ref) ?? 0) < until) pool.state.failedUntil.set(ref, until);
             }
           }
+          // #221: pool running low - webhook while healthy < warnBelowHealthy
+          const warnBelow = runtime.warnBelowHealthy ?? 0;
+          if (warnBelow > 0) {
+            let healthy = 0;
+            for (const ref of pool.refs) {
+              const fu = pool.state.failedUntil.get(ref);
+              if (fu !== undefined && fu > now) continue;
+              const exp = pool.expiresAt?.[ref];
+              if (exp !== undefined && now >= exp) continue;
+              healthy++;
+            }
+            if (healthy < warnBelow && shouldNotifyDaily(lowHealthNotifiedAt, pool.base, now)) {
+              console.warn(`[dsh-key-rotation] ${pool.base}: pool running low - ${healthy}/${pool.refs.length} healthy`);
+              if (runtime.notifyWebhook) {
+                const token = runtime.webhookActionToken ?? '';
+                webhookSender.send(runtime.notifyWebhook, {
+                  title: `Pool running low: ${pool.base}`,
+                  text: `${healthy}/${pool.refs.length} keys healthy (alert below ${warnBelow})`,
+                  provider: pool.base,
+                  kind: 'low-health',
+                  healthy,
+                  total: pool.refs.length,
+                  actionToken: token || undefined,
+                  actions: token ? [{ id: `reset-${pool.base}`, label: 'Reset cooldown' }] : undefined,
+                });
+              }
+            }
+          }
+          // #225: latency SLO - webhook when a key's p95 exceeds the threshold
+          const slo = runtime.latencySloMs ?? 0;
+          if (slo > 0) {
+            for (const ref of pool.refs) {
+              const snap = latencyHistogram.snapshot(ref);
+              if (!snap.p95 || snap.p95 <= slo) continue;
+              if (!shouldNotifyDaily(sloNotifiedAt, pool.base + ':' + ref + ':slo', now)) continue;
+              console.warn(`[dsh-key-rotation] ${pool.base}: ${ref} p95 ${Math.round(snap.p95)}ms > SLO ${slo}ms`);
+              if (runtime.notifyWebhook) {
+                webhookSender.send(runtime.notifyWebhook, {
+                  title: `Latency SLO exceeded: ${pool.base}`,
+                  text: `${ref} p95 ${Math.round(snap.p95)}ms > ${slo}ms (${snap.count} samples)`,
+                  provider: pool.base,
+                  kind: 'latency-slo',
+                  ref,
+                  p95: Math.round(snap.p95),
+                  slo,
+                });
+              }
+            }
+          }
         }
       } catch (_) { /* maintenance must never crash the sweep */ }
     }, 30000);
@@ -688,7 +773,8 @@ export function apply(ctx, config = {}) {
           if (exp !== undefined) parsedExpiry[refs[i]] = exp;
         }
       }
-      return { base, refs, weightedRefs: weightedRefs.length > 0 ? weightedRefs : refs,
+      return { base, refs, weights: refs.map((_, i) => (typeof w[i] === 'number' && w[i] > 0 ? Math.floor(w[i]) : 1)),
+               weightedRefs: weightedRefs.length > 0 ? weightedRefs : refs,
                state: makeState(base), cooldownMs: poolCooldown, maxCooldownMs: poolMax, expiresAt: parsedExpiry, rpmLimit };
     };
     for (const p of cfg.providers ?? []) {
@@ -743,7 +829,7 @@ export function apply(ctx, config = {}) {
       const weekly = typeof p.costBudgetWeekly === 'number' ? p.costBudgetWeekly : 0;
       if (daily > 0 || weekly > 0) providerBudgets.set(p.provider, { costBudgetDaily: daily, costBudgetWeekly: weekly, pauseOnBudget: p.pauseOnBudget ?? false });
     }
-    return { switchCodes, cooldownMs, maxCooldownMs, notifyWebhook, notifyThreshold, incidentThreshold, concurrencyLimit, canaryProbingEnabled, canaryIntervalMs, cascade, quotaResetWindow, backupDir, backupIntervalMs, backupKeep, rotationScheduleDays, rateLimitThreshold, rpmLimit, webhookActionToken, expiryWarnDays: cfg.expiryWarnDays ?? 7, providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider, cloneIds };
+    return { switchCodes, cooldownMs, maxCooldownMs, notifyWebhook, notifyThreshold, incidentThreshold, concurrencyLimit, canaryProbingEnabled, canaryIntervalMs, cascade, quotaResetWindow, backupDir, backupIntervalMs, backupKeep, rotationScheduleDays, rateLimitThreshold, rpmLimit, webhookActionToken, expiryWarnDays: cfg.expiryWarnDays ?? 7, switchNotify: cfg.switchNotify ?? false, switchNotifyThrottleMs: cfg.switchNotifyThrottleMs ?? 60000, warnBelowHealthy: cfg.warnBelowHealthy ?? 0, latencySloMs: cfg.latencySloMs ?? 0, providerTags, providerBudgets, poolByRef, providerToPool, modelPoolByProvider, cloneIds };
   }
 
   // ── patch credentials.resolve: pool refs resolve to the next healthy key ──
@@ -944,7 +1030,16 @@ export function apply(ctx, config = {}) {
                 pool.state.lastReason = String(code ?? 'UNKNOWN');
                 pool.state.lastSwitchAt = Date.now();
                 lastFailure = chunk;
-                console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(pool.state.lastUsed ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) — next key`);
+                console.warn(`[dsh-key-rotation] ${options.provider}: key ${String(pool.state.lastUsed ?? '?')} failed (${String(code)} ${String(message).slice(0, 100)}) - next key`);
+                // #216: per-switch webhook (opt-in switchNotify), deduped per provider
+                if (buildRuntime().switchNotify && pool.state.lastUsed) {
+                  notifySwitch(buildRuntime(), pool, {
+                    provider: options.provider,
+                    from: pool.state.lastUsed,
+                    code: String(code ?? 'UNKNOWN'),
+                    at: pool.state.lastSwitchAt,
+                  });
+                }
                 switching = true;
                 break;
               }
@@ -1062,6 +1157,7 @@ export function apply(ctx, config = {}) {
       const { poolByRef, providerTags, providerBudgets } = buildRuntime();
       const base = ctx.get('credentials');
       const now = Date.now();
+      const latencySloMs = buildRuntime().latencySloMs;
       const seen = new Set();
       const providers = [];
       for (const pool of poolByRef.values()) {
@@ -1107,6 +1203,8 @@ export function apply(ctx, config = {}) {
             cooldownMsLeft: until !== undefined && until > now ? until - now : 0,
             // #210: RPM capacity snapshot (null when rpmLimit is off)
             rpm: bucketInfo(pool.state.rpmWindows, ref, pool.rpmLimit, now),
+            // #215: effective round-robin weight of this key
+            weight: pool.weights?.[pool.refs.indexOf(ref)] ?? 1,
             usage: pool.state.usageCounts?.get(ref) ?? 0,
             byModel: pool.state.byModel?.get(ref) ? Object.fromEntries(pool.state.byModel.get(ref)) : {},
             usageDays: pool.state.usageDays?.get(ref) ? Object.fromEntries(pool.state.usageDays.get(ref)) : {},
@@ -1127,6 +1225,12 @@ export function apply(ctx, config = {}) {
           lastExhaustionAt: pool.state.lastExhaustionAt ?? null,
           exhaustionCount: pool.state.exhaustionCount ?? 0,
           totalUsage: [...(pool.state.usageCounts?.values() ?? [])].reduce((a, b) => a + b, 0),
+          // #225: aggregate p95 across the pool's keys
+          p95: (() => {
+            const vals = (pool.refs ?? []).map((r) => latencyHistogram.snapshot(r)).filter((s) => s && s.p95 != null).map((s) => s.p95);
+            return vals.length ? Math.round(Math.max(...vals)) : null;
+          })(),
+          latencySloMs,
           events: (pool.state.events ?? []).slice(-50),
           healthScore: computeHealthScore(pool.state),
           // #208: today/week spend + configured budget for the card
@@ -1182,6 +1286,58 @@ export function apply(ctx, config = {}) {
     },
   }), 'dsh-key-rotation: usage route');
 
+  // #218: full config snapshot - one JSON file to move between machines.
+  // Secret values never travel: only credential/env names. Token fields are
+  // exported as empty strings; on import they keep existing values when empty.
+  ctx.effect(() => ctx.webServer.register({
+    kind: 'exact',
+    path: SNAPSHOT_PATH,
+    handler: async (req, res) => {
+      if (req.method !== 'GET' && req.method !== 'POST') { json(res, 405, { error: { code: 'method', message: 'GET (export) or POST (import) only' } }); return; }
+      if (!isTrustedBridgeRequest(req)) { json(res, 403, { error: { code: 'forbidden', message: 'dsh-key-rotation: snapshot is local-only' } }); return; }
+      if (req.method === 'GET') {
+        const descriptor = descriptorOf(ctx, NS);
+        const value = descriptor?.value ?? {};
+        const exportable = { ...value };
+        // token-shaped fields stay empty in the file; refs are names, not secrets
+        exportable.webhookActionToken = '';
+        if (exportable.incidentGitHubToken) exportable.incidentGitHubToken = '';
+        json(res, 200, { at: Date.now(), version: 1, snapshot: exportable });
+        return;
+      }
+      // POST = import: { snapshot } -> merge with current section, PUT semantics
+      let body;
+      try { body = await readJson(req); } catch (e) { json(res, 400, { error: { code: 'bad-request', message: String(e?.message ?? e) } }); return; }
+      const snap = body?.snapshot;
+      if (!snap || typeof snap !== 'object' || Array.isArray(snap)) { json(res, 400, { error: { code: 'bad-format', message: 'dsh-key-rotation: POST requires {"snapshot": {...}}' } }); return; }
+      // #200 leak guard applies to imported content too
+      try {
+        const masked = structuredClone(snap);
+        if (masked.incidentGitHubToken) masked.incidentGitHubToken = '***';
+        if (masked.webhookActionToken) masked.webhookActionToken = '***';
+        if (masked.notifyWebhook) masked.notifyWebhook = '***';
+        const findings = findSecrets(JSON.stringify(masked));
+        if (findings.length > 0) { json(res, 400, { error: { code: 'secret-in-snapshot', message: 'dsh-key-rotation: snapshot carries a live-looking credential', findings } }); return; }
+      } catch { /* scanning must never block a valid import */ }
+      const settings = ctx.get('settings');
+      if (!settings) { json(res, 503, { error: { code: 'settings-rejected', message: 'dsh-key-rotation: no settings provider' } }); return; }
+      const desc = descriptorOf(ctx, NS);
+      if (desc === void 0) { json(res, 500, { error: { code: 'settings-rejected', message: 'dsh-key-rotation: namespace missing' } }); return; }
+      const cur = desc.value ?? {};
+      // empty token fields in the file keep the current values (never wipe a secret)
+      const merged = { ...cur, ...snap };
+      if (!snap.webhookActionToken) merged.webhookActionToken = cur.webhookActionToken ?? '';
+      if (!snap.incidentGitHubToken) merged.incidentGitHubToken = cur.incidentGitHubToken ?? '';
+      try {
+        await settings.replace(NS, merged, desc.revision);
+        const after = descriptorOf(ctx, NS);
+        json(res, 200, { ok: true, revision: after?.revision });
+      } catch (e) {
+        json(res, e?.code === 'SETTINGS_CONFLICT' ? 409 : 400, { error: { code: 'settings-rejected', message: String(e?.message ?? e) } });
+      }
+    },
+  }), 'dsh-key-rotation: snapshot route');
+
   // ── key route: store a key value without leaving the rotation card ──
   //
   // Adding a key used to mean two screens: create the credential elsewhere,
@@ -1496,6 +1652,28 @@ export function apply(ctx, config = {}) {
       if (!action && typeof body?.callback_data === 'string') {
         try { action = String(JSON.parse(body.callback_data)?.id ?? ''); } catch { action = ''; }
       }
+      // #222: Telegram update envelope {update_id, callback_query:{data}}
+      if (!action && typeof body?.callback_query?.data === 'string') {
+        try { action = String(JSON.parse(body.callback_query.data)?.id ?? ''); } catch { action = ''; }
+      }
+      // #222: Telegram setWebhook registration helper
+      if (typeof body?.setWebhook === 'object' && body.setWebhook) {
+        const botToken = typeof body.setWebhook.botToken === 'string' ? body.setWebhook.botToken : '';
+        if (!botToken) { json(res, 400, { error: { code: 'bad-request', message: 'dsh-key-rotation: setWebhook.botToken required' } }); return; }
+        // derive the public URL from request headers; explicit URL wins
+        const url = typeof body.setWebhook.url === 'string' && body.setWebhook.url ? body.setWebhook.url : `https://${String(req.headers.host ?? '')}/dsh-key-rotation/webhook-action`;
+        try {
+          const hookRes = await fetch(`https://api.telegram.org/bot${botToken}/setWebhook`, {
+            method: 'POST', headers: { 'content-type': 'application/json' },
+            body: JSON.stringify({ url, allowed_updates: ['callback_query'] }),
+          });
+          const hookData = await hookRes.json().catch(() => ({}));
+          json(res, 200, { ok: hookRes.ok, url, telegram: hookData });
+        } catch (e) {
+          json(res, 502, { error: { code: 'telegram-failed', message: String(e?.message ?? e) } });
+        }
+        return;
+      }
       if (!action) { json(res, 400, { error: { code: 'bad-action', message: 'dsh-key-rotation: no action in payload' } }); return; }
       const provider = action.startsWith('pause-') || action.startsWith('reset-') ? action.replace(/^(pause|reset)-/, '') : '';
       try {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@goodandready/dsh-key-rotation",
-  "version": "0.7.28",
+  "version": "0.7.30",
   "description": "Per-provider API key rotation for DeepSeek Harness: a key pool per provider, auto-created clone routes, and switching to the next key on quota/rate-limit errors. Includes a Settings section (Key Rotation) to edit the key pools, cooldown and switch codes.",
   "keywords": [
     "deepseek-harness",