@goodandready/dsh-key-rotation 0.7.30 → 0.7.32

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2026 dsh-key-rotation contributors
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2026 GooDAnDReaDY
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,172 +1,254 @@
-# dsh-key-rotation
-
-**Per-provider API key rotation** for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (dsh). Instead of failing on a quota/rate-limit error, the plugin transparently retries the request on the **next healthy key** in a per-provider pool.
-
-> Hermes-style rotation: every configured provider has a key pool; when a key's limit is exhausted, the request is retried on the next key. Exhausted keys stay in cooldown and return to rotation after `cooldownMs`.
-
-## What it does
-
-- **Key pools per provider** — list the API keys (as credential/env names) that a provider may rotate through.
-- **The provider you picked stays the provider** — rotation swaps the key, never the route, so a multi-call turn does not break. Legacy clone routes remain registered but are hidden from the model dropdown.
-- **Transparent on-failure rotation** — on a switchable error (`QUOTA`, `RATE_LIMIT`, `AUTH`/`INVALID`…) the request is retried on the next key.
-- **Cooldown** — an exhausted key is skipped for `cooldownMs`, then returns.
-- **Dead/revoked key handling** — an auth/invalid key rotates to the next pool key instead of erroring out.
-- **Settings GUI** — a **Settings → Key Rotation** section to manage everything without touching config files:
-  - **add a key in one place** — press *Add key*, paste the value, done. The credential name is generated for you (`<PROVIDER>_API_KEY`, then `_2`, `_3`, …) and shown only on hover; the card lists keys as *Key 1*, *Key 2*.
-  - **live key status** — per key: in use / ready / cooling down with a countdown / **no such credential**, which is what catches a mistyped name that would otherwise fail silently.
-  - **rotation counter** — how many times a provider switched key, on which failure, and how long ago.
-  - **key order** — ↑/↓ buttons; the order of keys is the order they are tried.
-  - **switch codes as checkboxes** instead of a comma-separated string.
-  - **Exponential backoff** — repeated failures on the same key double its cooldown (base → ×2 → ×4 → cap ×8), so a dead key is not retried every window.
-  - **Reset cooldown** — a *Reset cooldown* button in the card clears a provider's cooldown immediately (also via `POST /dsh-key-rotation/reset`).
-  - **Env bootstrap** — if a pool ref (e.g. `MYPROVIDER_API_KEY`) is already set in `process.env`, it is treated as a transient credential without needing a DSH credential first.
-  - **Per-provider cooldown** — override `cooldownMs` (and `maxCooldownMs`) per provider, fallback to the global values.
-  - **Exhaustion warning** — when every key is cooling, a red warning appears in the card and `lastExhaustionAt`/`exhaustionCount` are exposed via `GET /dsh-key-rotation/status`.
-  - **Failure log** — last 20 failures per provider (`at`, `ref`, `reason`, `cooldownMs`) via `/status` and a collapsible *Recent failures* list.
-  - **Non-stream safety net** — an `agent/request-error` hook retries sync calls (embeddings, batch) with the next key when the error is switchable.
-  - **Search/filter providers** — a search box above the list filters providers by id.
-  - **Bulk edit cooldown** — checkboxes per provider + a cooldown input + *Apply to selected*.
-  - **Undo delete** — after removing a key or provider, an *Undo* bar appears for 5 seconds.
-  - **Per-key last used** — `lastUsedAt` shown as "ago" next to each key.
-  - **Total requests badge** — sum of usage across a provider's keys, shown in its header.
-  - **Export single provider** — ⬇ button exports just that provider's entry.
-  - **Import from .env** — pick a `.env` file; `KEY=val` names are added to the first pool.
-  - **Copy key name** — click a *Key N* label to copy its ref name.
-  - **Sort by usage** — ⇅ sorts a provider's keys by usage (desc).
-  - **Probe history** — health-probe events appear greyed in *Recent failures*.
-  - **RPM token bucket** (added in 0.7.27) — optional `rpmLimit` caps requests per minute **per key**: a key that hit the cap is skipped pre-emptively (before a real request is sent) and returns when its sliding 60s window frees up. `0` (default) disables it.
-  - **Tier/model-aware routing** (added in 0.7.27) — a pool defined for a model also serves model versions in its family: an exact model pool wins, then the longest name-prefix match (a pool for `gpt-4o` serves `gpt-4o-mini-2024`), then the provider base pool. Provider `tags` are free-form labels surfaced in `/status` for grouping.
-  - **Interactive webhooks** (added in 0.7.27) — set `webhookActionToken` and exhaustion notifications gain action buttons on Telegram (`inline_keyboard`), Discord (buttons) and Slack (actions block): *Reset cooldown* and *Pause 1h* per pool. Buttons call back to `POST /dsh-key-rotation/webhook-action` with a bearer token; actions `reset-<provider>`, `pause-<provider>` (1 h), `disable-rotation`, `enable-rotation`.
-  - **Token leak detector** (added in 0.7.27) — recognizes live key shapes (OpenAI `sk-`, Anthropic `sk-ant-`, Google `AIza…`, GitHub `ghp_…`, AWS `AKIA…`, Slack `xox…`, Telegram bot tokens, Stripe, PEM blocks). Saving the config section with a real key pasted into a wrong field is rejected (`400 secret-in-config`); the key-save response carries `looksLikeSecret`, and the card hints when the value does not look like an API key.
-  - **Header status chip** (added in 0.7.27) — a small chip in the session header shows one dot for all pools (green = all keys healthy, amber = some cooling, red = a pool fully exhausted) plus a `healthy/total` counter; polls every 4 s.
-  - **Key expiry pre-warning** (added in 0.7.28) — `expiryWarnDays` (default 7): keys expiring within that horizon send one webhook notification per key per day and show a yellow *expires in Nd* badge in the card.
-  - **Provider cost budget** (added in 0.7.28) — per-provider `costBudgetDaily` / `costBudgetWeekly` (0 = off): a warn webhook fires from 80% spend, and with `pauseOnBudget: true` the whole pool is paused for 24 h at 100%. The card shows a live budget line with warn/red coloring.
-  - **Usage report + CSV export** (added in 0.7.28) — `GET /dsh-key-rotation/usage?days=N[&provider=…][&format=csv]` returns per-key requests/cost over the window; an *Export CSV* button in the card downloads the same data for one provider.
-  - **RPM capacity indicator** (added in 0.7.28) — `/status` carries `rpm: {used, remaining, resetMs}` per key; the card shows a ⏱ counter next to the active key.
-  - **Real key test probe** (added in 0.7.28) — the per-key *Test* button now sends `probe=models` through the existing `/test` route, so it validates the key against the live API (models list + latency), not just credential presence.
-  - **Per-key weights in the GUI** (added in 0.7.29) — a small number input per key edits its round-robin weight (1 = equal share); reorder/remove/add keep the positional `weights` array in sync.
-  - **Switch notifications** (added in 0.7.29) — opt-in `switchNotify: true` sends a webhook on every key switch (who failed, why, when), deduped to at most one message per provider per `switchNotifyThrottleMs` (default 60 s).
-  - **Budget action buttons** (added in 0.7.29) — when `webhookActionToken` is set, budget notifications carry *Pause 1h* / *Reset cooldown* buttons (same callback route as exhaustion alerts).
-  - **Config snapshot export/restore** (added in 0.7.29) — *Snapshot ⬇* downloads the whole config as one JSON (token fields exported empty, keys are credential names only); *Restore* imports it back — empty token fields never wipe existing secrets, and a live-looking credential in the file is rejected.
-  - **Last probe result per key** (added in 0.7.29) — the card polls `/sandbox-cache` and shows the most recent probe outcome (✓/✕ + latency) next to each key, greyed out when older than 24 h.
-  - **Pre-exhaustion alert** (added in 0.7.30) — `warnBelowHealthy` (0 = off): while fewer keys than the threshold are healthy, a webhook fires (once per day per pool) with a *Reset cooldown* button, so the pool never silently dries up.
-  - **Telegram-native callbacks** (added in 0.7.30) — `/webhook-action` parses Telegram update envelopes (`callback_query.data`), so interactive buttons work natively; `POST {"setWebhook": {"botToken": …}}` registers the bot webhook in one call (the token is not stored).
-  - **Broken-key review** (added in 0.7.30) — keys quarantined for repeated AUTH failures get a *Re-test* button: a live probe (models list) that automatically lifts the 30-day quarantine when the key answers again.
-  - **7-day switches chart** (added in 0.7.30) — a second bar chart under the 24 h sparkline shows switches per day over the last week (client-side, no new server state).
-  - **Latency SLO alert** (added in 0.7.30) — `latencySloMs` (0 = off): when a key's p95 latency exceeds the threshold, a webhook fires (once per day per key); the card shows `p95 / SLO` per provider.
-
-## Install
-
-```bash
-# From npm after publishing:
-dsh plugin --profile web add @goodandready/dsh-key-rotation
-
-# From GitHub:
-dsh plugin --profile web add github:GooDAnDReaDY/dsh-key-rotation
-
-# Locally from a checkout:
-dsh plugin --profile web add /path/to/dsh-key-rotation
-```
-
-Restart the Web UI afterwards.
-
-## Configure
-
-### Web GUI (recommended)
-
-Open **Settings → Key Rotation** and, for each provider, list the credential names of its keys. The plugin stores this in the `dsh-key-rotation` settings namespace (same place as `settings.yaml`).
-
-### `settings.yaml`
-
-```yaml
-dsh-key-rotation:
-  switchCodes: [QUOTA, RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT, EMPTY_RESPONSE, UNKNOWN_MODEL]
-  cooldownMs: 60000
-  providers:
-    # `provider` is the id of a provider registered with dsh, as it appears
-    # in Settings -> Models. `keys` are CREDENTIAL NAMES, never key values.
-    - provider: my-provider
-      keys: [MY_PROVIDER_API_KEY, MY_PROVIDER_API_KEY_2, MY_PROVIDER_API_KEY_3]
-    - provider: another-provider
-      keys: [ANOTHER_PROVIDER_API_KEY, ANOTHER_PROVIDER_API_KEY_2]
-```
-
-| Field | Default | Description |
-|---|---|---|
-| `switchCodes` | `[QUOTA, RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT, EMPTY_RESPONSE, UNKNOWN_MODEL]` | Error codes that trigger a key switch. |
-| `cooldownMs` | `60000` | How long an exhausted key stays out of rotation. |
-| `providers` | — | `[{ provider, keys: [envName, ...] }]`. `keys` are credential/env **names**, not the key values themselves. |
-| `rpmLimit` | `0` | Requests-per-minute cap **per key** (0 = off). A capped key is skipped pre-emptively until its 60 s window frees up. |
-| `webhookActionToken` | `''` | Bearer token for the interactive webhook callback route. When set, exhaustion webhooks carry action buttons; empty disables them. |
-| `expiryWarnDays` | `7` | Pre-warning horizon (days) for keys with `expiresAt`: webhook + card badge. |
-| `switchNotify` | `false` | Send a webhook on every key switch (opt-in — can be chatty). |
-| `switchNotifyThrottleMs` | `60000` | Minimum gap between switch notifications for the same provider. |
-| `warnBelowHealthy` | `0` | Webhook "pool running low" while healthy keys < N (0 = off). |
-| `latencySloMs` | `0` | Latency SLO per key: webhook when p95 exceeds it (0 = off). |
-| `providers[].weights` | `[]` | Positional round-robin weights per key (editable in the card, 0.7.29). |
-| `providers[].costBudgetDaily` / `.costBudgetWeekly` | `0` | Daily / weekly spend budget per provider (0 = off). Warn webhook from 80%. |
-| `providers[].pauseOnBudget` | `false` | Pause the whole pool for 24 h when a budget is exceeded. |
-| `providers[].tags` | `[]` | Free-form labels for a provider pool, surfaced in `GET /status`. |
-
-### Interactive webhook actions (added in 0.7.27)
-
-When `webhookActionToken` is set, the exhaustion webhook payload is formatted per platform (Telegram / Discord / Slack, detected from the URL) with two buttons. A button POSTs to the plugin route:
-
-```bash
-curl -X POST http://127.0.0.1:3080/dsh-key-rotation/webhook-action \
-  -H "Authorization: Bearer <webhookActionToken>" \
-  -H "content-type: application/json" \
-  -d '{"action": "reset-my-provider"}'
-```
-
-Actions: `reset-<provider>` (clear cooldowns), `pause-<provider>` (pause the whole pool for 1 hour), `disable-rotation` / `enable-rotation` (global). Platform callback payloads (`callback_data`, Discord `custom_id`, Slack button `value`) are accepted too. Without the correct bearer token the route answers `401`; without a configured token — `503`.
-
-### Usage report (added in 0.7.28)
-
-```bash
-curl "http://127.0.0.1:3080/dsh-key-rotation/usage?days=7"           # JSON
-curl "http://127.0.0.1:3080/dsh-key-rotation/usage?format=csv&days=30&provider=my-provider" > usage.csv
-```
-
-Per-key rows: `requests`, `cost`, `active`, and per-day counts over the window (1–90 days, default 7).
-
-### How keys are stored
-
-The plugin config only ever references keys by **name** (e.g. `MY_PROVIDER_API_KEY`). The values live in the dsh **Credentials** service or `$DSH_HOME/.credentials.yaml` — never in the plugin config.
-
-A key typed into the Key Rotation card is written to that same credentials store: the value travels to the host once and is never sent back to the browser. Only its **last 5 characters** are, so two keys can be told apart in the UI. A key supplied by the launching environment is shown as read-only, because overwriting it here would be shadowed anyway.
-
-## How it works
-
-```
-request ──► {provider: rotation} clone route ──► pick next healthy key in pool
-        ┌────────┐   on switchable failure retry with next key, stay in cooldown
-        └─────────┘
-```
-
-- The plugin patches `ctx.credentials.resolve` so a pool reference resolves to the current healthy key (round-robin, skipping keys in cooldown).
-- It intercepts `llm/stream` to retry the request on the next key after a switchable failure, instead of surfacing the error to the caller. The hook is deliberately **not** `async`: the loop iterates its result directly, and returning a promise breaks every turn.
-- The provider identity never changes — only the resolved key does — which keeps the adapter's replay state consistent across a multi-call turn.
-
-Two local-only routes back the card: `GET /dsh-key-rotation/status` (key state, rotation counters, last 5 characters of each key) and `PUT|DELETE /dsh-key-rotation/key` (store or drop one key value). Both refuse anything that is not a same-origin request from loopback.
-
-## Structure
-
-```
-dsh-key-rotation/
-├── package.json            # dsh bundle/plugin metadata + peerDependencies
-├── cordis.patch.yml        # bundle layer: registers the virtual route "rotation"
-├── lib/index.js            # host: pools, credentials.resolve patch, stream retry
-├── lib/client.js           # browser: Settings → Key Rotation panel
-└── README.md
-```
-
-## Security notes
-
-- Key **values** never leave your Credentials store; the plugin config only holds env/credential **names**.
-- `switchCodes` are error classification strings, not expressions — no secrets involved.
-- The leak detector (0.7.27) rejects config saves that carry a live-looking key outside the credential fields; only the credential **type** and a short preview are ever reported, never the value.
-
-## License
-
-MIT
+# 📦 @goodandready/dsh-key-rotation
+
+<div align="center">
+
+<h3>Enterprise-Grade Transparent API Key Rotation, Rate-Limit Pre-emption & Failover Cascade for DeepSeek Harness</h3>
+
+<p align="center">
+  <a href="https://www.npmjs.com/package/@goodandready/dsh-key-rotation"><img src="https://img.shields.io/npm/v/@goodandready/dsh-key-rotation.svg?style=for-the-badge&color=6366f1&labelColor=1e1b4b" alt="npm version"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/github/license/GooDAnDReaDY/dsh-key-rotation.svg?style=for-the-badge&color=10b981&labelColor=064e3b" alt="license"></a>
+  <a href="https://github.com/topics/dsh-plugin"><img src="https://img.shields.io/badge/DSH-Plugin-8b5cf6.svg?style=for-the-badge&labelColor=2e1065" alt="DSH Plugin"></a>
+  <a href="https://nodejs.org"><img src="https://img.shields.io/badge/Node-20%2B-f59e0b.svg?style=for-the-badge&labelColor=451a03" alt="Node version"></a>
+</p>
+
+<!-- Showcase Link -->
+<p align="center">
+  <a href="https://goodandready.app/"><img src="https://img.shields.io/badge/🌐_DSH_Hub-goodandready.app-ff4500.svg?style=for-the-badge&labelColor=1a1a2e" alt="GoodAndReady Showcase"></a>
+</p>
+
+<p align="center">
+  <a href="README.md"><b>🇬🇧 English</b></a> •
+  <a href="README.ru.md"><b>🇷🇺 Русский</b></a> •
+  <a href="README.zh.md"><b>🇨🇳 中文说明</b></a>
+</p>
+
+</div>
+
+---
+
+## ⚡ Overview & The Problem
+
+### 🛠️ What's New in v0.7.32 (Stability & Bugfix Release)
+- **🔍 Resolved Key Probing BaseURL**: Fixed `resolveBaseUrl` to map key credential refs to owning provider pools, restoring live `probeModels` testing.
+- **🛡️ Guarded Cascade Recursion**: Prevented call stack overflow in cross-provider failover when circular cascade chains occur.
+- **🕒 Accurate Midnight PST Resets**: Corrected UTC-8 timezone calculation offset sign for calendar quota reset windows.
+- **🧹 Lifecycle Timer Cleanup**: Wrapped `canaryTimer` and `selfHealTimer` in Cordis effect scopes, eliminating background orphaned intervals on hot reload.
+- **⚡ Stale Lock Recovery in Load Balancer**: Added expired lock detection to `pickLeastLoaded` for uninterrupted least-connections routing.
+- **🌐 Full Chinese Localization**: Added complete `zh` locale dictionary to the React settings dashboard for comprehensive 3-language parity.
+
+
+### 🚀 What's New in v0.7.31
+- **⚡ O(1) TokenBucket Accumulator**: Upgraded rate limiting math to O(1) time and zero-allocation memory with adaptive header synchronization.
+- **🛡️ Soft vs Hard Backoff**: Differentiates transient infrastructure drops (502/503/timeouts: 10s flat cooldown) from hard quota errors (progressive doubling).
+- **⏳ Penalty Decay**: Stable keys that operate cleanly automatically decay their failure penalty multiplier every hour.
+- **🎲 Cooldown Jitter**: Adds ±12.5% random dispersion to recovery timers, eliminating thundering herd stampedes.
+- **🎯 Addressable Canary Probing**: Support for probing target pool models with lightweight single-token verification pings.
+- **📊 TTFT Percentiles (p50 / p95 / p99)**: Sub-second high-resolution latency percentile tracking across all key pools.
+- **🔔 Webhook Alert Digest**: Aggregates multiple rapid switch/cooldown events into consolidated incident digests for Telegram, Discord, and Slack.
+- **🧹 30-Day Usage Compaction**: Automatic bounded memory management with 30-day rolling window data pruning.
+- **✨ Optimistic UI & Filter Pills**: Instant zero-latency UI updates on reset, plus `All`, `Ready`, `In Cooldown`, and `With Errors` quick filter chips.
+
+
+High-throughput autonomous agent workflows, parallel subagent swarms, and multi-turn tool loops inevitably hit upstream API rate limits (HTTP 429, RPM/TPM exhaustion, daily quotas, or sudden provider outages). In standard DeepSeek Harness deployments, a single exhausted API key breaks the entire agent execution chain, requiring manual intervention and destroying the session's replay state.
+
+**`dsh-key-rotation`** provides a seamless, enterprise-ready **transparent API key pooling, pre-emptive rate-limiting, and cross-provider failover engine** built natively on the Cordis microkernel architecture.
+
+Unlike naive routing proxies that alter provider identifiers, `dsh-key-rotation` hooks into `ctx.credentials.resolve` and intercepts `llm/stream` at runtime:
+* **The provider identity never changes**: Agent replay states, multi-call turns, and tool schemas remain 100% consistent.
+* **Pre-emptive Token Bucket**: Throttled keys are skipped *before* issuing network calls, eliminating retry latency.
+* **Least-Connections Concurrency Control**: Balances in-flight streams across keys to prevent burst saturation.
+* **Autonomous Self-Healing & Cascades**: Proactively tests quarantined keys via canary probes and smoothly escalates to fallback providers if an entire pool is exhausted.
+
+---
+
+## 🏗️ Architecture & Request Lifecycle
+
+```mermaid
+graph LR
+    subgraph ClientLayer ["Client & Agent Turn"]
+        UserMsg["User / Subagent Message"] --> Adapter["pi-ai Model Adapter"]
+    end
+
+    subgraph RotationEngine ["dsh-key-rotation Core Engine"]
+        Adapter --> StreamHook["llm/stream Interceptor"]
+        StreamHook --> BucketCheck{"Token Bucket\nRPM / TPM Check"}
+        BucketCheck -->|Under Limit| ConcurrencyCheck{"Concurrency Tracker\nLeast-Connections"}
+        BucketCheck -->|Exceeded| NextKey1["Pick Next Healthy Key"]
+        ConcurrencyCheck -->|Slot Available| KeyResolver["ctx.credentials.resolve"]
+        ConcurrencyCheck -->|Saturated| NextKey1
+        
+        KeyResolver --> ActiveKey["Active Key (In Use)"]
+        
+        ActiveKey -.->|HTTP 429 / Quota / Error| Failover["Instant Failover Handler"]
+        Failover --> BackoffCalc["Exponential Backoff & Quarantine"]
+        Failover --> NextKey2["Retry Next Key (Zero Token Loss)"]
+        Failover -.->|All Pool Keys Exhausted| CascadeEngine["Cross-Provider Cascade"]
+        
+        BackoffCalc --> QuotaWindow["Calendar Reset / Midnight Window"]
+        BackoffCalc --> CanaryProbe["Active Canary Prober (Sandbox Ping)"]
+        CanaryProbe -->|Verified Healthy| PoolReady["Restored to Ready Pool"]
+    end
+
+    subgraph UpstreamLayer ["Model Provider Endpoints"]
+        ActiveKey --> UpstreamAPI["Primary Provider API"]
+        CascadeEngine --> FallbackAPI["Backup Provider API"]
+    end
+
+    style ClientLayer fill:#1e1e2e,stroke:#89b4fa,stroke-width:2px,color:#cdd6f4
+    style RotationEngine fill:#181825,stroke:#cba6f7,stroke-width:2px,color:#cdd6f4
+    style UpstreamLayer fill:#11111b,stroke:#a6e3a1,stroke-width:2px,color:#cdd6f4
+```
+
+---
+
+## ✨ Full Feature Breakdown
+
+### 🔄 1. Transparent Rotation & Failover
+* **Unchanged Provider Identity**: Rotates only the underlying resolved API credential ref, never the provider ID. Prevents `INVALID_REPLAY_STATE` crashes in `pi-ai` multi-turn sessions.
+* **Zero-Token-Loss Stream Retries**: If an API key encounters an error before the first content chunk is emitted, the request is transparently re-dispatched to the next healthy key in the pool.
+* **Comprehensive Switch Codes**: Automatically fails over on `QUOTA`, `RATE_LIMIT`, `SERVER`, `TIMEOUT`, `TRANSPORT`, `EMPTY_RESPONSE`, `UNKNOWN_MODEL`, `AUTH`, and `INVALID` error codes.
+* **Intelligent Message Pattern Matching**: Fallback regex classifier (`SWITCHABLE_MESSAGE_PATTERN`) identifies text-based quota/rate-limit errors thrown as generic exceptions by upstream SDKs.
+* **Non-Streaming Safety Net**: Synchronous calls (e.g., embeddings, batch evaluations) are protected via the `agent/request-error` lifecycle hook.
+
+### ⏱️ 2. Rate-Limit Pre-emption & Concurrency Control
+* **Token Bucket / Leaky Bucket (`lib/bucket.js`)**: Sliding-window tracking of Requests Per Minute (`rpmLimit`) and Tokens Per Minute (`tpmLimit`). Quarantines saturated keys *before* dispatching network requests, preventing 429 roundtrips.
+* **Least-Connections Balancer (`lib/concurrency.js`)**: Tracks active in-flight streams per key (`inFlight`). Distributes concurrent requests evenly across available credentials and enforces `maxConcurrency` limits.
+* **Stale Lock Auto-Release**: Deadlocks from disconnected clients or aborted network sockets are automatically purged after 5 minutes.
+
+### 🛡️ 3. Autonomous Healing & Cascade Escalation
+* **Cross-Provider Failover Cascade (`lib/cascade.js`)**: If all keys for a selected provider are in cooldown, requests automatically cascade to an alternative fallback provider pool (e.g., primary provider → fallback proxy / secondary provider).
+* **Active Canary Prober (`lib/canary.js`)**: Before releasing a key from quarantine, a lightweight background probe (`/models` probe or single-token check via `SandboxRunner`) validates upstream availability without exposing real user traffic to risk.
+* **Calendar & Rolling Quota Reset Windows (`lib/quota-window.js`)**: Supports scheduled quota reset alignments (`midnight_utc`, `midnight_pst`, and `rolling_24h`) so daily free/tier quotas unfreeze exactly when upstream resets them.
+* **Adaptive Exponential Backoff (`lib/pool.js`)**: Successive failures on a key double its quarantine duration (base → ×2 → ×4 → cap ×8). Successful requests gradually restore healthy status.
+
+### 🎯 4. Model-Aware & Geolocation Routing
+* **Model Sub-Pools (`lib/pool.js`)**: Configure dedicated key pools for specific model tiers (e.g. reasoning/heavy models vs fast/cheap utility models).
+* **Tag-Based Routing**: Assign operational tags (`production`, `background`, `eval`) to match key usage with workload priorities.
+* **Region Mapping (`lib/region.js`)**: Route queries through geographically optimal credentials and endpoints.
+
+### 📊 5. Observability, Telemetry & Webhooks
+* **Interactive Multi-Platform Webhooks (`lib/webhook.js`)**: Dispatches rich notifications with HMAC-signed action buttons for **Telegram** (Inline Keyboards), **Discord** (Action Rows), and **Slack** (Block Kit). Administrators can click buttons to reset cooldowns or pause providers directly from their mobile chat.
+* **Usage & Cost Reporting (`lib/usage-report.js`)**: Per-key daily request counters and estimated cost breakdown with one-click CSV/JSON export (`GET /dsh-key-rotation/usage-report`).
+* **Latency SLO & Histogram (`lib/histogram.js`)**: Tracks Time-To-First-Token (TTFT) and stream durations with health score degradation scoring (`0..100`).
+* **Automated Incident Reporting (`lib/incident.js`)**: Lazily creates structured GitHub Issues on sustained upstream outages.
+* **Shadow Traffic Routing (`lib/shadow.js`)**: Fork a configurable percentage of live requests to evaluate secondary providers in shadow mode.
+
+---
+
+## 🖥️ Rich Web GUI & Dashboard
+
+Access full visual management under **Settings → Key Rotation** or via the Header quick-widget.
+
+| Interface Feature | Description |
+|---|---|
+| **Header Status Widget** | Compact live badge in DSH header: 🟢 `All Healthy` \| 🟡 `Cooldown Active` \| 🔴 `Pool Exhausted` with quick popover actions. |
+| **1-Click Health Matrix** | "Health Matrix" dashboard running parallel sandbox probes across all providers, keys, and models with TTFT latency and status badges. |
+| **Instant Key Provisioning** | Add keys with auto-generated names (`<PROVIDER>_API_KEY`, `_2`, `_3`) and automatic key-tail disambiguation. |
+| **Live Status Badges** | Visual states: `In Use`, `Ready`, `Cooling Down` (with live countdown timer), and `Not Found`. |
+| **Drag & Priority Ordering** | Reorder keys with <kbd>↑</kbd> and <kbd>↓</kbd> buttons to fine-tune selection precedence. |
+| **Switch Code Toggles** | Interactive checkboxes for switchable error conditions. |
+| **Secret Leak Detector** | Real-time input sanitizer (`lib/keycheck.js`) catching accidental pastes of private keys, SSH keys, or misplaced tokens. |
+| **Batch `.env` Import** | Parse standard `.env` key-value pairs directly into corresponding provider pools. |
+| **5-Second Undo Bar** | Non-destructive undo bar for accidental key or pool removals. |
+| **Usage Analytics Chart** | Interactive breakdown of lifetime requests and daily trends per key. |
+
+---
+
+## 🔒 Security & Safe Storage
+
+* **Zero Plaintext Secrets in Plugin Config**: Configuration files store only environment variable reference names (e.g. `MY_PROVIDER_API_KEY`).
+* **Secure Vault Storage**: Actual secret values reside securely in `$DSH_HOME/.credentials.yaml` managed by the DSH `Credentials` service.
+* **5-Character Masking (`keyTail`)**: Full secret values are never sent to the client browser; only the trailing 5 characters are exposed for visual identification.
+* **Loopback & Same-Origin Fencing**: Administrative endpoints (`GET /status`, `PUT /key`, `POST /reset`, `POST /test-matrix`) strictly enforce loopback origin checks (`isTrustedBridgeRequest`).
+
+---
+
+## 📦 Installation
+
+```bash
+# Install via DSH Plugin Manager (Web Profile):
+dsh plugin --profile web add @goodandready/dsh-key-rotation
+
+# Or directly from GitHub:
+dsh plugin --profile web add github:GooDAnDReaDY/dsh-key-rotation
+```
+
+> [!IMPORTANT]
+> Restart DeepSeek Harness web service after installation and refresh your browser tab:
+> ```bash
+> systemctl --user restart dsh-web
+> ```
+
+---
+
+## ⚙️ Configuration Reference (`settings.yaml`)
+
+```yaml
+dsh-key-rotation:
+  switchCodes:
+    - QUOTA
+    - RATE_LIMIT
+    - SERVER
+    - TIMEOUT
+    - TRANSPORT
+    - EMPTY_RESPONSE
+    - UNKNOWN_MODEL
+    - AUTH
+  cooldownMs: 60000
+  canaryProbing: true
+  concurrencyLimit: 5
+  quotaResetWindow:
+    type: midnight_utc
+    hour: 0
+  cascade:
+    - provider: backup-provider-id
+      model: your-backup-model-id
+  webhookUrl: "https://api.telegram.org/bot<TOKEN>/sendMessage?chat_id=<CHAT_ID>"
+  providers:
+    - provider: your-primary-provider
+      rpmLimit: 60
+      tpmLimit: 100000
+      keys:
+        - PRIMARY_API_KEY
+        - PRIMARY_API_KEY_2
+        - PRIMARY_API_KEY_BACKUP
+    - provider: secondary-provider
+      keys:
+        - SECONDARY_API_KEY
+        - SECONDARY_API_KEY_2
+```
+
+### Parameter Reference
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `switchCodes` | `string[]` | `[QUOTA, RATE_LIMIT, ...]` | List of error codes that immediately trigger failover. |
+| `cooldownMs` | `number` | `60000` (1 min) | Base penalty duration (in ms) for quarantined keys. |
+| `canaryProbing` | `boolean` | `true` | Runs background ping probe before restoring quarantined keys. |
+| `concurrencyLimit` | `number` | `0` (disabled) | Max concurrent in-flight streams per key (0 = unlimited). |
+| `quotaResetWindow` | `object` | `null` | Calendar reset alignment (`midnight_utc`, `midnight_pst`, `rolling_24h`). |
+| `cascade` | `array` | `[]` | Fallback provider chain when primary pool is completely exhausted. |
+| `webhookUrl` | `string` | `""` | Target URL for interactive Telegram, Discord, Slack, or generic alerts. |
+| `providers` | `array` | `[]` | List of `{ provider, keys, rpmLimit, tpmLimit, modelPools }` definitions. |
+
+---
+
+## 🔌 HTTP Bridge API Reference
+
+All management routes require loopback authentication (`127.0.0.1` / `::1`) with same-origin validation:
+
+| Route | Method | Description |
+|---|---|---|
+| `/dsh-key-rotation/status` | `GET` | Returns real-time health snapshots, active keys, and cooldown states. |
+| `/dsh-key-rotation/config` | `GET` / `PUT` | Read and update active key rotation settings and provider pools. |
+| `/dsh-key-rotation/key` | `PUT` / `DELETE` | Add, update, or remove credentials in host storage and pool. |
+| `/dsh-key-rotation/reset` | `POST` | Instantly resets all cooldowns and restores all keys to `ready`. |
+| `/dsh-key-rotation/test-matrix` | `POST` | Triggers parallel health check across all configured keys and models. |
+| `/dsh-key-rotation/usage-report` | `GET` | Returns aggregated usage metrics in JSON or CSV format (`?format=csv`). |
+| `/dsh-key-rotation/webhook-callback`| `POST` | Receives and executes interactive actions from Telegram/Slack callbacks. |
+
+---
+
+## 📄 License
+
+MIT © [GooDAnDReaDY](https://github.com/GooDAnDReaDY)