better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/features/fallback-providers.md

@@ -0,0 +1,444 @@
+---
+title: Fallback Providers
+description: Configure automatic failover to backup LLM providers when your primary model is unavailable.
+sidebar_label: Fallback Providers
+sidebar_position: 8
+---
+
+# Fallback Providers
+
+Hermes Agent has three layers of resilience that keep your sessions running when providers hit issues:
+
+1. **[Credential pools](./credential-pools.md)** — rotate across multiple API keys for the *same* provider (tried first)
+2. **Primary model fallback** — automatically switches to a *different* provider:model when your main model fails
+3. **Auxiliary task fallback** — independent provider resolution for side tasks like vision and compression
+
+Credential pools handle same-provider rotation (e.g., multiple OpenRouter keys). This page covers cross-provider fallback. Both are optional and work independently.
+
+## Primary Model Fallback
+
+When your main LLM provider encounters errors — rate limits, server overload, auth failures, connection drops — Hermes can automatically switch to a backup provider:model pair mid-session without losing your conversation.
+
+### Configuration
+
+The easiest path is the interactive manager:
+
+```bash
+hermes fallback
+```
+
+`hermes fallback` reuses the provider picker from `hermes model` — same provider list, same credential prompts, same validation. Use the subcommands `add`, `list` (alias `ls`), `remove` (alias `rm`), and `clear` to manage the chain. Changes persist under the top-level `fallback_providers:` list in `config.yaml`.
+
+If you'd rather edit the YAML directly, add a top-level `fallback_providers` list to `~/.hermes/config.yaml`:
+
+```yaml
+fallback_providers:
+  - provider: openrouter
+    model: anthropic/claude-sonnet-4
+```
+
+Each entry requires both `provider` and `model`. Entries missing either field are ignored.
+
+Gemini fallback entries accept `gemini`, `google`, `google-gemini`, and
+`google-ai-studio`. On Google's native API endpoint, all use the native Gemini
+client, including its `generationConfig.thinkingConfig` translation. A custom
+OpenAI-compatible base URL continues to use the compatible client instead.
+
+:::note `fallback_model` vs `fallback_providers`
+`fallback_providers` (plural, list) is the current config shape and supports multiple fallbacks tried in order. `fallback_model` (singular) is the legacy single-fallback key — Hermes still honors it for back-compat, but `hermes fallback` writes the current `fallback_providers` key and migrates legacy config on write. When both are set, `fallback_providers` takes priority.
+:::
+
+### Supported Providers
+
+| Provider | Value | Requirements |
+|----------|-------|-------------|
+| AI Gateway | `ai-gateway` | `AI_GATEWAY_API_KEY` |
+| OpenRouter | `openrouter` | `OPENROUTER_API_KEY` |
+| Nous Portal | `nous` | `hermes setup --portal` (fresh) or `hermes auth add nous` (OAuth) |
+| OpenAI Codex | `openai-codex` | `hermes model` → **ChatGPT or Codex Subscription** (ChatGPT OAuth) |
+| GitHub Copilot | `copilot` | `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` |
+| GitHub Copilot ACP | `copilot-acp` | External process (editor integration) |
+| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` or Claude Code credentials |
+| z.ai / GLM | `zai` | `GLM_API_KEY` |
+| Kimi / Moonshot | `kimi-coding` | `KIMI_API_KEY` |
+| MiniMax | `minimax` | `MINIMAX_API_KEY` |
+| MiniMax (China) | `minimax-cn` | `MINIMAX_CN_API_KEY` |
+| DeepSeek | `deepseek` | `DEEPSEEK_API_KEY` |
+| NVIDIA NIM | `nvidia` | `NVIDIA_API_KEY` (optional: `NVIDIA_BASE_URL`) |
+| GMI Cloud | `gmi` | `GMI_API_KEY` (optional: `GMI_BASE_URL`) |
+| Upstage Solar | `upstage` (alias `solar`) | `UPSTAGE_API_KEY` (optional: `UPSTAGE_BASE_URL`) |
+| StepFun | `stepfun` | `STEPFUN_API_KEY` (optional: `STEPFUN_BASE_URL`) |
+| Ollama Cloud | `ollama-cloud` | `OLLAMA_API_KEY` |
+| Google AI Studio | `gemini` | `GOOGLE_API_KEY` (alias: `GEMINI_API_KEY`) |
+| xAI (Grok) | `xai` (alias `grok`) | `XAI_API_KEY` (optional: `XAI_BASE_URL`) |
+| xAI Grok OAuth (SuperGrok) | `xai-oauth` (alias `grok-oauth`) | `hermes model` → xAI Grok OAuth (browser login; SuperGrok subscription) |
+| AWS Bedrock | `bedrock` | Standard boto3 auth (`AWS_REGION` + `AWS_PROFILE` or `AWS_ACCESS_KEY_ID`) |
+| Qwen Portal (OAuth) | `qwen-oauth` | `hermes model` (Qwen Portal OAuth; optional: `HERMES_QWEN_BASE_URL`) |
+| MiniMax (OAuth) | `minimax-oauth` | `hermes model` (MiniMax portal OAuth) |
+| OpenCode Zen | `opencode-zen` | `OPENCODE_ZEN_API_KEY` |
+| CommandCode | `commandcode` (alias `commandcode-chat`; Claude via `commandcode-anthropic`) | `COMMANDCODE_API_KEY` |
+| OpenCode Go | `opencode-go` | `OPENCODE_GO_API_KEY` |
+| OpenCode Free | `opencode-free` | — (keyless, no credential) |
+| Kilo Code | `kilocode` | `KILOCODE_API_KEY` |
+| Ramp Router | `router` | `RAMP_ROUTER_API_KEY` |
+| Xiaomi MiMo | `xiaomi` | `XIAOMI_API_KEY` |
+| Arcee AI | `arcee` | `ARCEEAI_API_KEY` |
+| GMI Cloud | `gmi` | `GMI_API_KEY` |
+| Nebius Token Factory | `nebius-token-factory` | `NEBIUS_API_KEY` |
+| Alibaba / DashScope | `alibaba` | `DASHSCOPE_API_KEY` |
+| Alibaba Coding Plan | `alibaba-coding-plan` | `ALIBABA_CODING_PLAN_API_KEY` (falls back to `DASHSCOPE_API_KEY`) |
+| Kimi / Moonshot (China) | `kimi-coding-cn` | `KIMI_CN_API_KEY` |
+| StepFun | `stepfun` | `STEPFUN_API_KEY` |
+| Tencent TokenHub | `tencent-tokenhub` | `TOKENHUB_API_KEY` |
+| Tencent TokenPlan | `tencent-tokenplan` | `TOKENPLAN_API_KEY` |
+| Microsoft Foundry | `azure-foundry` | `AZURE_FOUNDRY_API_KEY` + `AZURE_FOUNDRY_BASE_URL` |
+| LM Studio (local) | `lmstudio` | `LM_API_KEY` (or none for local) + `LM_BASE_URL` |
+| Hugging Face | `huggingface` | `HF_TOKEN` |
+| Custom endpoint | `custom` | `base_url` + `key_env` (see below) |
+
+### Custom Endpoint Fallback
+
+For a custom OpenAI-compatible endpoint, add `base_url` and optionally `key_env`:
+
+```yaml
+fallback_providers:
+  - provider: custom
+    model: my-local-model
+    base_url: http://localhost:8000/v1
+    key_env: MY_LOCAL_KEY            # env var name containing the API key
+```
+
+### When Fallback Triggers
+
+The fallback activates automatically when the primary model fails with:
+
+- **Rate limits** (HTTP 429) — after exhausting retry attempts
+- **Server errors** (HTTP 500, 502, 503) — after exhausting retry attempts
+- **Auth failures** (HTTP 401, 403) — immediately (no point retrying)
+- **Not found** (HTTP 404) — immediately
+- **Invalid responses** — when the API returns malformed or empty responses repeatedly. A streamed refusal (the model declining with an explanation on the refusal channel) is a terminal `content_filter` result, not an empty response, so it is surfaced rather than retried.
+
+When triggered, Hermes:
+
+1. Resolves credentials for the fallback provider (including named custom providers using `key_cmd`)
+2. Builds a new API client, preserving a dynamic credential source across timeout and request-client rebuilds
+3. Swaps the model, provider, and client in-place
+4. Resets the retry counter and continues the conversation
+
+The switch is seamless — your conversation history, tool calls, and context are preserved. The agent continues from exactly where it left off, just using a different model.
+
+:::warning Fallback resets the prompt cache
+Prompt caches are keyed to the model (and on most providers, the account) serving the request. When fallback fires, the new provider:model has no cached prefix for your conversation, so the next request re-reads the entire history at full input-token price instead of the ~75–90% discounted cached rate. The same applies when the turn ends and the primary is restored — that first request back on the primary is a full re-read too (unless the primary's cache TTL hasn't expired). This is unavoidable — it's the cost of staying alive through an outage — but it's why a long session that bounces between providers can cost noticeably more than one that stays put.
+:::
+
+:::info Per-Turn, Not Per-Session
+Fallback is **turn-scoped**: each new user message starts with the primary model restored. If the primary fails mid-turn, fallback activates for that turn only. On the next message, Hermes tries the primary again. Within a single turn, fallback activates at most once — if the fallback also fails, normal error handling takes over (retries, then error message). This prevents cascading failover loops within a turn while giving the primary model a fresh chance every turn.
+
+The per-turn retry is **reset-aware**: when the primary's credentials report a rate-limit reset time that hasn't elapsed yet (subscription windows like Claude Pro/Max's 5-hour blocks or Codex weekly limits report these as hours or days), Hermes skips the doomed retry and stays on the fallback until the reset passes — avoiding two pointless provider switches (and two prompt-cache invalidations) per turn. Expiry makes the primary eligible for a later retry; it does not schedule a retry or guarantee recovery. Transient 429s without a reset time use an exponential cooldown.
+
+When a switch arms that cooldown, the fallback notice includes its approximate remaining duration, for example: `Primary retry eligible in ~60 s; recovery is not guaranteed.` Non-rate-limit switches and switches from an already-active cross-provider fallback do not announce a new primary cooldown.
+:::
+
+### Examples
+
+**OpenRouter as fallback for Anthropic native:**
+```yaml
+model:
+  provider: anthropic
+  default: claude-sonnet-4-6
+
+fallback_providers:
+  - provider: openrouter
+    model: anthropic/claude-sonnet-4
+```
+
+**Nous Portal as fallback for OpenRouter:**
+```yaml
+model:
+  provider: openrouter
+  default: anthropic/claude-opus-4
+
+fallback_providers:
+  - provider: nous
+    model: nous-hermes-3
+```
+
+**Local model as fallback for cloud:**
+```yaml
+fallback_providers:
+  - provider: custom
+    model: llama-3.1-70b
+    base_url: http://localhost:8000/v1
+    key_env: LOCAL_API_KEY
+```
+
+**Codex OAuth as fallback:**
+```yaml
+fallback_providers:
+  - provider: openai-codex
+    model: gpt-5.3-codex
+```
+
+### Where Fallback Works
+
+| Context | Fallback Supported |
+|---------|-------------------|
+| CLI sessions | ✔ |
+| Messaging gateway (Telegram, Discord, etc.) | ✔ |
+| Subagent delegation | ✔ (`delegation.fallback_providers` when set; otherwise only unpinned children inherit the parent chain; `[]` disables) |
+| Cron jobs | ✔ (cron agents inherit configured fallback providers) |
+| Auxiliary tasks on `provider: auto` | ✔ (try per-task fallback, then the main fallback chain before built-in aux discovery) |
+
+:::tip
+There are no environment variables for the primary fallback chain — configure it exclusively through `config.yaml` or `hermes fallback`. This is intentional: fallback configuration is a deliberate choice, not something a stale shell export should override.
+:::
+
+---
+
+## Auxiliary Task Fallback
+
+Hermes uses separate lightweight models for side tasks. Each task has its own provider resolution chain that acts as a built-in fallback system.
+
+### Tasks with Independent Provider Resolution
+
+| Task | What It Does | Config Key |
+|------|-------------|-----------|
+| Vision | Image analysis, browser screenshots | `auxiliary.vision` |
+| Compression | Context compression summaries | `auxiliary.compression` |
+| Skills Hub | Skill search and discovery | `auxiliary.skills_hub` |
+| MCP | MCP helper operations | `auxiliary.mcp` |
+| Approval | Smart command-approval classification | `auxiliary.approval` |
+| Title Generation | Session title summaries | `auxiliary.title_generation` |
+| Review | `/review` reviewer subagent (full agent, not a single LLM call) | `auxiliary.review` |
+| Triage Specifier | `hermes kanban specify` / dashboard ✨ button — fleshes out a one-liner triage task into a real spec | `auxiliary.triage_specifier` |
+
+### Auto-Detection Chain
+
+When a task's provider is set to `"auto"` (the default), Hermes first tries the main provider + main model for that auxiliary task. If that route is unavailable or later fails with a capacity-style error, Hermes follows your configured fallback policy and then stops:
+
+```text
+Main provider + main model → auxiliary.<task>.fallback_chain →
+fallback_providers / fallback_model → skip the task (warn)
+```
+
+A billing or quota failure quarantines only the failed custom endpoint for the auxiliary health cooldown, not every route registered as `custom`. A healthy local endpoint with a different base URL remains eligible for fallback and subsequent auto routing. Aliases for the same custom endpoint share its health state. Built-in providers retain their shared-account health checks.
+
+The task-specific chain is most precise and wins when present. The top-level `fallback_providers` chain is the same policy the main agent uses, so free-only or same-provider fallback rules apply to auxiliary tasks on `auto` as well.
+
+**Built-in text discovery chain (compression, web extract, title generation, etc.):**
+
+```text
+OpenRouter → Nous Portal → Custom endpoint → Codex OAuth →
+API-key providers (z.ai, Kimi, MiniMax, Xiaomi MiMo, Hugging Face, Anthropic) → give up
+```
+
+**Built-in vision discovery chain:**
+
+```text
+Main provider (if vision-capable) → OpenRouter → Nous Portal →
+Codex OAuth → Anthropic → Custom endpoint → give up
+```
+
+Those built-in chains run **only when no main provider is selected** (`model.provider: auto` or unset). Once you have picked a main provider, an unavailable main route with no `fallback_chain` / `fallback_providers` skips the auxiliary task with a warning instead of guessing another provider you happen to be logged into — an expired xAI or Codex session must never bill your Nous Portal or OpenRouter balance behind your back. Declare a fallback if you want one.
+
+### Configuring Auxiliary Providers
+
+Each task can be configured independently in `config.yaml`:
+
+```yaml
+auxiliary:
+  vision:
+    provider: "auto"              # auto | openrouter | nous | codex | main | anthropic
+    model: ""                     # e.g. "openai/gpt-4o"
+    base_url: ""                  # direct endpoint (takes precedence over provider)
+    api_key: ""                   # API key for base_url
+
+  compression:
+    provider: "auto"
+    model: ""
+    fallback_chain:              # optional, task-specific fallback policy
+      - provider: openrouter
+        model: inclusionai/ring-2.6-1t:free
+
+  skills_hub:
+    provider: "auto"
+    model: ""
+
+  mcp:
+    provider: "auto"
+    model: ""
+```
+
+Every task above follows the same **provider / model / base_url** pattern. Each task can also declare its own `fallback_chain`; if omitted, `provider: auto` uses the top-level `fallback_providers` chain (the built-in discovery chain applies only when no main provider is selected).
+
+Context compression is configured under `auxiliary.compression`:
+
+```yaml
+auxiliary:
+  compression:
+    provider: main                                    # Same provider options as other auxiliary tasks
+    model: google/gemini-3-flash-preview
+    base_url: null                                    # Custom OpenAI-compatible endpoint
+```
+
+And the primary fallback chain uses:
+
+```yaml
+fallback_providers:
+  - provider: openrouter
+    model: anthropic/claude-sonnet-4
+    # base_url: http://localhost:8000/v1             # Optional custom endpoint
+```
+
+All three — auxiliary, compression, fallback — work the same way: set `provider` to pick who handles the request, `model` to pick which model, and `base_url` to point at a custom endpoint (overrides provider).
+
+### Provider Options for Auxiliary Tasks
+
+These options apply to `auxiliary:`, `compression:`, and `fallback_providers:` entries only — `"main"` is **not** a valid value for your top-level `model.provider`. For custom endpoints, use `provider: custom` in your `model:` section (see [AI Providers](/integrations/providers)).
+
+| Provider | Description | Requirements |
+|----------|-------------|-------------|
+| `"auto"` | Try providers in order until one works (default) | At least one provider configured |
+| `"openrouter"` | Force OpenRouter | `OPENROUTER_API_KEY` |
+| `"nous"` | Force Nous Portal | `hermes auth` |
+| `"codex"` | Force Codex OAuth | `hermes model` → ChatGPT or Codex Subscription |
+| `"main"` | Use whatever provider the main agent uses (auxiliary tasks only) | Active main provider configured |
+| `"anthropic"` | Force Anthropic native | `ANTHROPIC_API_KEY` or Claude Code credentials |
+
+### Direct Endpoint Override
+
+For any auxiliary task, setting `base_url` bypasses provider resolution entirely and sends requests directly to that endpoint:
+
+```yaml
+auxiliary:
+  vision:
+    base_url: "http://localhost:1234/v1"
+    api_key: "local-key"
+    model: "qwen2.5-vl"
+```
+
+`base_url` takes precedence over `provider`. Hermes uses the configured `api_key` for authentication, falling back to `OPENAI_API_KEY` if not set. It does **not** reuse `OPENROUTER_API_KEY` for custom endpoints.
+
+---
+
+## Auxiliary Capacity-Error Fallback
+
+When you set an explicit auxiliary provider (e.g. `auxiliary.vision.provider: glm`), Hermes treats that as your preferred choice — but if the provider literally cannot serve the request because of a **capacity error** (HTTP 402 payment required, HTTP 429 daily-quota exhaustion, connection failure), Hermes falls back through a layered chain instead of failing silently:
+
+1. **Primary aux provider** — the one you configured (tried first, always)
+2. **`auxiliary.<task>.fallback_chain`** — your per-task override list, if you wrote one
+3. **Main agent provider + model** — last-resort safety net (always tried, even if you didn't write a chain)
+4. **Warn + re-raise** — if every layer fails, Hermes logs `Auxiliary <task>: ... all fallbacks exhausted` at WARNING level and re-raises the original error
+
+Transient HTTP 429 rate limits (`Retry-After: ...`) are treated as request constraints, not capacity problems — they respect your explicit provider choice and do **not** trigger the fallback ladder. Only daily/monthly quota exhaustion, payment errors, and connection failures bypass the explicit-provider gate.
+
+For users on `provider: auto` (no explicit aux provider), the existing auto-detection chain runs in place of steps 2–3. Its first step is already the main agent model, so `auto` users get the same outcome with zero config.
+
+### Optional: per-task fallback chain
+
+If you want a different fallback ordering than "main agent model first", configure `fallback_chain` explicitly. Each entry needs at least `provider`; `model`, `base_url`, and `api_key` are optional.
+
+```yaml
+auxiliary:
+  vision:
+    provider: glm
+    model: glm-4v-flash
+    fallback_chain:
+      - provider: openrouter
+        model: google/gemini-3-flash-preview
+      - provider: nous
+        model: anthropic/claude-sonnet-4
+
+  compression:
+    provider: openrouter
+    fallback_chain:
+      - provider: openai
+        model: gpt-4o-mini
+        timeout: 240            # optional — this candidate's own deadline (seconds)
+```
+
+You do **not** need to configure `fallback_chain` to get fallback — the main-agent safety net runs regardless. Use it only when you specifically want a different order than the default.
+
+Each `fallback_chain` entry may also declare its own `timeout` (seconds). Without it, a fallback candidate inherits the task-level timeout — which may be tuned for the primary provider. Declaring a per-entry `timeout` lets a slower-but-reliable fallback (e.g. a large-context summarizer) get the budget it actually needs instead of dying on the primary's clock.
+
+### Provider quota errors that trigger fallback
+
+Hermes recognizes these as capacity-equivalent to 402 credit exhaustion (not transient rate limits):
+
+- Bedrock / LiteLLM: `Too many tokens per day`, `daily limit`, `tokens per day`
+- Vertex AI / GCP: `quota exceeded`, `resource exhausted`, `RESOURCE_EXHAUSTED`
+- Generic: `daily quota`, `quota_exceeded`
+
+If your provider returns a different phrase for daily-quota exhaustion and Hermes doesn't trigger fallback, that's a bug — open an issue with the exact error string.
+
+---
+
+## Context Compression Fallback
+
+Context compression uses the `auxiliary.compression` config block to control which model and provider handles summarization:
+
+```yaml
+auxiliary:
+  compression:
+    provider: "auto"                              # auto | openrouter | nous | main
+    model: "google/gemini-3-flash-preview"
+```
+
+:::info Legacy migration
+Older configs with `compression.summary_model` / `compression.summary_provider` / `compression.summary_base_url` are automatically migrated to `auxiliary.compression.*` on first load (config version 17).
+:::
+
+If no provider is available for compression, Hermes drops middle conversation turns without generating a summary rather than failing the session.
+
+---
+
+## Delegation Provider Override
+
+Subagents spawned by `delegate_task` inherit the parent agent's primary fallback chain. You can still route subagents to a different primary provider:model pair for cost optimization:
+
+```yaml
+delegation:
+  provider: "openrouter"                      # override provider for all subagents
+  model: "google/gemini-3-flash-preview"      # override model
+  # base_url: "http://localhost:1234/v1"      # or use a direct endpoint
+  # api_key: "local-key"
+```
+
+See [Subagent Delegation](/user-guide/features/delegation) for full configuration details.
+
+---
+
+## Cron Job Providers
+
+Cron jobs inherit your configured `fallback_providers` chain (or legacy `fallback_model`) when they create an agent. To use a different primary provider for a cron job, configure `provider` and `model` overrides on the cron job itself:
+
+```python
+cronjob(
+    action="create",
+    schedule="every 2h",
+    prompt="Check server status",
+    provider="openrouter",
+    model="google/gemini-3-flash-preview"
+)
+```
+
+See [Scheduled Tasks (Cron)](/user-guide/features/cron) for full configuration details.
+
+---
+
+## Summary
+
+| Feature | Fallback Mechanism | Config Location |
+|---------|-------------------|----------------|
+| Main agent model | `fallback_providers` in config.yaml — per-turn failover on errors (primary restored each turn) | `fallback_providers:` (top-level list) |
+| Auxiliary tasks (any) — auto users | Full auto-detection chain (main agent model first, then provider chain) on capacity errors | `auxiliary.<task>.provider: auto` |
+| Auxiliary tasks (any) — explicit provider | `fallback_chain` (if set) → main agent model → warn + raise, on capacity errors only | `auxiliary.<task>.fallback_chain` |
+| Vision | Layered (see above) + internal OpenRouter retry | `auxiliary.vision` |
+| Context compression | Layered (see above); degrades to no-summary if all layers unavailable | `auxiliary.compression` |
+| Skills hub | Layered (see above) | `auxiliary.skills_hub` |
+| MCP helpers | Layered (see above) | `auxiliary.mcp` |
+| Approval classification | Layered (see above) | `auxiliary.approval` |
+| Title generation | Layered (see above) | `auxiliary.title_generation` |
+| Triage specifier | Layered (see above) | `auxiliary.triage_specifier` |
+| Delegation | Uses `delegation.fallback_providers` when declared; otherwise only unpinned children inherit the parent chain | `delegation.provider` / `delegation.model` / `delegation.fallback_providers` |
+| Cron jobs | Inherit the configured `fallback_providers` chain; optional per-job provider override | Per-job `provider` / `model` |