llm-proxy-cli 0.5.0__tar.gz

llm_proxy_cli-0.5.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kerem Barbaros Karnabat
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_proxy_cli-0.5.0/PKG-INFO

@@ -0,0 +1,117 @@
+Metadata-Version: 2.4
+Name: llm-proxy-cli
+Version: 0.5.0
+Summary: A lightweight CLI tool for delegating LLM tasks to expert models across multiple providers.
+Author-email: Kerem Barbaros Karnabat <kbarbaros@hotmail.com>
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: filelock>=3.12.0
+Requires-Dist: anthropic>=0.30.0
+Dynamic: license-file
+
+# LLM Proxy CLI
+
+![CI Status](https://img.shields.io/github/actions/workflow/status/cadakerem/llm-proxy-cli/ci.yml?branch=master&label=CI&logo=github)
+![PyPI Version](https://img.shields.io/pypi/v/llm-proxy-cli?color=blue&logo=pypi)
+![License](https://img.shields.io/github/license/cadakerem/llm-proxy-cli)
+
+> A lightweight, fault-tolerant CLI tool for delegating LLM tasks to expert models across multiple providers (Nvidia NIM, Groq, OpenAI, Anthropic Claude, Gemini).
+
+
+## ⚡ Features
+- **Multi-Provider Support**: Seamlessly route requests to `nvidia`, `groq`, `openai`, `anthropic`, or `gemini`.
+- **Dynamic Model Discovery**: Never hardcode a model name again. Use `auto-smart` or `auto-fast` and the router will auto-select the best model.
+- **Active Liveness Verification**: Pings candidates with a minimal chat request to drop fake/gated models before they crash your task.
+- **Automatic Fallbacks**: Provide a comma-separated list of models. If one fails, it instantly falls back to the next.
+- **Circuit Breaker**: Built-in health tracking and cooldowns to prevent spamming dead endpoints.
+- **Reasoning Extraction**: Automatically extracts and formats hidden `<thought>` or `reasoning` blocks (e.g., from Nemotron).
+- **Streaming Native**: Built on the official OpenAI SDK for fast and reliable streaming chunks.
+
+## 🏗️ Architecture & Under the Hood
+- **Language**: Python 3
+- **Libraries**: openai, filelock, anthropic
+- **Design Pattern**: Circuit Breaker, Chain of Responsibility (Fallback Routing), and Dynamic Caching.
+
+The router uses a `FileLock`-backed JSON state (`circuit_breaker.json`) to track failures across concurrent runs. 
+If an endpoint times out or returns a 5xx error more than `MAX_FAILURES` times, the circuit trips and forces the router to skip that endpoint for the next 120 seconds, immediately trying the next fallback model.
+
+
+## 📦 Installation
+
+```bash
+# Install via pip
+pip install llm-proxy-cli
+
+# Or for local development:
+# git clone https://github.com/cadakerem/llm-proxy-cli.git
+# cd llm-proxy-cli
+# pip install -e .
+```
+
+## 🔑 Configuration & API Keys
+
+The router looks for API keys in your environment variables or in ~/.config/llm-proxy-cli/keys.json.
+
+Supported environment variables:
+- NVIDIA_API_KEY
+- GROQ_API_KEY
+- OPENAI_API_KEY
+- ANTHROPIC_API_KEY
+- GEMINI_API_KEY
+
+## 💻 Usage
+
+The tool is designed to **automatically** discover and use the best model without you having to memorize model names (using `auto-smart` and `auto-fast`).
+
+### 1. Automatic Model Selection (Recommended)
+Instead of guessing which model is currently the best or active on the API, simply use `auto-smart` (for complex coding/reasoning tasks) or `auto-fast` (for quick tasks).
+
+```bash
+# Auto-select the smartest model on Nvidia (e.g., Nemotron or Llama 3.1 405B)
+llm-proxy-cli -m "nvidia:auto-smart" -p "Write a React button."
+
+# Auto-select the fastest model on Groq
+llm-proxy-cli -m "groq:auto-fast" -p "Summarize this text."
+```
+
+### 2. Chained Automatic Fallback
+If Nvidia goes down or hits a rate limit, you can instantly fall back to Groq's best model by separating them with a comma:
+
+```bash
+llm-proxy-cli -m "nvidia:auto-smart,groq:auto-smart" -p "Refactor this python script."
+```
+
+### 3. Specific / Manual Model Selection
+If you have a specific model you want to use, you can still hardcode it directly:
+
+```bash
+# Use Laguna, and fallback to a specific Groq model if it fails
+llm-proxy-cli -m "nvidia:poolside/laguna-xs-2.1,groq:groq/compound" -p "Explain quantum entanglement."
+```
+
+### ⚠️ Troubleshooting & Known Quirks: Nvidia EULA (404 Not Found)
+Nvidia NIM requires users to manually accept the **End User License Agreement (EULA)** for certain models on their website before using them via API. If you haven't accepted the EULA for a dynamically discovered model, Nvidia returns a cryptic `404 Not Found` error.
+The Smart Router intercepts this behavior automatically and will print a clear warning.
+
+**How to Fix:**
+1. Log into the [Nvidia Build Portal](https://build.nvidia.com).
+2. Search for the exact model name shown in the warning and click to run a quick test prompt to accept the terms.
+3. Or bypass auto-discovery completely by explicitly hardcoding a model:
+```bash
+llm-proxy-cli -m "nvidia:meta/llama-3.2-11b-vision-instruct" -p "Hello"
+```
+
+## 🧑‍💻 Developer & Contributions
+Developed by Kerem Barbaros Karnabat ([@cadakerem](https://github.com/cadakerem)).
+
+> **Note on Repository Structure:** The core routing logic, dynamic model discovery, and circuit breaker patterns are entirely contained within `llm_proxy_cli.py` to ensure maximum portability. Unit tests are located in the `tests/` directory, and `SKILL.md` provides instructions for integrating this tool as a native AI agent skill.
+
+Contributions, issues, and feature requests are welcome! Feel free to check the [Issues page](../../issues).
+
+## 📜 License
+This project is licensed under the [MIT License](LICENSE).

llm_proxy_cli-0.5.0/README.md

@@ -0,0 +1,101 @@
+# LLM Proxy CLI
+
+![CI Status](https://img.shields.io/github/actions/workflow/status/cadakerem/llm-proxy-cli/ci.yml?branch=master&label=CI&logo=github)
+![PyPI Version](https://img.shields.io/pypi/v/llm-proxy-cli?color=blue&logo=pypi)
+![License](https://img.shields.io/github/license/cadakerem/llm-proxy-cli)
+
+> A lightweight, fault-tolerant CLI tool for delegating LLM tasks to expert models across multiple providers (Nvidia NIM, Groq, OpenAI, Anthropic Claude, Gemini).
+
+
+## ⚡ Features
+- **Multi-Provider Support**: Seamlessly route requests to `nvidia`, `groq`, `openai`, `anthropic`, or `gemini`.
+- **Dynamic Model Discovery**: Never hardcode a model name again. Use `auto-smart` or `auto-fast` and the router will auto-select the best model.
+- **Active Liveness Verification**: Pings candidates with a minimal chat request to drop fake/gated models before they crash your task.
+- **Automatic Fallbacks**: Provide a comma-separated list of models. If one fails, it instantly falls back to the next.
+- **Circuit Breaker**: Built-in health tracking and cooldowns to prevent spamming dead endpoints.
+- **Reasoning Extraction**: Automatically extracts and formats hidden `<thought>` or `reasoning` blocks (e.g., from Nemotron).
+- **Streaming Native**: Built on the official OpenAI SDK for fast and reliable streaming chunks.
+
+## 🏗️ Architecture & Under the Hood
+- **Language**: Python 3
+- **Libraries**: openai, filelock, anthropic
+- **Design Pattern**: Circuit Breaker, Chain of Responsibility (Fallback Routing), and Dynamic Caching.
+
+The router uses a `FileLock`-backed JSON state (`circuit_breaker.json`) to track failures across concurrent runs. 
+If an endpoint times out or returns a 5xx error more than `MAX_FAILURES` times, the circuit trips and forces the router to skip that endpoint for the next 120 seconds, immediately trying the next fallback model.
+
+
+## 📦 Installation
+
+```bash
+# Install via pip
+pip install llm-proxy-cli
+
+# Or for local development:
+# git clone https://github.com/cadakerem/llm-proxy-cli.git
+# cd llm-proxy-cli
+# pip install -e .
+```
+
+## 🔑 Configuration & API Keys
+
+The router looks for API keys in your environment variables or in ~/.config/llm-proxy-cli/keys.json.
+
+Supported environment variables:
+- NVIDIA_API_KEY
+- GROQ_API_KEY
+- OPENAI_API_KEY
+- ANTHROPIC_API_KEY
+- GEMINI_API_KEY
+
+## 💻 Usage
+
+The tool is designed to **automatically** discover and use the best model without you having to memorize model names (using `auto-smart` and `auto-fast`).
+
+### 1. Automatic Model Selection (Recommended)
+Instead of guessing which model is currently the best or active on the API, simply use `auto-smart` (for complex coding/reasoning tasks) or `auto-fast` (for quick tasks).
+
+```bash
+# Auto-select the smartest model on Nvidia (e.g., Nemotron or Llama 3.1 405B)
+llm-proxy-cli -m "nvidia:auto-smart" -p "Write a React button."
+
+# Auto-select the fastest model on Groq
+llm-proxy-cli -m "groq:auto-fast" -p "Summarize this text."
+```
+
+### 2. Chained Automatic Fallback
+If Nvidia goes down or hits a rate limit, you can instantly fall back to Groq's best model by separating them with a comma:
+
+```bash
+llm-proxy-cli -m "nvidia:auto-smart,groq:auto-smart" -p "Refactor this python script."
+```
+
+### 3. Specific / Manual Model Selection
+If you have a specific model you want to use, you can still hardcode it directly:
+
+```bash
+# Use Laguna, and fallback to a specific Groq model if it fails
+llm-proxy-cli -m "nvidia:poolside/laguna-xs-2.1,groq:groq/compound" -p "Explain quantum entanglement."
+```
+
+### ⚠️ Troubleshooting & Known Quirks: Nvidia EULA (404 Not Found)
+Nvidia NIM requires users to manually accept the **End User License Agreement (EULA)** for certain models on their website before using them via API. If you haven't accepted the EULA for a dynamically discovered model, Nvidia returns a cryptic `404 Not Found` error.
+The Smart Router intercepts this behavior automatically and will print a clear warning.
+
+**How to Fix:**
+1. Log into the [Nvidia Build Portal](https://build.nvidia.com).
+2. Search for the exact model name shown in the warning and click to run a quick test prompt to accept the terms.
+3. Or bypass auto-discovery completely by explicitly hardcoding a model:
+```bash
+llm-proxy-cli -m "nvidia:meta/llama-3.2-11b-vision-instruct" -p "Hello"
+```
+
+## 🧑‍💻 Developer & Contributions
+Developed by Kerem Barbaros Karnabat ([@cadakerem](https://github.com/cadakerem)).
+
+> **Note on Repository Structure:** The core routing logic, dynamic model discovery, and circuit breaker patterns are entirely contained within `llm_proxy_cli.py` to ensure maximum portability. Unit tests are located in the `tests/` directory, and `SKILL.md` provides instructions for integrating this tool as a native AI agent skill.
+
+Contributions, issues, and feature requests are welcome! Feel free to check the [Issues page](../../issues).
+
+## 📜 License
+This project is licensed under the [MIT License](LICENSE).

llm_proxy_cli-0.5.0/llm_proxy_cli.egg-info/PKG-INFO

@@ -0,0 +1,117 @@
+Metadata-Version: 2.4
+Name: llm-proxy-cli
+Version: 0.5.0
+Summary: A lightweight CLI tool for delegating LLM tasks to expert models across multiple providers.
+Author-email: Kerem Barbaros Karnabat <kbarbaros@hotmail.com>
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: filelock>=3.12.0
+Requires-Dist: anthropic>=0.30.0
+Dynamic: license-file
+
+# LLM Proxy CLI
+
+![CI Status](https://img.shields.io/github/actions/workflow/status/cadakerem/llm-proxy-cli/ci.yml?branch=master&label=CI&logo=github)
+![PyPI Version](https://img.shields.io/pypi/v/llm-proxy-cli?color=blue&logo=pypi)
+![License](https://img.shields.io/github/license/cadakerem/llm-proxy-cli)
+
+> A lightweight, fault-tolerant CLI tool for delegating LLM tasks to expert models across multiple providers (Nvidia NIM, Groq, OpenAI, Anthropic Claude, Gemini).
+
+
+## ⚡ Features
+- **Multi-Provider Support**: Seamlessly route requests to `nvidia`, `groq`, `openai`, `anthropic`, or `gemini`.
+- **Dynamic Model Discovery**: Never hardcode a model name again. Use `auto-smart` or `auto-fast` and the router will auto-select the best model.
+- **Active Liveness Verification**: Pings candidates with a minimal chat request to drop fake/gated models before they crash your task.
+- **Automatic Fallbacks**: Provide a comma-separated list of models. If one fails, it instantly falls back to the next.
+- **Circuit Breaker**: Built-in health tracking and cooldowns to prevent spamming dead endpoints.
+- **Reasoning Extraction**: Automatically extracts and formats hidden `<thought>` or `reasoning` blocks (e.g., from Nemotron).
+- **Streaming Native**: Built on the official OpenAI SDK for fast and reliable streaming chunks.
+
+## 🏗️ Architecture & Under the Hood
+- **Language**: Python 3
+- **Libraries**: openai, filelock, anthropic
+- **Design Pattern**: Circuit Breaker, Chain of Responsibility (Fallback Routing), and Dynamic Caching.
+
+The router uses a `FileLock`-backed JSON state (`circuit_breaker.json`) to track failures across concurrent runs. 
+If an endpoint times out or returns a 5xx error more than `MAX_FAILURES` times, the circuit trips and forces the router to skip that endpoint for the next 120 seconds, immediately trying the next fallback model.
+
+
+## 📦 Installation
+
+```bash
+# Install via pip
+pip install llm-proxy-cli
+
+# Or for local development:
+# git clone https://github.com/cadakerem/llm-proxy-cli.git
+# cd llm-proxy-cli
+# pip install -e .
+```
+
+## 🔑 Configuration & API Keys
+
+The router looks for API keys in your environment variables or in ~/.config/llm-proxy-cli/keys.json.
+
+Supported environment variables:
+- NVIDIA_API_KEY
+- GROQ_API_KEY
+- OPENAI_API_KEY
+- ANTHROPIC_API_KEY
+- GEMINI_API_KEY
+
+## 💻 Usage
+
+The tool is designed to **automatically** discover and use the best model without you having to memorize model names (using `auto-smart` and `auto-fast`).
+
+### 1. Automatic Model Selection (Recommended)
+Instead of guessing which model is currently the best or active on the API, simply use `auto-smart` (for complex coding/reasoning tasks) or `auto-fast` (for quick tasks).
+
+```bash
+# Auto-select the smartest model on Nvidia (e.g., Nemotron or Llama 3.1 405B)
+llm-proxy-cli -m "nvidia:auto-smart" -p "Write a React button."
+
+# Auto-select the fastest model on Groq
+llm-proxy-cli -m "groq:auto-fast" -p "Summarize this text."
+```
+
+### 2. Chained Automatic Fallback
+If Nvidia goes down or hits a rate limit, you can instantly fall back to Groq's best model by separating them with a comma:
+
+```bash
+llm-proxy-cli -m "nvidia:auto-smart,groq:auto-smart" -p "Refactor this python script."
+```
+
+### 3. Specific / Manual Model Selection
+If you have a specific model you want to use, you can still hardcode it directly:
+
+```bash
+# Use Laguna, and fallback to a specific Groq model if it fails
+llm-proxy-cli -m "nvidia:poolside/laguna-xs-2.1,groq:groq/compound" -p "Explain quantum entanglement."
+```
+
+### ⚠️ Troubleshooting & Known Quirks: Nvidia EULA (404 Not Found)
+Nvidia NIM requires users to manually accept the **End User License Agreement (EULA)** for certain models on their website before using them via API. If you haven't accepted the EULA for a dynamically discovered model, Nvidia returns a cryptic `404 Not Found` error.
+The Smart Router intercepts this behavior automatically and will print a clear warning.
+
+**How to Fix:**
+1. Log into the [Nvidia Build Portal](https://build.nvidia.com).
+2. Search for the exact model name shown in the warning and click to run a quick test prompt to accept the terms.
+3. Or bypass auto-discovery completely by explicitly hardcoding a model:
+```bash
+llm-proxy-cli -m "nvidia:meta/llama-3.2-11b-vision-instruct" -p "Hello"
+```
+
+## 🧑‍💻 Developer & Contributions
+Developed by Kerem Barbaros Karnabat ([@cadakerem](https://github.com/cadakerem)).
+
+> **Note on Repository Structure:** The core routing logic, dynamic model discovery, and circuit breaker patterns are entirely contained within `llm_proxy_cli.py` to ensure maximum portability. Unit tests are located in the `tests/` directory, and `SKILL.md` provides instructions for integrating this tool as a native AI agent skill.
+
+Contributions, issues, and feature requests are welcome! Feel free to check the [Issues page](../../issues).
+
+## 📜 License
+This project is licensed under the [MIT License](LICENSE).

llm_proxy_cli-0.5.0/llm_proxy_cli.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE
+README.md
+llm_proxy_cli.py
+pyproject.toml
+llm_proxy_cli.egg-info/PKG-INFO
+llm_proxy_cli.egg-info/SOURCES.txt
+llm_proxy_cli.egg-info/dependency_links.txt
+llm_proxy_cli.egg-info/entry_points.txt
+llm_proxy_cli.egg-info/requires.txt
+llm_proxy_cli.egg-info/top_level.txt
+tests/test_circuit.py
+tests/test_discovery.py
+tests/test_query_ai.py

llm_proxy_cli-0.5.0/llm_proxy_cli.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

llm_proxy_cli-0.5.0/llm_proxy_cli.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+llm-proxy-cli = llm_proxy_cli:main

llm_proxy_cli-0.5.0/llm_proxy_cli.egg-info/requires.txt

@@ -0,0 +1,3 @@
+openai>=1.0.0
+filelock>=3.12.0
+anthropic>=0.30.0

llm_proxy_cli-0.5.0/llm_proxy_cli.egg-info/top_level.txt

@@ -0,0 +1 @@
+llm_proxy_cli

llm_proxy_cli-0.5.0/llm_proxy_cli.py

@@ -0,0 +1,495 @@
+import sys
+import os
+import json
+import time
+import random
+import re
+import argparse
+import logging
+from openai import OpenAI
+from filelock import FileLock, Timeout
+
+__version__ = "0.5.0"
+
+# Optional import for anthropic
+try:
+    import anthropic
+    HAS_ANTHROPIC = True
+except ImportError:
+    HAS_ANTHROPIC = False
+
+# Setup Logging
+logger = logging.getLogger("smart_router")
+handler = logging.StreamHandler(sys.stderr)
+formatter = logging.Formatter("[%(levelname)s] %(message)s")
+handler.setFormatter(formatter)
+logger.addHandler(handler)
+logger.setLevel(logging.INFO)
+
+def get_config_dir():
+    d = os.path.expanduser("~/.config/llm-proxy-cli")
+    os.makedirs(d, exist_ok=True)
+    return d
+
+def get_cache_dir():
+    d = os.path.expanduser("~/.cache/llm-proxy-cli")
+    os.makedirs(d, exist_ok=True)
+    return d
+
+# --- Auto model discovery config ---
+AUTO_CACHE_TTL = 24 * 60 * 60     # refresh discovery at most once per day
+AUTO_DISCOVERY_TIMEOUT = 5        # seconds - keep the "auto" resolve snappy
+
+
+def get_api_key(provider):
+    keys_file = os.path.join(get_config_dir(), "keys.json")
+    if os.path.exists(keys_file):
+        try:
+            with open(keys_file, 'r', encoding='utf-8') as f:
+                keys = json.load(f)
+            env_name = f"{provider.upper()}_API_KEY"
+            if keys.get(env_name):
+                return keys[env_name]
+        except Exception:
+            pass
+    return os.environ.get(f"{provider.upper()}_API_KEY")
+
+
+PROVIDERS = {
+    "nvidia": {"base_url": "https://integrate.api.nvidia.com/v1", "api_key": get_api_key("NVIDIA")},
+    "openai": {"base_url": "https://api.openai.com/v1", "api_key": get_api_key("OPENAI")},
+    "groq": {"base_url": "https://api.groq.com/openai/v1", "api_key": get_api_key("GROQ")},
+    "gemini": {"base_url": "https://generativelanguage.googleapis.com/v1beta/openai/", "api_key": get_api_key("GEMINI")},
+    "anthropic": {"api_key": get_api_key("ANTHROPIC")}
+}
+
+
+# ---------------------------------------------------------------------------
+# Dynamic Model Auto-Discovery
+#
+# Usage: pass "provider:auto-smart" or "provider:auto-fast" instead of a
+# hardcoded model name, e.g.
+#   llm-proxy-cli -m "groq:auto-smart,nvidia:auto-fast" -p "..."
+# ("auto" and "auto-max" are kept as aliases of "auto-smart" for backwards
+# compatibility with existing scripts/cron jobs.)
+#
+# The router hits the provider's /models endpoint, drops anything that isn't
+# a general-purpose chat/completions model (guardrail filters, moderation,
+# embeddings, TTS/STT, rerankers - these show up in /models but will 400 on
+# a normal chat request), scores what's left, and picks the best one.
+# Results are cached locally per (provider, mode) for AUTO_CACHE_TTL seconds
+# so normal calls never pay the discovery latency.
+# ---------------------------------------------------------------------------
+
+# Model ids containing any of these are never valid chat-completion candidates,
+# regardless of mode - excluding them up front is what "auto-fast" needs to
+# avoid latching onto a tiny 86M-parameter safety/guard classifier just
+# because it has the smallest size in its name.
+NON_CHAT_KEYWORDS = (
+    "guard", "moderation", "safety", "embed", "embedding", "rerank",
+    "whisper", "tts", "speech", "audio", "clip", "classifier",
+)
+
+
+def _is_chat_candidate(model_id):
+    name = model_id.lower()
+    return not any(kw in name for kw in NON_CHAT_KEYWORDS)
+
+
+def _score_model_smart(model_id):
+    """Higher score = bigger / newer / more capable. Used by auto-smart."""
+    name = model_id.lower()
+    # Strip dates like 2024-08-06 or context sizes like 32768
+    name = re.sub(r'20\d{2}[-]?\d{2}[-]?\d{2}', '', name)
+    name = re.sub(r'\d{4,}', '', name)
+    score = 0.0
+    
+    if "opus" in name: score += 50
+    elif "sonnet" in name: score += 40
+    elif "haiku" in name: score += 30
+
+    # Parameter size, e.g. "8b", "70b", "405b" -> biggest wins, weighted heavily.
+    size_matches = re.findall(r'(\d+(?:\.\d+)?)\s*b(?!\w)', name)
+    if size_matches:
+        score += max(float(s) for s in size_matches) * 10
+        name_wo_size = re.sub(r'\d+(?:\.\d+)?\s*b(?!\w)', '', name)
+    else:
+        name_wo_size = name
+
+    # Any remaining digits are treated as generation/version numbers,
+    # e.g. "llama-4", "gemini-2.5", "v3.3" -> higher wins.
+    version_matches = re.findall(r'(\d+(?:\.\d+)?)', name_wo_size)
+    if version_matches:
+        score += max(float(v) for v in version_matches) * 5
+
+    for kw, bonus in (("instruct", 3), ("versatile", 3), ("reasoning", 4), ("chat", 1)):
+        if kw in name:
+            score += bonus
+    for kw, penalty in (("mini", -2), ("lite", -2), ("tiny", -3), ("preview", -1), ("deprecated", -100)):
+        if kw in name:
+            score += penalty
+
+    return score
+
+
+def _score_model_fast(model_id):
+    """Higher score = smaller / snappier chat model. Used by auto-fast.
+    NON_CHAT_KEYWORDS models are filtered out before this ever runs, so
+    "smallest wins" can't land on a non-chat model anymore."""
+    name = model_id.lower()
+    name = re.sub(r'20\d{2}[-]?\d{2}[-]?\d{2}', '', name)
+    name = re.sub(r'\d{4,}', '', name)
+    score = 0.0
+
+    if "haiku" in name: score += 20
+    elif "sonnet" in name: score += 10
+
+    size_matches = re.findall(r'(\d+(?:\.\d+)?)\s*b(?!\w)', name)
+    if size_matches:
+        score -= max(float(s) for s in size_matches) * 10  # smaller size = higher score
+        name_wo_size = re.sub(r'\d+(?:\.\d+)?\s*b(?!\w)', '', name)
+    else:
+        name_wo_size = name
+
+    version_matches = re.findall(r'(\d+(?:\.\d+)?)', name_wo_size)
+    if version_matches:
+        score += max(float(v) for v in version_matches) * 5  # still prefer the newer generation
+
+    for kw, bonus in (("instant", 4), ("flash", 4), ("turbo", 4), ("mini", 3), ("lite", 3), ("small", 2), ("instruct", 1)):
+        if kw in name:
+            score += bonus
+    for kw, penalty in (("preview", -1), ("deprecated", -100)):
+        if kw in name:
+            score += penalty
+
+    return score
+
+
+_SCORERS = {"smart": _score_model_smart, "fast": _score_model_fast}
+
+
+def _auto_cache_path(provider, mode):
+    return os.path.join(get_cache_dir(), f"auto_models_cache_{provider}_{mode}.json")
+
+
+def _load_auto_cache(provider, mode):
+    path = _auto_cache_path(provider, mode)
+    if os.path.exists(path):
+        try:
+            with open(path, 'r', encoding='utf-8') as f:
+                data = json.load(f)
+            if time.time() - data.get('timestamp', 0) < AUTO_CACHE_TTL:
+                return data.get('best_model')
+        except Exception:
+            pass
+    return None
+
+
+def _save_auto_cache(provider, mode, best_model):
+    path = _auto_cache_path(provider, mode)
+    try:
+        with open(path, 'w', encoding='utf-8') as f:
+            json.dump({'timestamp': time.time(), 'best_model': best_model}, f)
+    except Exception:
+        pass
+
+
+VERIFY_TIMEOUT = 8          # seconds per live verification call
+MAX_VERIFY_CANDIDATES = 3   # how many top-scored candidates to actually try before giving up
+
+
+def _verify_chat_model(provider, provider_config, model_id):
+    """Send one minimal real chat request to confirm this model id actually serves
+    chat/completions (not TTS, embeddings, a gated model needing terms acceptance, etc.).
+    This is the real safety net - NON_CHAT_KEYWORDS is only a cheap pre-filter to cut
+    down how many of these we need to make; the keyword list will always be incomplete
+    on its own (see: 'orpheus-v1-english', a TTS model with no matching keyword)."""
+    try:
+        if provider == "anthropic":
+            client = anthropic.Anthropic(api_key=provider_config["api_key"], timeout=VERIFY_TIMEOUT)
+            client.messages.create(model=model_id, max_tokens=1, messages=[{"role": "user", "content": "hi"}])
+        else:
+            client = OpenAI(base_url=provider_config["base_url"], api_key=provider_config["api_key"], timeout=VERIFY_TIMEOUT)
+            client.chat.completions.create(model=model_id, messages=[{"role": "user", "content": "hi"}], max_tokens=1)
+        return True
+    except Exception as e:
+        error_str = str(e).lower()
+        if provider == "nvidia" and "404" in error_str and "not found for account" in error_str:
+            logger.warning(f"Auto-discovery: Nvidia model '{model_id}' requires EULA approval. Please visit https://build.nvidia.com to search and accept the terms for this model.")
+        else:
+            logger.debug(f"Auto-discovery: '{model_id}' failed live verification: {e}")
+        return False
+
+
+def discover_best_model(provider, mode="smart", force_refresh=False):
+    """Return the best available chat model id for `provider` under `mode`
+    ("smart" = biggest/most capable, "fast" = smallest/snappiest), using a
+    24h local cache per (provider, mode). Returns None on any failure so
+    callers can fall back gracefully.
+
+    Only runs on a cache miss (once per day per provider/mode), so paying
+    a few extra live-verification calls here is worth it for correctness -
+    every call after this one comes straight from cache."""
+    if mode not in _SCORERS:
+        logger.warning(f"Auto-discovery: unknown mode '{mode}', defaulting to 'smart'.")
+        mode = "smart"
+
+    if not force_refresh:
+        cached = _load_auto_cache(provider, mode)
+        if cached:
+            return cached
+
+    provider_config = PROVIDERS.get(provider)
+    if not provider_config or not provider_config.get("api_key"):
+        logger.warning(f"Auto-discovery: provider '{provider}' not configured or missing API key.")
+        return None
+
+    try:
+        if provider == "anthropic":
+            if not HAS_ANTHROPIC:
+                return None
+            client = anthropic.Anthropic(api_key=provider_config["api_key"], timeout=AUTO_DISCOVERY_TIMEOUT)
+            model_ids = [m.id for m in client.models.list().data]
+        else:
+            client = OpenAI(
+                base_url=provider_config["base_url"], api_key=provider_config["api_key"], timeout=AUTO_DISCOVERY_TIMEOUT
+            )
+            model_ids = [m.id for m in client.models.list().data]
+    except Exception as e:
+        logger.warning(f"Auto-discovery failed for '{provider}': {e}")
+        return None
+
+    # Cheap pre-filter by name, purely to reduce how many live calls we make below.
+    chat_candidates = [m for m in model_ids if _is_chat_candidate(m)]
+    if not chat_candidates:
+        logger.warning(f"Auto-discovery: no chat-capable models found for '{provider}' (got {len(model_ids)} total, all filtered out by name).")
+        return None
+
+    ranked = sorted(chat_candidates, key=_SCORERS[mode], reverse=True)
+
+    for candidate in ranked[:MAX_VERIFY_CANDIDATES]:
+        if _verify_chat_model(provider, provider_config, candidate):
+            logger.info(f"Auto-discovery ({mode}): picked '{candidate}' for '{provider}' (live-verified) out of {len(chat_candidates)}/{len(model_ids)} name-filtered candidates.")
+            _save_auto_cache(provider, mode, candidate)
+            return candidate
+        logger.warning(f"Auto-discovery ({mode}): '{candidate}' failed live verification (gated, non-chat, or otherwise unusable), trying next candidate.")
+
+    logger.warning(f"Auto-discovery ({mode}): none of the top {min(MAX_VERIFY_CANDIDATES, len(ranked))} name-filtered candidates for '{provider}' passed live verification.")
+    return None
+
+
+class CircuitBreaker:
+    def __init__(self, project_id, max_failures=2, cooldown_seconds=120):
+        self.max_failures = max_failures
+        self.cooldown_seconds = cooldown_seconds
+        safe_proj = "".join([c if c.isalnum() else "_" for c in project_id])
+        self.circuit_file = os.path.join(get_cache_dir(), f"circuit_breaker_{safe_proj}.json")
+        self.lock_file = os.path.join(get_cache_dir(), f"circuit_breaker_{safe_proj}.json.lock")
+
+    def load(self):
+        for _ in range(3):
+            if os.path.exists(self.circuit_file):
+                try:
+                    with open(self.circuit_file, 'r', encoding='utf-8') as f:
+                        return json.load(f)
+                except Exception:
+                    time.sleep(random.uniform(0.01, 0.05))
+        return {}
+
+    def save(self, data):
+        for _ in range(3):
+            try:
+                current_time = time.time()
+                active_data = {
+                    k: v for k, v in data.items()
+                    if v.get('failures', 0) > 0 or v.get('cooldown_until', 0) > current_time
+                }
+                if not active_data:
+                    if os.path.exists(self.circuit_file):
+                        os.remove(self.circuit_file)
+                    return
+                with open(self.circuit_file, 'w', encoding='utf-8') as f:
+                    json.dump(active_data, f)
+                break
+            except Exception:
+                time.sleep(random.uniform(0.01, 0.05))
+
+    def check_health(self, model_id):
+        try:
+            with FileLock(self.lock_file, timeout=5):
+                circuit = self.load()
+                if model_id in circuit:
+                    stats = circuit[model_id]
+                    if stats.get('cooldown_until', 0) > time.time():
+                        return False
+                return True
+        except Timeout:
+            logger.debug(f"Timeout acquiring lock for {model_id} health check. Assuming healthy.")
+            return True
+
+    def record_failure(self, model_id):
+        try:
+            with FileLock(self.lock_file, timeout=5):
+                circuit = self.load()
+                if model_id not in circuit:
+                    circuit[model_id] = {'failures': 0, 'cooldown_until': 0}
+                circuit[model_id]['failures'] += 1
+                if circuit[model_id]['failures'] >= self.max_failures:
+                    circuit[model_id]['cooldown_until'] = time.time() + self.cooldown_seconds
+                    circuit[model_id]['failures'] = 0
+                    logger.warning(f"CIRCUIT BREAKER: {model_id} tripped! Cooldown: {self.cooldown_seconds}s.")
+                self.save(circuit)
+        except Timeout:
+            logger.debug(f"Timeout acquiring lock. Could not record failure for {model_id}.")
+
+    def record_success(self, model_id):
+        try:
+            with FileLock(self.lock_file, timeout=5):
+                circuit = self.load()
+                if model_id in circuit and circuit[model_id]['failures'] > 0:
+                    circuit[model_id]['failures'] = 0
+                self.save(circuit)
+        except Timeout:
+            pass
+
+
+# "auto" / "auto-max" are kept as aliases of "auto-smart" for backwards compatibility.
+_AUTO_ALIASES = {"auto": "smart", "auto-max": "smart", "auto-smart": "smart", "auto-fast": "fast"}
+
+
+def parse_model(model_string, force_refresh_auto=False):
+    if ":" in model_string:
+        provider, model = model_string.split(":", 1)
+        provider, model = provider.strip().lower(), model.strip()
+    else:
+        provider, model = "nvidia", model_string.strip()
+
+    mode = _AUTO_ALIASES.get(model.lower())
+    if mode:
+        resolved = discover_best_model(provider, mode=mode, force_refresh=force_refresh_auto)
+        if resolved:
+            return provider, resolved
+        logger.error(f"Auto-discovery unavailable for '{provider}' ({mode}) and no static fallback was given.")
+        return provider, model  # will fail fast in query_ai (model not found / no key)
+
+    return provider, model
+
+
+def query_ai(models_list, prompt, cb: CircuitBreaker, max_retries=2, base_timeout=30, force_refresh_auto=False):
+    if isinstance(models_list, str):
+        models_list = [m.strip() for m in models_list.split(',')]
+
+    for current_model_str in models_list:
+        provider, current_model = parse_model(current_model_str, force_refresh_auto=force_refresh_auto)
+        # Circuit breaker tracks the *resolved* model, not the literal "auto" alias,
+        # since "auto" can point at a different real model over time.
+        resolved_key = f"{provider}:{current_model}"
+
+        if not cb.check_health(resolved_key):
+            logger.info(f"Health Check Failed: {resolved_key} is in cooldown. Skipping...")
+            continue
+
+        provider_config = PROVIDERS.get(provider)
+        if not provider_config or not provider_config.get("api_key"):
+            logger.error(f"Provider '{provider}' not configured or missing API key. Skipping...")
+            continue
+
+        is_nemotron = "nemotron" in current_model.lower()
+        model_timeout = 90 if is_nemotron else base_timeout
+
+        for attempt in range(max_retries):
+            try:
+                full_reasoning = ""
+                full_content = ""
+                if provider == "anthropic":
+                    if not HAS_ANTHROPIC:
+                        raise ImportError("Anthropic package is missing. 'pip install anthropic' required.")
+                    client = anthropic.Anthropic(api_key=provider_config["api_key"], timeout=model_timeout, max_retries=0)
+                    with client.messages.stream(
+                        model=current_model, max_tokens=4096,
+                        messages=[{"role": "user", "content": prompt}], temperature=0.7
+                    ) as stream:
+                        for text in stream.text_stream:
+                            full_content += text
+                else:
+                    client = OpenAI(
+                        base_url=provider_config["base_url"], api_key=provider_config["api_key"], timeout=model_timeout, max_retries=0
+                    )
+                    extra_body = {"chat_template_kwargs": {"enable_thinking": True}} if (provider == "nvidia" and "nemotron" in current_model.lower()) else {}
+                    completion = client.chat.completions.create(
+                        model=current_model, messages=[{"role": "user", "content": prompt}],
+                        temperature=0.7, max_tokens=4096, extra_body=extra_body if extra_body else None, stream=True
+                    )
+                    for chunk in completion:
+                        if not chunk.choices: continue
+                        reasoning = getattr(chunk.choices[0].delta, "reasoning_content", None)
+                        if reasoning: full_reasoning += reasoning
+                        content = chunk.choices[0].delta.content
+                        if content: full_content += content
+
+                cb.record_success(resolved_key)
+                output = ""
+                if full_reasoning: output += f"--- REASONING ({resolved_key}) ---\n{full_reasoning}\n--- END REASONING ---\n\n"
+                output += full_content
+                return output
+
+            except Exception as e:
+                error_msg = str(e).lower()
+                logger.error(f"Attempt {attempt+1} failed for {resolved_key}: {str(e)}")
+                cb.record_failure(resolved_key)
+                
+                status_code = getattr(e, "status_code", None)
+                if status_code in (429, 404, 401, 403) or "404" in error_msg or "not found" in error_msg or "auth" in error_msg:
+                    break
+                
+                if attempt == max_retries - 1: break
+                time.sleep((2 ** attempt) + random.uniform(0.1, 1.5))
+
+    raise RuntimeError("All fallback models failed, timed out, or are in cooldown.")
+
+
+def main():
+    if hasattr(sys.stdout, 'reconfigure'):
+        sys.stdout.reconfigure(encoding='utf-8')
+
+    parser = argparse.ArgumentParser(description="Smart Router: A fault-tolerant CLI tool for LLM delegation.")
+    parser.add_argument("-v", "--version", action="version", version=f"Smart Router v{__version__}")
+    parser.add_argument("-m", "--models", required=True, help="Comma-separated list of provider:model fallbacks (e.g. nvidia:nemotron,groq:llama3, or groq:auto-smart / groq:auto-fast).")
+    parser.add_argument("-p", "--prompt", help="The prompt text to send to the model.")
+    parser.add_argument("-f", "--file", help="Path to a text file containing the prompt.")
+    parser.add_argument("--project", default="default", help="Project ID for isolating circuit breaker state.")
+    parser.add_argument("--max-failures", type=int, default=2, help="Failures before tripping the circuit breaker.")
+    parser.add_argument("--cooldown", type=int, default=120, help="Cooldown in seconds when circuit is tripped.")
+    parser.add_argument("--refresh-models", action="store_true", help="Ignore the 24h auto-discovery cache and re-query provider model lists now.")
+    args = parser.parse_args()
+
+    prompt_text = ""
+    if args.prompt:
+        prompt_text = args.prompt
+    elif args.file:
+        try:
+            with open(args.file, "r", encoding="utf-8") as f:
+                prompt_text = f.read()
+        except Exception as e:
+            logger.error(f"Error reading file: {e}")
+            sys.exit(1)
+    elif not sys.stdin.isatty():
+        prompt_text = sys.stdin.read()
+    else:
+        parser.error("You must provide a prompt via -p, -f, or stdin (piped input).")
+
+    if not prompt_text.strip():
+        logger.error("Prompt cannot be empty.")
+        sys.exit(1)
+
+    cb = CircuitBreaker(args.project, args.max_failures, args.cooldown)
+    try:
+        response = query_ai(args.models, prompt_text, cb, force_refresh_auto=args.refresh_models)
+        print(response)
+    except Exception as e:
+        logger.error(str(e))
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

llm_proxy_cli-0.5.0/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "llm-proxy-cli"
+version = "0.5.0"
+authors = [
+  { name="Kerem Barbaros Karnabat", email="kbarbaros@hotmail.com" }
+]
+description = "A lightweight CLI tool for delegating LLM tasks to expert models across multiple providers."
+readme = "README.md"
+requires-python = ">=3.8"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+    "openai>=1.0.0",
+    "filelock>=3.12.0",
+    "anthropic>=0.30.0"
+]
+
+[project.scripts]
+llm-proxy-cli = "llm_proxy_cli:main"
+
+

llm_proxy_cli-0.5.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

llm_proxy_cli-0.5.0/tests/test_circuit.py

@@ -0,0 +1,48 @@
+import os
+import time
+import pytest
+from llm_proxy_cli import CircuitBreaker
+
+def test_circuit_breaker_isolation(tmp_path):
+    # Test that different projects don't share state
+    cb1 = CircuitBreaker("proj1", max_failures=2, cooldown_seconds=10)
+    cb2 = CircuitBreaker("proj2", max_failures=2, cooldown_seconds=10)
+    
+    cb1.circuit_file = str(tmp_path / "cb1.json")
+    cb1.lock_file = str(tmp_path / "cb1.json.lock")
+    cb2.circuit_file = str(tmp_path / "cb2.json")
+    cb2.lock_file = str(tmp_path / "cb2.json.lock")
+    
+    cb1.record_failure("modelA")
+    cb1.record_failure("modelA")
+    
+    # cb1 should be tripped
+    assert cb1.check_health("modelA") == False
+    # cb2 should be unaffected
+    assert cb2.check_health("modelA") == True
+
+def test_circuit_breaker_cooldown(tmp_path):
+    cb = CircuitBreaker("test", max_failures=1, cooldown_seconds=1)
+    cb.circuit_file = str(tmp_path / "cb.json")
+    cb.lock_file = str(tmp_path / "cb.json.lock")
+    
+    assert cb.check_health("modelB") == True
+    cb.record_failure("modelB")
+    
+    assert cb.check_health("modelB") == False
+    time.sleep(1.1)
+    # After cooldown, should be healthy again
+    assert cb.check_health("modelB") == True
+    
+def test_circuit_breaker_success_reset(tmp_path):
+    cb = CircuitBreaker("test2", max_failures=2, cooldown_seconds=10)
+    cb.circuit_file = str(tmp_path / "cb.json")
+    cb.lock_file = str(tmp_path / "cb.json.lock")
+    
+    cb.record_failure("modelC")
+    circuit = cb.load()
+    assert circuit["modelC"]["failures"] == 1
+    
+    cb.record_success("modelC")
+    circuit = cb.load()
+    assert "modelC" not in circuit or circuit["modelC"]["failures"] == 0

llm_proxy_cli-0.5.0/tests/test_discovery.py

@@ -0,0 +1,61 @@
+"""
+Unit tests for the auto-discovery scoring/filtering logic in llm_proxy_cli.py.
+These are all pure functions - no network calls, no API keys needed - so they
+run in CI the same way test_circuit.py does. The live verification call itself
+(_verify_chat_model) is intentionally NOT unit tested here since it requires a
+real API round trip; that's what the --refresh-models manual smoke test is for.
+"""
+import llm_proxy_cli as router
+
+
+def test_is_chat_candidate_filters_known_non_chat_models():
+    assert not router._is_chat_candidate("meta-llama/llama-prompt-guard-2-86m")
+    assert not router._is_chat_candidate("text-embedding-3-large")
+    assert not router._is_chat_candidate("whisper-large-v3")
+    assert not router._is_chat_candidate("llama-guard-4-12b")
+
+
+def test_is_chat_candidate_keeps_ordinary_chat_models():
+    assert router._is_chat_candidate("llama-3.3-70b-versatile")
+    assert router._is_chat_candidate("gpt-oss-120b")
+    assert router._is_chat_candidate("gemma2-9b-it")
+
+
+def test_is_chat_candidate_does_not_catch_everything():
+    # Documents the known gap this filter has: a non-chat model whose name gives
+    # no hint (e.g. Orpheus, a TTS model) will pass the name filter and must be
+    # caught by live verification instead. This is why verification exists.
+    assert router._is_chat_candidate("canopylabs/orpheus-v1-english")
+
+
+def test_score_smart_prefers_bigger_models():
+    small = router._score_model_smart("llama-3.1-8b-instant")
+    big = router._score_model_smart("llama-3.1-405b-instruct")
+    assert big > small
+
+
+def test_score_smart_prefers_newer_generation_at_similar_size():
+    older = router._score_model_smart("llama-3.1-70b-versatile")
+    newer = router._score_model_smart("llama-4-70b-versatile")
+    assert newer > older
+
+
+def test_score_fast_prefers_smaller_models():
+    small = router._score_model_smart("llama-3.1-8b-instant")
+    big = router._score_model_smart("llama-3.1-405b-instruct")
+    assert router._score_model_fast("llama-3.1-8b-instant") > router._score_model_fast("llama-3.1-405b-instruct")
+    # sanity: smart and fast should disagree on which of these two is "better"
+    assert (big > small) != (
+        router._score_model_fast("llama-3.1-405b-instruct") > router._score_model_fast("llama-3.1-8b-instant")
+    )
+
+
+def test_score_fast_rewards_speed_keywords():
+    plain = router._score_model_fast("llama-3.1-8b")
+    instant = router._score_model_fast("llama-3.1-8b-instant")
+    assert instant > plain
+
+
+def test_discover_best_model_returns_none_without_api_key(monkeypatch):
+    monkeypatch.setitem(router.PROVIDERS, "groq", {"base_url": "https://api.groq.com/openai/v1", "api_key": None})
+    assert router.discover_best_model("groq", mode="smart") is None

llm_proxy_cli-0.5.0/tests/test_query_ai.py

@@ -0,0 +1,199 @@
+"""
+Integration-style tests for query_ai(): fallback between models, retry behavior,
+and its interaction with CircuitBreaker - all with the network mocked out via
+unittest.mock, so these run in CI with no API keys and no real HTTP calls.
+
+test_circuit.py already covers CircuitBreaker's own file-lock/open-close logic
+in isolation, so here the CircuitBreaker is a MagicMock: we're testing query_ai's
+control flow (which model gets tried, when it moves on, when it retries), not
+the breaker's persistence.
+
+IMPORTANT: query_ai() constructs a fresh OpenAI(...) client on every retry
+attempt, not once per model. Mocking the OpenAI constructor with a plain
+side_effect=[client_a, client_b] list is a trap - a retried attempt silently
+consumes the *next* list entry (meant for the fallback model) and the test
+passes for the wrong reason. openai_factory() below keys the returned mock
+client by base_url instead, so it's stable across any number of retries.
+"""
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+import llm_proxy_cli as router
+
+
+def make_openai_chunk(content=None, reasoning=None):
+    """Fake streaming chunk shaped like the OpenAI SDK's ChatCompletionChunk - just
+    enough for query_ai's `chunk.choices[0].delta.content` / .reasoning_content access."""
+    delta = SimpleNamespace(content=content)
+    if reasoning is not None:
+        delta.reasoning_content = reasoning
+    return SimpleNamespace(choices=[SimpleNamespace(delta=delta)])
+
+
+def make_anthropic_stream_cm(text_chunks):
+    """Fake context manager shaped like client.messages.stream(...)."""
+    cm = MagicMock()
+    cm.__enter__.return_value = SimpleNamespace(text_stream=iter(text_chunks))
+    cm.__exit__.return_value = False
+    return cm
+
+
+def openai_factory(clients_by_base_url):
+    """Fake OpenAI(...) constructor returning a fixed mock client per base_url,
+    however many times it's called - see module docstring for why this matters."""
+    def _factory(*args, base_url=None, **kwargs):
+        return clients_by_base_url[base_url]
+    return _factory
+
+
+GROQ_URL = "https://api.groq.com/openai/v1"
+NVIDIA_URL = "https://integrate.api.nvidia.com/v1"
+
+
+@pytest.fixture(autouse=True)
+def fast_backoff(monkeypatch):
+    # Retry backoff sleeps for real seconds otherwise; skip it in tests.
+    monkeypatch.setattr(router.time, "sleep", lambda *_: None)
+
+
+@pytest.fixture
+def mock_cb():
+    cb = MagicMock(spec=router.CircuitBreaker)
+    cb.check_health.return_value = True
+    return cb
+
+
+def test_query_ai_returns_content_on_first_model_success(monkeypatch, mock_cb):
+    monkeypatch.setitem(router.PROVIDERS, "groq", {"base_url": GROQ_URL, "api_key": "test-key"})
+
+    client = MagicMock()
+    client.chat.completions.create.return_value = [
+        make_openai_chunk(content="Hello"), make_openai_chunk(content=" world"),
+    ]
+    with patch.object(router, "OpenAI", side_effect=openai_factory({GROQ_URL: client})):
+        result = router.query_ai("groq:llama-3.3-70b-versatile", "hi", mock_cb)
+
+    assert result == "Hello world"
+    mock_cb.record_success.assert_called_once_with("groq:llama-3.3-70b-versatile")
+    mock_cb.record_failure.assert_not_called()
+
+
+def test_query_ai_falls_back_to_next_model_on_failure(monkeypatch, mock_cb):
+    monkeypatch.setitem(router.PROVIDERS, "groq", {"base_url": GROQ_URL, "api_key": "key-a"})
+    monkeypatch.setitem(router.PROVIDERS, "nvidia", {"base_url": NVIDIA_URL, "api_key": "key-b"})
+
+    failing_client = MagicMock()
+    failing_client.chat.completions.create.side_effect = Exception("500 Internal Server Error")
+    working_client = MagicMock()
+    working_client.chat.completions.create.return_value = [make_openai_chunk(content="fallback worked")]
+
+    factory = openai_factory({GROQ_URL: failing_client, NVIDIA_URL: working_client})
+    with patch.object(router, "OpenAI", side_effect=factory):
+        result = router.query_ai("groq:llama-3.3-70b-versatile,nvidia:nemotron", "hi", mock_cb, max_retries=2)
+
+    assert result == "fallback worked"
+    # groq must be retried max_retries=2 times (both against the SAME failing client) before moving on
+    assert failing_client.chat.completions.create.call_count == 2
+    assert mock_cb.record_failure.call_count == 2
+    mock_cb.record_failure.assert_called_with("groq:llama-3.3-70b-versatile")
+    mock_cb.record_success.assert_called_once_with("nvidia:nemotron")
+
+
+def test_query_ai_stops_retrying_immediately_on_404(monkeypatch, mock_cb):
+    monkeypatch.setitem(router.PROVIDERS, "groq", {"base_url": GROQ_URL, "api_key": "key-a"})
+    monkeypatch.setitem(router.PROVIDERS, "nvidia", {"base_url": NVIDIA_URL, "api_key": "key-b"})
+
+    not_found_client = MagicMock()
+    not_found_client.chat.completions.create.side_effect = Exception("Error code: 404 - model not found")
+    working_client = MagicMock()
+    working_client.chat.completions.create.return_value = [make_openai_chunk(content="ok")]
+
+    factory = openai_factory({GROQ_URL: not_found_client, NVIDIA_URL: working_client})
+    with patch.object(router, "OpenAI", side_effect=factory):
+        result = router.query_ai("groq:does-not-exist,nvidia:nemotron", "hi", mock_cb, max_retries=3)
+
+    assert result == "ok"
+    # A 404 must break the retry loop after a single attempt, not consume all max_retries=3.
+    assert not_found_client.chat.completions.create.call_count == 1
+    assert mock_cb.record_failure.call_count == 1
+
+
+def test_query_ai_skips_models_in_cooldown(monkeypatch, mock_cb):
+    monkeypatch.setitem(router.PROVIDERS, "groq", {"base_url": GROQ_URL, "api_key": "key-a"})
+    monkeypatch.setitem(router.PROVIDERS, "nvidia", {"base_url": NVIDIA_URL, "api_key": "key-b"})
+    # groq is in cooldown, nvidia is healthy
+    mock_cb.check_health.side_effect = lambda model_id: model_id != "groq:llama-3.3-70b-versatile"
+
+    working_client = MagicMock()
+    working_client.chat.completions.create.return_value = [make_openai_chunk(content="from nvidia")]
+    # No entry for GROQ_URL at all - if the code wrongly tried to call groq, this KeyErrors.
+    factory = openai_factory({NVIDIA_URL: working_client})
+    with patch.object(router, "OpenAI", side_effect=factory):
+        result = router.query_ai("groq:llama-3.3-70b-versatile,nvidia:nemotron", "hi", mock_cb)
+
+    assert result == "from nvidia"
+    working_client.chat.completions.create.assert_called_once()
+
+
+def test_query_ai_skips_provider_without_api_key(monkeypatch, mock_cb):
+    monkeypatch.setitem(router.PROVIDERS, "groq", {"base_url": GROQ_URL, "api_key": None})
+    monkeypatch.setitem(router.PROVIDERS, "nvidia", {"base_url": NVIDIA_URL, "api_key": "key-b"})
+
+    working_client = MagicMock()
+    working_client.chat.completions.create.return_value = [make_openai_chunk(content="ok")]
+    # No entry for GROQ_URL - a missing-key provider must never reach the OpenAI constructor.
+    factory = openai_factory({NVIDIA_URL: working_client})
+    with patch.object(router, "OpenAI", side_effect=factory):
+        result = router.query_ai("groq:llama-3.3-70b-versatile,nvidia:nemotron", "hi", mock_cb)
+
+    assert result == "ok"
+    mock_cb.record_failure.assert_not_called()
+
+
+def test_query_ai_exits_when_every_model_fails(monkeypatch, mock_cb):
+    monkeypatch.setitem(router.PROVIDERS, "groq", {"base_url": GROQ_URL, "api_key": "key-a"})
+
+    failing_client = MagicMock()
+    failing_client.chat.completions.create.side_effect = Exception("timeout")
+    factory = openai_factory({GROQ_URL: failing_client})
+    with patch.object(router, "OpenAI", side_effect=factory):
+        with pytest.raises(SystemExit) as exc_info:
+            router.query_ai("groq:llama-3.3-70b-versatile", "hi", mock_cb, max_retries=2)
+
+    assert exc_info.value.code == 1
+    assert failing_client.chat.completions.create.call_count == 2
+
+
+def test_query_ai_uses_anthropic_client_for_anthropic_provider(monkeypatch, mock_cb):
+    if not router.HAS_ANTHROPIC:
+        pytest.skip("anthropic package not installed")
+    monkeypatch.setitem(router.PROVIDERS, "anthropic", {"api_key": "key-c"})
+
+    mock_client = MagicMock()
+    mock_client.messages.stream.return_value = make_anthropic_stream_cm(["Hi", " there"])
+    with patch.object(router.anthropic, "Anthropic", return_value=mock_client):
+        result = router.query_ai("anthropic:claude-sonnet-4-6", "hi", mock_cb)
+
+    assert result == "Hi there"
+    mock_cb.record_success.assert_called_once_with("anthropic:claude-sonnet-4-6")
+
+
+def test_query_ai_resolves_auto_alias_before_calling_provider(monkeypatch, mock_cb):
+    monkeypatch.setitem(router.PROVIDERS, "groq", {"base_url": GROQ_URL, "api_key": "key-a"})
+    monkeypatch.setattr(router, "discover_best_model", lambda provider, mode="smart", force_refresh=False: "resolved-model-xyz")
+
+    client = MagicMock()
+    client.chat.completions.create.return_value = [make_openai_chunk(content="ok")]
+    with patch.object(router, "OpenAI", side_effect=openai_factory({GROQ_URL: client})):
+        result = router.query_ai("groq:auto-smart", "hi", mock_cb)
+
+    assert result == "ok"
+    client.chat.completions.create.assert_called_once_with(
+        model="resolved-model-xyz", messages=[{"role": "user", "content": "hi"}],
+        temperature=0.7, max_tokens=4096, extra_body=None, stream=True,
+    )
+    # circuit breaker must key on the resolved model, never on the literal "auto-smart" alias
+    mock_cb.check_health.assert_called_with("groq:resolved-model-xyz")
+    mock_cb.record_success.assert_called_once_with("groq:resolved-model-xyz")