llmcycle 0.1.0__tar.gz

llmcycle-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+.env

llmcycle-0.1.0/.python-version

@@ -0,0 +1 @@
+3.11

llmcycle-0.1.0/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.4
+Name: llmcycle
+Version: 0.1.0
+Summary: Cycle through multiple LLM providers with smart fallback, load balancing, and unified API
+Requires-Python: >=3.11
+Requires-Dist: fastapi>=0.136.1
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: jinja2>=3.1.6
+Requires-Dist: pydantic>=2.13.4
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: python-multipart>=0.0.29
+Requires-Dist: uvicorn>=0.47.0
+Description-Content-Type: text/markdown
+
+# LLMCycle ♻️
+
+An enterprise-grade, highly resilient LLM management and routing framework. Designed to be **better than LiteLLM** with advanced multi-key support, customized routing (sort order), robust mid-stream error failovers, and a premium **Web Dashboard**.
+
+## 🚀 Key Features
+
+*   **🔑 Universal Provider Support**: Supports *any* provider on the market instantly. Just add `<PROVIDER_NAME>_API_KEYS` to your `.env`!
+*   **⚖️ Auto Load-Balancing**: Load multiple API keys for the same provider simply by comma-separating them in your `.env`. LLMCycle automatically round-robins across them and tracks rate limits locally.
+*   **🛣️ Custom Fallback Routing**: Configure custom routing. If a primary provider fails, it automatically falls back to your configured secondary.
+*   **🛡️ Streaming Time Resilience**: If an LLM disconnects *while streaming a response*, LLMCycle captures the generated text, silently switches to your fallback model, and resumes the stream seamlessly. The client never notices!
+*   **🖥️ Premium Web Dashboard**: Manage and view your keys, active providers, and fallback routes through a beautifully designed, secure UI.
+
+## 📦 Installation
+
+```bash
+uv add llmcycle
+uv add python-dotenv httpx fastapi uvicorn jinja2 python-multipart
+```
+
+## ⚙️ Configuration (`.env`)
+
+Drop your keys into a `.env` file. To use multiple keys for load balancing, just separate them with commas!
+
+```env
+DEEPSEEK_API_KEYS=sk-deepseek-1,sk-deepseek-2
+OPENAI_API_KEYS=sk-openai-primary
+TOGETHER_API_KEYS=sk-together-1
+
+# You can even use completely custom providers!
+# LLMCycle will default the base URL to https://api.mycustomai.com/v1
+MYCUSTOMAI_API_KEYS=sk-custom
+# Or explicitly define the base URL for custom providers
+OLLAMA_API_KEYS=local
+OLLAMA_BASE_URL=http://localhost:11434/v1
+
+# UI Dashboard Auth
+LLMCYCLE_USER_ADMIN=admin
+LLMCYCLE_USER_ADMIN_PAASWORD=admin
+```
+
+## 🖥️ Starting the Web Dashboard
+
+We built a gorgeous, premium Glassmorphism dashboard to monitor your providers.
+
+```bash
+# Make sure your PYTHONPATH is set if running from source:
+# Windows: $env:PYTHONPATH="src"
+# Linux/Mac: export PYTHONPATH="src"
+
+uv run llmcycle ui
+```
+*Navigate to `http://127.0.0.1:8000` and login with the credentials defined in your `.env`!*
+
+## 💻 Usage: Everything in One!
+
+```python
+import asyncio
+from llmcycle import LLMCycle
+
+async def main():
+    # 1. Initialization (Auto-loads all providers & keys from .env)
+    client = LLMCycle(
+        env_path=".env",
+        custom_fallbacks={
+            "deepseek": ["openai", "together"] # Sort order / Fallback chain
+        }
+    )
+    
+    # 2. List all dynamically loaded providers
+    providers = client.get_available_providers()
+    print("Loaded Providers:", providers) 
+    
+    # 3. Query models supported by a provider
+    models = await client.get_provider_models("deepseek")
+    print("DeepSeek Models:", models)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## 🔌 Massive Universal Provider Registry
+
+LLMCycle is pre-configured with base URLs for the most popular platforms:
+`OPENAI`, `DEEPSEEK`, `ANTHROPIC`, `TOGETHER`, `GROQ`, `MISTRAL`, `PERPLEXITY`, `ANYSCALE`, `FIREWORKS`, `COHERE`, `DATABRICKS`, `HUGGINGFACE`.
+
+**Wildcard Support:** If you type `RANDOM_API_KEYS`, LLMCycle will automatically assume `https://api.random.com/v1`. If that's wrong, just define `RANDOM_BASE_URL` in your `.env`!

llmcycle-0.1.0/README.md

@@ -0,0 +1,86 @@
+# LLMCycle ♻️
+
+An enterprise-grade, highly resilient LLM management and routing framework. Designed to be **better than LiteLLM** with advanced multi-key support, customized routing (sort order), robust mid-stream error failovers, and a premium **Web Dashboard**.
+
+## 🚀 Key Features
+
+*   **🔑 Universal Provider Support**: Supports *any* provider on the market instantly. Just add `<PROVIDER_NAME>_API_KEYS` to your `.env`!
+*   **⚖️ Auto Load-Balancing**: Load multiple API keys for the same provider simply by comma-separating them in your `.env`. LLMCycle automatically round-robins across them and tracks rate limits locally.
+*   **🛣️ Custom Fallback Routing**: Configure custom routing. If a primary provider fails, it automatically falls back to your configured secondary.
+*   **🛡️ Streaming Time Resilience**: If an LLM disconnects *while streaming a response*, LLMCycle captures the generated text, silently switches to your fallback model, and resumes the stream seamlessly. The client never notices!
+*   **🖥️ Premium Web Dashboard**: Manage and view your keys, active providers, and fallback routes through a beautifully designed, secure UI.
+
+## 📦 Installation
+
+```bash
+uv add llmcycle
+uv add python-dotenv httpx fastapi uvicorn jinja2 python-multipart
+```
+
+## ⚙️ Configuration (`.env`)
+
+Drop your keys into a `.env` file. To use multiple keys for load balancing, just separate them with commas!
+
+```env
+DEEPSEEK_API_KEYS=sk-deepseek-1,sk-deepseek-2
+OPENAI_API_KEYS=sk-openai-primary
+TOGETHER_API_KEYS=sk-together-1
+
+# You can even use completely custom providers!
+# LLMCycle will default the base URL to https://api.mycustomai.com/v1
+MYCUSTOMAI_API_KEYS=sk-custom
+# Or explicitly define the base URL for custom providers
+OLLAMA_API_KEYS=local
+OLLAMA_BASE_URL=http://localhost:11434/v1
+
+# UI Dashboard Auth
+LLMCYCLE_USER_ADMIN=admin
+LLMCYCLE_USER_ADMIN_PAASWORD=admin
+```
+
+## 🖥️ Starting the Web Dashboard
+
+We built a gorgeous, premium Glassmorphism dashboard to monitor your providers.
+
+```bash
+# Make sure your PYTHONPATH is set if running from source:
+# Windows: $env:PYTHONPATH="src"
+# Linux/Mac: export PYTHONPATH="src"
+
+uv run llmcycle ui
+```
+*Navigate to `http://127.0.0.1:8000` and login with the credentials defined in your `.env`!*
+
+## 💻 Usage: Everything in One!
+
+```python
+import asyncio
+from llmcycle import LLMCycle
+
+async def main():
+    # 1. Initialization (Auto-loads all providers & keys from .env)
+    client = LLMCycle(
+        env_path=".env",
+        custom_fallbacks={
+            "deepseek": ["openai", "together"] # Sort order / Fallback chain
+        }
+    )
+    
+    # 2. List all dynamically loaded providers
+    providers = client.get_available_providers()
+    print("Loaded Providers:", providers) 
+    
+    # 3. Query models supported by a provider
+    models = await client.get_provider_models("deepseek")
+    print("DeepSeek Models:", models)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## 🔌 Massive Universal Provider Registry
+
+LLMCycle is pre-configured with base URLs for the most popular platforms:
+`OPENAI`, `DEEPSEEK`, `ANTHROPIC`, `TOGETHER`, `GROQ`, `MISTRAL`, `PERPLEXITY`, `ANYSCALE`, `FIREWORKS`, `COHERE`, `DATABRICKS`, `HUGGINGFACE`.
+
+**Wildcard Support:** If you type `RANDOM_API_KEYS`, LLMCycle will automatically assume `https://api.random.com/v1`. If that's wrong, just define `RANDOM_BASE_URL` in your `.env`!

llmcycle-0.1.0/example.py

@@ -0,0 +1,58 @@
+import asyncio
+from llmcycle.core.keys import KeyManager
+from llmcycle.core.router import ModelRouter, FallbackRouter
+from llmcycle.core.stream import StreamResilienceManager
+from llmcycle.schema import CompletionRequest, Message
+from llmcycle.providers.base import LLMProvider
+from typing import AsyncGenerator
+
+class MockProvider(LLMProvider):
+    """A mock provider that fails midway through the stream to test resilience."""
+    def __init__(self, should_fail: bool):
+        self.should_fail = should_fail
+        
+    async def generate(self, request: CompletionRequest, api_key: str) -> str:
+        return "Hello World"
+        
+    async def generate_stream(self, request: CompletionRequest, api_key: str) -> AsyncGenerator[str, None]:
+        chunks = ["Hello", " world", ",", " how", " are", " you?"]
+        for i, chunk in enumerate(chunks):
+            if self.should_fail and i == 3:
+                raise ConnectionError("Mock streaming disconnect")
+            yield chunk
+            await asyncio.sleep(0.1)
+            
+    async def get_models(self, api_key: str) -> list[str]:
+        return ["gpt-4", "gpt-4-turbo"]
+
+async def main():
+    print("Setting up LLMCycle Manager...")
+    km = KeyManager()
+    km.add_key("gpt-4-turbo", "sk-mock-1")
+    km.add_key("gpt-4", "sk-mock-2")
+    
+    router = ModelRouter(FallbackRouter({"gpt-4-turbo": ["gpt-4"]}))
+    
+    providers = {
+        "gpt-4-turbo": MockProvider(should_fail=True),
+        "gpt-4": MockProvider(should_fail=False)
+    }
+    
+    stream_manager = StreamResilienceManager(router, km, providers)
+    
+    request = CompletionRequest(
+        model="gpt-4-turbo",
+        messages=[Message(role="user", content="Say hello!")]
+    )
+    
+    print("\nStarting robust stream...")
+    try:
+        async for chunk in stream_manager.safe_stream(request):
+            print(chunk, end="", flush=True)
+    except Exception as e:
+        print(f"\nStream failed completely: {e}")
+        
+    print("\n\nFinished stream successfully, even with mid-stream disconnect!")
+
+if __name__ == "__main__":
+    asyncio.run(main())

llmcycle-0.1.0/example_auto.py

@@ -0,0 +1,29 @@
+import asyncio
+from llmcycle import LLMCycle
+
+async def main():
+    print("Initializing LLMCycle (auto-loading from .env)...")
+    # You can provide fallbacks, so if deepseek fails, it falls back to openai
+    client = LLMCycle(
+        env_path=".env",
+        custom_fallbacks={
+            "deepseek": ["openai"]
+        }
+    )
+    
+    # 1. Get Available Providers automatically loaded from env
+    providers = client.get_available_providers()
+    print(f"\nProviders automatically loaded: {providers}")
+    
+    # 2. Get Models for a specific provider (This will make a network request to the base URL)
+    # Note: Since the keys in our .env are fake, this might fail unless we catch it
+    if "deepseek" in providers:
+        print("\nFetching models for DeepSeek using loaded keys...")
+        models = await client.get_provider_models("deepseek")
+        if models:
+            print(f"DeepSeek Models: {models[:3]} ...")
+        else:
+            print("Failed to fetch models (Fake API key used in .env)")
+
+if __name__ == "__main__":
+    asyncio.run(main())

llmcycle-0.1.0/main.py

@@ -0,0 +1,6 @@
+def main():
+    print("Hello from llmcycle!")
+
+
+if __name__ == "__main__":
+    main()

llmcycle-0.1.0/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "llmcycle"
+version = "0.1.0"
+description = "Cycle through multiple LLM providers with smart fallback, load balancing, and unified API"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "fastapi>=0.136.1",
+    "httpx>=0.28.1",
+    "jinja2>=3.1.6",
+    "pydantic>=2.13.4",
+    "python-dotenv>=1.2.2",
+    "python-multipart>=0.0.29",
+    "uvicorn>=0.47.0",
+]
+
+[project.scripts]
+llmcycle = "llmcycle.cli:main"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.3",
+    "pytest-asyncio>=1.3.0",
+]

llmcycle-0.1.0/src/llmcycle/__init__.py

@@ -0,0 +1,25 @@
+"""
+LLMCycle - Production-Grade Universal LLM Router
+=================================================
+Auto multi-key rotation, intelligent error handling, streaming resilience,
+and support for 50+ providers out-of-the-box.
+"""
+from .client import LLMCycle
+from .schema import CompletionRequest, Message, CompletionResponse, StreamChunk
+from .core.keys import KeyManager, KeyStatus
+from .core.router import ModelRouter, RoutingStrategy
+from .core.errors import (
+    LLMCycleError, RateLimitError, AuthenticationError,
+    ProviderError, AllProvidersFailedError, StreamInterruptedError,
+)
+
+__all__ = [
+    "LLMCycle",
+    "CompletionRequest", "Message", "CompletionResponse", "StreamChunk",
+    "KeyManager", "KeyStatus",
+    "ModelRouter", "RoutingStrategy",
+    "LLMCycleError", "RateLimitError", "AuthenticationError",
+    "ProviderError", "AllProvidersFailedError", "StreamInterruptedError",
+]
+
+__version__ = "0.2.0"

llmcycle-0.1.0/src/llmcycle/cli.py

@@ -0,0 +1,17 @@
+import argparse
+import uvicorn
+
+def main():
+    parser = argparse.ArgumentParser(description="LLMCycle CLI")
+    parser.add_argument("command", choices=["ui"], help="Command to run")
+    parser.add_argument("--host", default="127.0.0.1", help="Host for the UI")
+    parser.add_argument("--port", type=int, default=8000, help="Port for the UI")
+
+    args = parser.parse_args()
+
+    if args.command == "ui":
+        print(f"Starting LLMCycle Dashboard on http://{args.host}:{args.port}")
+        uvicorn.run("llmcycle.ui.app:app", host=args.host, port=args.port, reload=True)
+
+if __name__ == "__main__":
+    main()

llmcycle-0.1.0/src/llmcycle/client.py

@@ -0,0 +1,73 @@
+import os
+from dotenv import load_dotenv
+from llmcycle.core.keys import KeyManager
+from llmcycle.core.router import ModelRouter, FallbackRouter
+from llmcycle.core.stream import StreamResilienceManager
+from llmcycle.providers.openai_compatible import OpenAICompatibleProvider
+
+# Massive default registry
+PROVIDER_BASE_URLS = {
+    "OPENAI": "https://api.openai.com/v1",
+    "DEEPSEEK": "https://api.deepseek.com/v1",
+    "ANTHROPIC": "https://api.anthropic.com/v1",
+    "TOGETHER": "https://api.together.xyz/v1",
+    "GROQ": "https://api.groq.com/openai/v1",
+    "MISTRAL": "https://api.mistral.ai/v1",
+    "PERPLEXITY": "https://api.perplexity.ai",
+    "ANYSCALE": "https://api.endpoints.anyscale.com/v1",
+    "FIREWORKS": "https://api.fireworks.ai/inference/v1",
+    "COHERE": "https://api.cohere.com/v1",
+    "DATABRICKS": "https://serving.api.databricks.com/serving-endpoints",
+    "HUGGINGFACE": "https://api-inference.huggingface.co/models",
+}
+
+class LLMCycle:
+    """Main entrypoint for LLMCycle with Universal Provider Support."""
+    
+    def __init__(self, env_path: str = ".env", custom_fallbacks: dict = None):
+        load_dotenv(env_path)
+        
+        self.key_manager = KeyManager()
+        self.providers = {}
+        
+        # Auto-discover
+        self._auto_load_keys()
+        
+        # Setup Routing Strategy
+        fallbacks = custom_fallbacks or {}
+        self.router = ModelRouter(FallbackRouter(fallbacks))
+        self.stream_manager = StreamResilienceManager(self.router, self.key_manager, self.providers)
+
+    def _auto_load_keys(self):
+        """Finds any env var ending with _API_KEYS and universally registers the provider."""
+        for key, val in os.environ.items():
+            if key.endswith("_API_KEYS"):
+                provider_name = key.replace("_API_KEYS", "").upper()
+                keys = [k.strip() for k in val.split(",") if k.strip()]
+                
+                if not keys:
+                    continue
+                    
+                # 1. Check if user explicitly defined a BASE URL for this provider
+                # 2. Check the massive default registry
+                # 3. Fallback: Assume a standard OpenAI compatible format
+                base_url = os.environ.get(f"{provider_name}_BASE_URL")
+                if not base_url:
+                    base_url = PROVIDER_BASE_URLS.get(provider_name, f"https://api.{provider_name.lower()}.com/v1")
+                
+                if base_url:
+                    self.providers[provider_name.lower()] = OpenAICompatibleProvider(base_url)
+                    for k in keys:
+                        self.key_manager.add_key(provider_name.lower(), k)
+
+    def get_available_providers(self) -> list[str]:
+        return list(self.providers.keys())
+
+    async def get_provider_models(self, provider_name: str) -> list[str]:
+        p_name = provider_name.lower()
+        if p_name not in self.providers:
+            return []
+        key = self.key_manager.get_next_key(p_name)
+        if not key:
+            return []
+        return await self.providers[p_name].get_models(key)

llmcycle-0.1.0/src/llmcycle/core/errors.py

@@ -0,0 +1,71 @@
+"""
+Custom exceptions for LLMCycle.
+All errors map from HTTP status codes so the router knows exactly
+what to do: retry, rotate key, skip provider, or give up.
+"""
+
+class LLMCycleError(Exception):
+    """Base error for all LLMCycle exceptions."""
+    def __init__(self, message: str, provider: str = "", model: str = "", status_code: int = 0):
+        super().__init__(message)
+        self.provider = provider
+        self.model = model
+        self.status_code = status_code
+
+class RateLimitError(LLMCycleError):
+    """429: Rate limit exceeded. Rotate key and retry."""
+    pass
+
+class AuthenticationError(LLMCycleError):
+    """401: Invalid API key. Disable key permanently."""
+    pass
+
+class ProviderError(LLMCycleError):
+    """400/500+: Provider-side error. Try next provider."""
+    pass
+
+class QuotaExceededError(LLMCycleError):
+    """402/429 with quota message: Key quota exhausted. Rotate key."""
+    pass
+
+class ContentPolicyError(LLMCycleError):
+    """400: Content policy violation. Do NOT retry - fail fast."""
+    pass
+
+class StreamInterruptedError(LLMCycleError):
+    """Mid-stream disconnection. Contains partial text generated so far."""
+    def __init__(self, message: str, partial_text: str = "", **kwargs):
+        super().__init__(message, **kwargs)
+        self.partial_text = partial_text
+
+class AllProvidersFailedError(LLMCycleError):
+    """Raised when every provider in the fallback chain fails."""
+    def __init__(self, errors: list):
+        msg = f"All {len(errors)} providers failed. Last error: {errors[-1]}"
+        super().__init__(msg)
+        self.errors = errors
+
+
+# Maps HTTP status codes to exception types
+def classify_http_error(status_code: int, response_text: str, provider: str, model: str) -> LLMCycleError:
+    """Factory: Convert an HTTP error into the correct LLMCycle exception."""
+    text_lower = response_text.lower()
+
+    if status_code == 401:
+        return AuthenticationError(f"[{provider}] Auth failed (401): {response_text}", provider=provider, model=model, status_code=status_code)
+
+    if status_code == 429:
+        if any(w in text_lower for w in ("quota", "limit exceeded", "exhausted", "billing")):
+            return QuotaExceededError(f"[{provider}] Quota exceeded (429): {response_text}", provider=provider, model=model, status_code=status_code)
+        return RateLimitError(f"[{provider}] Rate limited (429): {response_text}", provider=provider, model=model, status_code=status_code)
+
+    if status_code == 402:
+        return QuotaExceededError(f"[{provider}] Payment required (402): {response_text}", provider=provider, model=model, status_code=status_code)
+
+    if status_code == 400:
+        if any(w in text_lower for w in ("content_policy", "safety", "harmful", "violat")):
+            return ContentPolicyError(f"[{provider}] Content policy (400): {response_text}", provider=provider, model=model, status_code=status_code)
+        return ProviderError(f"[{provider}] Bad request (400): {response_text}", provider=provider, model=model, status_code=status_code)
+
+    # All other 4xx/5xx
+    return ProviderError(f"[{provider}] HTTP {status_code}: {response_text[:200]}", provider=provider, model=model, status_code=status_code)