agentic-sre-gateway 1.0.0__tar.gz

agentic_sre_gateway-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Manik Bodamwad
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agentic_sre_gateway-1.0.0/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.4
+Name: agentic-sre-gateway
+Version: 1.0.0
+Summary: An SRE-Optimized API Gateway for dynamic LLM routing, Redis caching, and Prometheus telemetry.
+Home-page: https://github.com/ManikBodamwad/LLM-Latency-Cost-Router
+Author: Manik Bodamwad
+Author-email: bodamwadm@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.100.0
+Requires-Dist: uvicorn>=0.23.0
+Requires-Dist: litellm>=1.0.0
+Requires-Dist: redis>=5.0.0
+Requires-Dist: prometheus_client>=0.17.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Agentic API Gateway | SRE Edge Router
+
+A production-grade, highly resilient API Gateway that dynamically routes Large Language Model (LLM) prompts based on complexity and minimizes cost through semantic caching. 
+
+Built strictly with **Site Reliability Engineering (SRE)** principles, it implements automated failovers, Redis-backed rate limiters, token-cost telemetry routing to Prometheus, and features a glowing, immersive UI dashboard built beautifully in Vanilla JS/Vite.
+
+## 🚀 Core SRE Features
+
+- **Dynamic Tier Routing**: Uses a heuristics engine to parse prompt intent. Simple queries route to blazing-fast models (`groq/llama-3.1-8b`), while complex queries (e.g. system design) automatically route to heavy models (`groq/llama-3.1-70b`).
+- **Zero-Latency Semantic Caching**: SHA-256 hashes intercepts inbound requests. Identical requests skip the LLM network entirely, serving an exact match directly from Redis memory in `< 0ms` for `$0USD` cost.
+- **Intelligent Failover Resiliency**: Wraps primary model calls in strict `asyncio` timeouts. If a provider throws a quota limit, `503`, or hangs, the router gracefully degrades to alternative models before ever throwing an error to the user.
+- **Vite SRE Telemetry Dashboard**: Complete visual interface built without bulky frameworks—utilizing raw CSS glassmorphism, flexbox scaling, and micro-animated charts showcasing true request latency and cost updates on every stream.
+- **DDoS/Billing Defense**: Implements a Redis token-bucket API rate limiter (50 req/min) requiring an `x-api-key` header to prevent billing exhaustion.
+- **Prometheus & Grafana Observability**: Instrumentated with custom Python metrics exposing End-to-End Latency Histograms, LLM Token Cost accumulations, and Routing Cache Hit/Miss rates to `/metrics`.
+
+## 🛠️ Tech Stack & Architecture
+
+- **Backend Route Logic**: `Python 3.11`, `FastAPI`, `LiteLLM` (for multi-provider standardization)
+- **Frontend Dashboard**: Raw `HTML5`, Vanilla Base `CSS`, `Vite` Node-Server, `marked.js`
+- **Cache & Memory**: `Redis` alpine container
+- **Orchestration**: `Docker Compose`
+- **Observability**: `Prometheus` (Scraping), `Grafana` (Visualization)
+- **Inference Hardware**: `Groq LPU` (Llama 3.1 models config standard)
+
+## ⚡ Quickstart
+
+1. **Clone the repository:**
+   ```bash
+   git clone https://github.com/ManikBodamwad/LLM-Latency-Cost-Router.git
+   cd "LLM Latency & Cost router"
+   ```
+
+2. **Supply your API Keys:**
+   Create a `.env` file in the root directory:
+   ```env
+   GROQ_API_KEY="gsk_your_groq_key_here"
+   ```
+
+3. **Deploy via Docker Compose:**
+   ```bash
+   docker compose up -d --build
+   ```
+
+4. **Experience the Application:**
+   Open [http://localhost:5173](http://localhost:5173) in your web browser. Type a complex prompt like *"Can you explain the Medallion Architecture?"* and observe the SRE dashboard dynamically tracking the latency, the exact Token Cost, and the routing strategy in real-time.
+
+## 📦 Usage as a Python Package
+
+This repository is built as a portable Python package so engineering teams can inject edge-routing into their own systems natively without bulky Docker containers!
+
+If you install this via pip:
+```bash
+pip install agentic-sre-gateway
+```
+
+You can instantly spin up the SRE-optimized Routing API on your local terminal using the globally injected command:
+```bash
+export GROQ_API_KEY="your_key"
+export REDIS_URL="redis://localhost:6379/0"
+agentic-gateway
+```
+This serves teams that want a drop-in API proxy to massively reduce LLM bills and monitor token consumption locally without rewriting complex LiteLLM and Prometheus wrappers themselves.
+
+## 📊 View Local Development Telemetry
+
+For local visualization during development, the `docker-compose` orchestration automatically spins up standard metrics scrape targets. 
+
+- **Prometheus Scraper UI**: `http://localhost:9090`
+- **Grafana Workspace**: `http://localhost:3000` 
+  *(Note: This uses the default local-dev credentials Login: `admin` / Password: `admin`)*
+
+---
+
+*Developed by Manik Bodamwad to solve enterprise-level LLM deployment friction points: Cost Runaway, High Latency, and Provider Downtime.*

agentic_sre_gateway-1.0.0/README.md

@@ -0,0 +1,74 @@
+# Agentic API Gateway | SRE Edge Router
+
+A production-grade, highly resilient API Gateway that dynamically routes Large Language Model (LLM) prompts based on complexity and minimizes cost through semantic caching. 
+
+Built strictly with **Site Reliability Engineering (SRE)** principles, it implements automated failovers, Redis-backed rate limiters, token-cost telemetry routing to Prometheus, and features a glowing, immersive UI dashboard built beautifully in Vanilla JS/Vite.
+
+## 🚀 Core SRE Features
+
+- **Dynamic Tier Routing**: Uses a heuristics engine to parse prompt intent. Simple queries route to blazing-fast models (`groq/llama-3.1-8b`), while complex queries (e.g. system design) automatically route to heavy models (`groq/llama-3.1-70b`).
+- **Zero-Latency Semantic Caching**: SHA-256 hashes intercepts inbound requests. Identical requests skip the LLM network entirely, serving an exact match directly from Redis memory in `< 0ms` for `$0USD` cost.
+- **Intelligent Failover Resiliency**: Wraps primary model calls in strict `asyncio` timeouts. If a provider throws a quota limit, `503`, or hangs, the router gracefully degrades to alternative models before ever throwing an error to the user.
+- **Vite SRE Telemetry Dashboard**: Complete visual interface built without bulky frameworks—utilizing raw CSS glassmorphism, flexbox scaling, and micro-animated charts showcasing true request latency and cost updates on every stream.
+- **DDoS/Billing Defense**: Implements a Redis token-bucket API rate limiter (50 req/min) requiring an `x-api-key` header to prevent billing exhaustion.
+- **Prometheus & Grafana Observability**: Instrumentated with custom Python metrics exposing End-to-End Latency Histograms, LLM Token Cost accumulations, and Routing Cache Hit/Miss rates to `/metrics`.
+
+## 🛠️ Tech Stack & Architecture
+
+- **Backend Route Logic**: `Python 3.11`, `FastAPI`, `LiteLLM` (for multi-provider standardization)
+- **Frontend Dashboard**: Raw `HTML5`, Vanilla Base `CSS`, `Vite` Node-Server, `marked.js`
+- **Cache & Memory**: `Redis` alpine container
+- **Orchestration**: `Docker Compose`
+- **Observability**: `Prometheus` (Scraping), `Grafana` (Visualization)
+- **Inference Hardware**: `Groq LPU` (Llama 3.1 models config standard)
+
+## ⚡ Quickstart
+
+1. **Clone the repository:**
+   ```bash
+   git clone https://github.com/ManikBodamwad/LLM-Latency-Cost-Router.git
+   cd "LLM Latency & Cost router"
+   ```
+
+2. **Supply your API Keys:**
+   Create a `.env` file in the root directory:
+   ```env
+   GROQ_API_KEY="gsk_your_groq_key_here"
+   ```
+
+3. **Deploy via Docker Compose:**
+   ```bash
+   docker compose up -d --build
+   ```
+
+4. **Experience the Application:**
+   Open [http://localhost:5173](http://localhost:5173) in your web browser. Type a complex prompt like *"Can you explain the Medallion Architecture?"* and observe the SRE dashboard dynamically tracking the latency, the exact Token Cost, and the routing strategy in real-time.
+
+## 📦 Usage as a Python Package
+
+This repository is built as a portable Python package so engineering teams can inject edge-routing into their own systems natively without bulky Docker containers!
+
+If you install this via pip:
+```bash
+pip install agentic-sre-gateway
+```
+
+You can instantly spin up the SRE-optimized Routing API on your local terminal using the globally injected command:
+```bash
+export GROQ_API_KEY="your_key"
+export REDIS_URL="redis://localhost:6379/0"
+agentic-gateway
+```
+This serves teams that want a drop-in API proxy to massively reduce LLM bills and monitor token consumption locally without rewriting complex LiteLLM and Prometheus wrappers themselves.
+
+## 📊 View Local Development Telemetry
+
+For local visualization during development, the `docker-compose` orchestration automatically spins up standard metrics scrape targets. 
+
+- **Prometheus Scraper UI**: `http://localhost:9090`
+- **Grafana Workspace**: `http://localhost:3000` 
+  *(Note: This uses the default local-dev credentials Login: `admin` / Password: `admin`)*
+
+---
+
+*Developed by Manik Bodamwad to solve enterprise-level LLM deployment friction points: Cost Runaway, High Latency, and Provider Downtime.*

agentic_sre_gateway-1.0.0/agentic_sre_gateway.egg-info/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.4
+Name: agentic-sre-gateway
+Version: 1.0.0
+Summary: An SRE-Optimized API Gateway for dynamic LLM routing, Redis caching, and Prometheus telemetry.
+Home-page: https://github.com/ManikBodamwad/LLM-Latency-Cost-Router
+Author: Manik Bodamwad
+Author-email: bodamwadm@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.100.0
+Requires-Dist: uvicorn>=0.23.0
+Requires-Dist: litellm>=1.0.0
+Requires-Dist: redis>=5.0.0
+Requires-Dist: prometheus_client>=0.17.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Agentic API Gateway | SRE Edge Router
+
+A production-grade, highly resilient API Gateway that dynamically routes Large Language Model (LLM) prompts based on complexity and minimizes cost through semantic caching. 
+
+Built strictly with **Site Reliability Engineering (SRE)** principles, it implements automated failovers, Redis-backed rate limiters, token-cost telemetry routing to Prometheus, and features a glowing, immersive UI dashboard built beautifully in Vanilla JS/Vite.
+
+## 🚀 Core SRE Features
+
+- **Dynamic Tier Routing**: Uses a heuristics engine to parse prompt intent. Simple queries route to blazing-fast models (`groq/llama-3.1-8b`), while complex queries (e.g. system design) automatically route to heavy models (`groq/llama-3.1-70b`).
+- **Zero-Latency Semantic Caching**: SHA-256 hashes intercepts inbound requests. Identical requests skip the LLM network entirely, serving an exact match directly from Redis memory in `< 0ms` for `$0USD` cost.
+- **Intelligent Failover Resiliency**: Wraps primary model calls in strict `asyncio` timeouts. If a provider throws a quota limit, `503`, or hangs, the router gracefully degrades to alternative models before ever throwing an error to the user.
+- **Vite SRE Telemetry Dashboard**: Complete visual interface built without bulky frameworks—utilizing raw CSS glassmorphism, flexbox scaling, and micro-animated charts showcasing true request latency and cost updates on every stream.
+- **DDoS/Billing Defense**: Implements a Redis token-bucket API rate limiter (50 req/min) requiring an `x-api-key` header to prevent billing exhaustion.
+- **Prometheus & Grafana Observability**: Instrumentated with custom Python metrics exposing End-to-End Latency Histograms, LLM Token Cost accumulations, and Routing Cache Hit/Miss rates to `/metrics`.
+
+## 🛠️ Tech Stack & Architecture
+
+- **Backend Route Logic**: `Python 3.11`, `FastAPI`, `LiteLLM` (for multi-provider standardization)
+- **Frontend Dashboard**: Raw `HTML5`, Vanilla Base `CSS`, `Vite` Node-Server, `marked.js`
+- **Cache & Memory**: `Redis` alpine container
+- **Orchestration**: `Docker Compose`
+- **Observability**: `Prometheus` (Scraping), `Grafana` (Visualization)
+- **Inference Hardware**: `Groq LPU` (Llama 3.1 models config standard)
+
+## ⚡ Quickstart
+
+1. **Clone the repository:**
+   ```bash
+   git clone https://github.com/ManikBodamwad/LLM-Latency-Cost-Router.git
+   cd "LLM Latency & Cost router"
+   ```
+
+2. **Supply your API Keys:**
+   Create a `.env` file in the root directory:
+   ```env
+   GROQ_API_KEY="gsk_your_groq_key_here"
+   ```
+
+3. **Deploy via Docker Compose:**
+   ```bash
+   docker compose up -d --build
+   ```
+
+4. **Experience the Application:**
+   Open [http://localhost:5173](http://localhost:5173) in your web browser. Type a complex prompt like *"Can you explain the Medallion Architecture?"* and observe the SRE dashboard dynamically tracking the latency, the exact Token Cost, and the routing strategy in real-time.
+
+## 📦 Usage as a Python Package
+
+This repository is built as a portable Python package so engineering teams can inject edge-routing into their own systems natively without bulky Docker containers!
+
+If you install this via pip:
+```bash
+pip install agentic-sre-gateway
+```
+
+You can instantly spin up the SRE-optimized Routing API on your local terminal using the globally injected command:
+```bash
+export GROQ_API_KEY="your_key"
+export REDIS_URL="redis://localhost:6379/0"
+agentic-gateway
+```
+This serves teams that want a drop-in API proxy to massively reduce LLM bills and monitor token consumption locally without rewriting complex LiteLLM and Prometheus wrappers themselves.
+
+## 📊 View Local Development Telemetry
+
+For local visualization during development, the `docker-compose` orchestration automatically spins up standard metrics scrape targets. 
+
+- **Prometheus Scraper UI**: `http://localhost:9090`
+- **Grafana Workspace**: `http://localhost:3000` 
+  *(Note: This uses the default local-dev credentials Login: `admin` / Password: `admin`)*
+
+---
+
+*Developed by Manik Bodamwad to solve enterprise-level LLM deployment friction points: Cost Runaway, High Latency, and Provider Downtime.*

agentic_sre_gateway-1.0.0/agentic_sre_gateway.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+LICENSE
+README.md
+setup.py
+agentic_sre_gateway.egg-info/PKG-INFO
+agentic_sre_gateway.egg-info/SOURCES.txt
+agentic_sre_gateway.egg-info/dependency_links.txt
+agentic_sre_gateway.egg-info/entry_points.txt
+agentic_sre_gateway.egg-info/requires.txt
+agentic_sre_gateway.egg-info/top_level.txt
+app/__init__.py
+app/cli.py
+app/main.py

agentic_sre_gateway-1.0.0/agentic_sre_gateway.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

agentic_sre_gateway-1.0.0/agentic_sre_gateway.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+agentic-gateway = app.cli:start_gateway

agentic_sre_gateway-1.0.0/agentic_sre_gateway.egg-info/requires.txt

@@ -0,0 +1,5 @@
+fastapi>=0.100.0
+uvicorn>=0.23.0
+litellm>=1.0.0
+redis>=5.0.0
+prometheus_client>=0.17.0

agentic_sre_gateway-1.0.0/agentic_sre_gateway.egg-info/top_level.txt

@@ -0,0 +1 @@
+app

agentic_sre_gateway-1.0.0/app/__init__.py

@@ -0,0 +1 @@
+# Making the app directory a discoverable python package.

agentic_sre_gateway-1.0.0/app/cli.py

@@ -0,0 +1,13 @@
+import uvicorn
+import os
+
+def start_gateway():
+    """
+    Entrypoint for the CLI. Starts the Agentic SRE Gateway via Uvicorn.
+    """
+    print("🚀 Starting Agentic SRE Gateway on http://0.0.0.0:8000")
+    print("Make sure you have Redis running and GROQ_API_KEY in your environment.")
+    uvicorn.run("app.main:app", host="0.0.0.0", port=8000, reload=False)
+
+if __name__ == "__main__":
+    start_gateway()

agentic_sre_gateway-1.0.0/app/main.py

@@ -0,0 +1,199 @@
+from fastapi import FastAPI, HTTPException, Request, Header, Depends
+from fastapi.responses import StreamingResponse
+from pydantic import BaseModel
+import time
+import asyncio
+import os
+import hashlib
+import redis.asyncio as redis
+from prometheus_client import make_asgi_app, Counter, Histogram
+from fastapi.middleware.cors import CORSMiddleware
+
+app = FastAPI(title="LLM Router API")
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=False,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Initialize Redis client
+redis_client = redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379/0"), decode_responses=True)
+
+# Metrics definitions
+REQUEST_LATENCY = Histogram(
+    'llm_request_latency_seconds',
+    'Latency of LLM requests',
+    ['model', 'tier']
+)
+
+TOKEN_COST = Counter(
+    'llm_token_cost_total',
+    'Estimated cost in USD',
+    ['model', 'tier']
+)
+
+FAILOVER_COUNT = Counter(
+    'llm_failover_count_total',
+    'Number of failovers',
+    ['original_model', 'fallback_model', 'reason']
+)
+
+CACHE_HITS = Counter('llm_cache_hits_total', 'Number of cache hits')
+CACHE_MISSES = Counter('llm_cache_misses_total', 'Number of cache misses')
+
+import litellm
+
+# Constants for pricing mock will be handled by litellm's completion_cost
+class ChatRequest(BaseModel):
+    prompt: str
+    stream: bool = False
+
+async def verify_api_key(x_api_key: str = Header(...)):
+    if not x_api_key:
+        raise HTTPException(status_code=401, detail="API Key required")
+    # Fixed window rate limiter (50 req / minute)
+    key = f"rate_limit:{x_api_key}"
+    requests = await redis_client.incr(key)
+    if requests == 1:
+        await redis_client.expire(key, 60)
+    if requests > 50:
+        raise HTTPException(status_code=429, detail="Rate limit exceeded: 50 requests per minute")
+    return x_api_key
+
+def determine_complexity(prompt: str) -> str:
+    complex_keywords = ["analyze", "code", "architecture", "summarize", "complex"]
+    if len(prompt) > 1000:
+        return "complex"
+    if any(kw in prompt.lower() for kw in complex_keywords):
+        return "complex"
+    return "simple"
+
+@app.post("/api/v1/chat/completions")
+async def chat_completions(req: ChatRequest, api_key: str = Depends(verify_api_key)):
+    tier = determine_complexity(req.prompt)
+    
+    if tier == "simple":
+        target_model = "groq/llama-3.1-8b-instant"
+        fallback_model = "groq/llama-3.1-8b-instant"
+        timeout_seconds = 5.0
+    else:
+        target_model = "groq/llama-3.1-70b-versatile"
+        fallback_model = "groq/llama-3.1-8b-instant"  
+        timeout_seconds = 10.0
+
+    start_time = time.time()
+    response = None
+    used_model = target_model
+    
+    messages = [{"role": "user", "content": req.prompt}]
+
+    # Check cache first
+    prompt_hash = hashlib.sha256(req.prompt.encode()).hexdigest()
+    cached_response = await redis_client.get(prompt_hash)
+    
+    if cached_response:
+        CACHE_HITS.inc()
+        latency = time.time() - start_time
+        REQUEST_LATENCY.labels(model="cache", tier=tier).observe(latency)
+        
+        if req.stream:
+            async def cache_streamer():
+                yield cached_response
+            return StreamingResponse(cache_streamer(), media_type="text/event-stream")
+            
+        return {
+            "tier": tier,
+            "latency": round(latency, 3),
+            "used_model": "cache",
+            "response": cached_response,
+            "cost_usd": 0.0
+        }
+        
+    CACHE_MISSES.inc()
+
+    if req.stream:
+        try:
+            response_stream = await litellm.acompletion(model=target_model, messages=messages, timeout=timeout_seconds, stream=True)
+        except Exception as e:
+            error_type = "timeout" if isinstance(e, asyncio.TimeoutError) else "error_503"
+            FAILOVER_COUNT.labels(original_model=target_model, fallback_model=fallback_model, reason=error_type).inc()
+            used_model = fallback_model
+            
+            try:
+                response_stream = await litellm.acompletion(model=fallback_model, messages=messages, timeout=timeout_seconds, stream=True)
+            except Exception as e2:
+                raise HTTPException(status_code=503, detail=f"Core and Fallback LLM failed: {str(e2)}")
+
+        async def stream_generator():
+            full_text = ""
+            async for chunk in response_stream:
+                content = chunk.choices[0].delta.content or ""
+                full_text += content
+                yield content
+                
+            latency = time.time() - start_time
+            REQUEST_LATENCY.labels(model=used_model, tier=tier).observe(latency)
+            
+            try:
+                await redis_client.setex(prompt_hash, 3600, full_text)
+            except Exception:
+                pass
+                
+        return StreamingResponse(stream_generator(), media_type="text/event-stream")
+
+    try:
+        # Attempt primary model
+        response = await litellm.acompletion(
+            model=target_model,
+            messages=messages,
+            timeout=timeout_seconds
+        )
+    except Exception as e:
+        # Handle timeout or 503 or any exception by falling back
+        error_type = "timeout" if isinstance(e, asyncio.TimeoutError) else "error_503"
+        FAILOVER_COUNT.labels(original_model=target_model, fallback_model=fallback_model, reason=error_type).inc()
+        used_model = fallback_model
+        
+        try:
+            # Call fallback
+            response = await litellm.acompletion(
+                model=fallback_model,
+                messages=messages,
+                timeout=timeout_seconds
+            )
+        except Exception as e2:
+            raise HTTPException(status_code=503, detail=f"LLM Quota Exceeded or fully down: {str(e2)}")
+    
+    latency = time.time() - start_time
+    REQUEST_LATENCY.labels(model=used_model, tier=tier).observe(latency)
+    
+    try:
+        cost = litellm.completion_cost(completion_response=response, model=used_model) or 0.0
+    except Exception:
+        cost = 0.0
+        
+    TOKEN_COST.labels(model=used_model, tier=tier).inc(cost)
+    
+    # Extract string from response format
+    content_str = response.choices[0].message.content
+    
+    # Store in cache for 1 hour (3600 seconds)
+    try:
+        await redis_client.setex(prompt_hash, 3600, content_str)
+    except Exception as e:
+        pass # Ignore cache write errors
+
+    return {
+        "tier": tier, 
+        "latency": round(latency, 3), 
+        "used_model": used_model,
+        "response": content_str,
+        "cost_usd": cost
+    }
+
+# Mount prometheus metrics endpoint
+metrics_app = make_asgi_app()
+app.mount("/metrics", metrics_app)

agentic_sre_gateway-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

agentic_sre_gateway-1.0.0/setup.py

@@ -0,0 +1,36 @@
+from setuptools import setup, find_packages
+
+with open("README.md", "r", encoding="utf-8") as fh:
+    long_description = fh.read()
+
+setup(
+    name="agentic-sre-gateway",
+    version="1.0.0",
+    author="Manik Bodamwad",
+    author_email="bodamwadm@gmail.com",
+    description="An SRE-Optimized API Gateway for dynamic LLM routing, Redis caching, and Prometheus telemetry.",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/ManikBodamwad/LLM-Latency-Cost-Router",
+    packages=find_packages(),
+    install_requires=[
+        "fastapi>=0.100.0",
+        "uvicorn>=0.23.0",
+        "litellm>=1.0.0",
+        "redis>=5.0.0",
+        "prometheus_client>=0.17.0"
+    ],
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+        "Topic :: Software Development :: Libraries :: Python Modules",
+        "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    ],
+    python_requires=">=3.9",
+    entry_points={
+        "console_scripts": [
+            "agentic-gateway=app.cli:start_gateway",
+        ],
+    },
+)