modelswitch 0.1.1__tar.gz

modelswitch-0.1.1/.github/workflows/ci.yml

@@ -0,0 +1,63 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+    tags: ["v*"]
+  pull_request:
+    branches: [main]
+
+permissions:
+  contents: write
+  id-token: write
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install pytest pytest-asyncio pytest-timeout litellm==1.82.6
+
+      - name: Run tests
+        run: python -m pytest tests/ -v --timeout=60
+
+  publish:
+    name: Publish to PyPI
+    if: startsWith(github.ref, 'refs/tags/v')
+    needs: test
+    runs-on: ubuntu-latest
+    environment: pypi
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tools
+        run: python -m pip install --upgrade pip build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

modelswitch-0.1.1/.gitignore

@@ -0,0 +1,16 @@
+logs/
+data/
+__pycache__/
+*.pyc
+.env
+.venv
+.coverage
+config.yaml
+.DS_Store
+.claude/
+.pytest_cache/
+.qoder/
+.vscode/
+dist/
+*.egg-info/
+build/

modelswitch-0.1.1/.zedrules

@@ -0,0 +1,77 @@
+# ModelSwitch Development Rules
+
+## Project Overview
+
+ModelSwitch is an LLM gateway proxy exposing OpenAI-compatible and Anthropic-compatible APIs with automatic fallback chains, rate limiting, and usage tracking.
+
+## Key Constraints
+
+### Dependencies
+- `litellm` is pinned to `1.82.6` — **NEVER upgrade** (supply chain attack on 1.82.7/1.82.8)
+
+### Middleware
+- `GatewayMiddleware` is pure ASGI (`__call__(scope, receive, send)`)
+- **NEVER** use `BaseHTTPMiddleware` or `@app.middleware("http")` — causes infinite recursion
+
+### Streaming
+- Use `resp_ref` closure pattern to capture usage from async stream generators
+- Adapters return `AdapterResponse` with `body` (non-stream) or `stream` (async generator)
+- Route handlers read `_stream_adapter_info` dict after stream completes
+
+## Development Workflow
+
+### Before Starting
+1. Run `git status` — commit any uncommitted changes first
+2. Create feature branch: `git checkout -b feature/[name]`
+
+### During Development
+1. Follow existing code patterns
+2. Check `CLAUDE.md` for architecture details
+3. Make small, focused commits
+
+### Before Committing
+1. Run tests: `python -m pytest tests/ -v --timeout=60`
+2. Update documentation if architecture changed
+3. Add test cases for new functionality
+
+### Commit Format
+```
+<type>: <description>
+
+- Change 1
+- Change 2
+
+Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+```
+
+Types: `feat`, `fix`, `refactor`, `docs`, `test`, `chore`, `ci`
+
+### After Pushing
+- CI runs automatically on Python 3.10–3.13
+- Check: https://github.com/ddmonster/modelswitch/actions
+
+## Quick Commands
+
+```bash
+# Start server
+python -m uvicorn app.main:app --host 0.0.0.0 --port 8000
+
+# Run tests
+python -m pytest tests/ -v --timeout=60
+
+# Run specific test file
+python -m pytest tests/test_message_converter.py -v
+
+# Health check
+curl -s http://localhost:8000/api/config/health
+```
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `config.yaml` | Providers, models, API keys (hot-reloaded) |
+| `app/core/chain_router.py` | Routes requests with fallback |
+| `app/adapters/*.py` | Provider-specific adapters |
+| `app/utils/message_converter.py` | OpenAI ↔ Anthropic conversion |
+| `CLAUDE.md` | Full architecture documentation |

modelswitch-0.1.1/CLAUDE.md

@@ -0,0 +1,123 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+ModelSwitch is an LLM gateway proxy that exposes OpenAI-compatible and Anthropic-compatible APIs. It routes requests to upstream providers (DashScope, GLM Plan, OpenAI, Anthropic) with automatic fallback chains, per-key rate limiting, usage tracking, tool use conversion, and a web management UI.
+
+## Running the Server
+
+```bash
+# Start server (default port 8000)
+python -m uvicorn app.main:app --host 0.0.0.0 --port 8000
+
+# Docker
+docker-compose up --build
+```
+
+The server takes ~10s to start due to provider connection warmup. Health check: `GET /api/config/health`.
+
+## Key Commands
+
+```bash
+# Install dependencies (litellm is pinned to 1.82.6 — do not upgrade due to supply chain attack on 1.82.7/1.82.8)
+pip install -r requirements.txt
+
+# Run all tests (pytest-asyncio with strict mode)
+python -m pytest tests/ -v
+
+# Run a single test file
+python -m pytest tests/test_message_converter.py -v
+
+# Run with coverage
+python -m pytest tests/ --cov=app --cov-report=term-missing
+
+# Test an endpoint
+curl -s http://localhost:8000/v1/chat/completions \
+  -H "Authorization: Bearer sk-gateway-admin" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"glm5","messages":[{"role":"user","content":"hi"}],"max_tokens":100}'
+```
+
+## Architecture
+
+### Request Flow
+
+```
+Client → GatewayMiddleware (auth/rate-limit) → Route Handler → ChainRouter → LiteLLMAdapter → Upstream Provider
+```
+
+### Config-Driven Design (`config.yaml`)
+
+- **Providers** are top-level connection definitions (name, base_url, api_key, protocol type). API keys support `${ENV_VAR}` substitution.
+- **Models** reference providers by name with priority/timeout. Two modes:
+  - `chain`: tries adapters by priority, falls back on failure (with circuit breaker + 1 retry per adapter)
+  - `adapter`: direct call to a single provider, no fallback
+- **API Keys** are configured in the YAML, not a database. They have a `roles` field (`admin` | `user`, default `["user"]`). Admin keys can manage providers/models/keys via management API.
+
+### Core Layer (`app/core/`)
+
+| Component | Purpose |
+|---|---|
+| `chain_router.py` | Routes requests to adapters. Chain mode does first-chunk probe for streaming fallback. Uses `_adapter_info` dict to pass adapter name/latency/usage from stream generator to caller. |
+| `circuit_breaker.py` | Per-provider circuit breaker: 5 failures → 30s open → half-open probe |
+| `middleware.py` | Pure ASGI middleware (NOT BaseHTTPMiddleware — that causes infinite recursion). Three-tier auth: public paths (no auth), API key auth (any valid key), admin auth (role check). Plus rate limiting and CORS. |
+| `config_watcher.py` | watchdog-based hot reload with 2s debounce |
+
+### Middleware Constraint
+
+`GatewayMiddleware` is a **pure ASGI middleware** (raw `__call__(scope, receive, send)`), registered via `app.add_middleware()`. It does **NOT** extend `BaseHTTPMiddleware` and must not be wrapped with `@app.middleware("http")`. Mixing these two patterns causes infinite recursion because `call_next` re-triggers the decorator wrapper.
+
+### Adapter Layer (`app/adapters/`)
+
+`LiteLLMAdapter` wraps `litellm.acompletion()`. Model names are prefixed with the provider type: `openai/glm-5`, `anthropic/claude-sonnet-4-20250514`. Returns `AdapterResponse` dataclass with `body` (non-stream) or `stream` (async generator).
+
+Streaming usage: OpenAI adapter sets `stream_options: {include_usage: true}` to capture token counts from the final chunk. Usage is stored in `AdapterResponse.usage` after stream completes.
+
+### Protocol Conversion (`app/utils/message_converter.py`)
+
+Bidirectional conversion between Anthropic and OpenAI formats:
+- `anthropic_to_openai_messages()`: System, messages, tools/tool_choice, tool_use blocks → tool_calls, tool_result blocks → role:"tool"
+- `openai_stream_to_anthropic()`: OpenAI SSE chunks → Anthropic SSE events, including tool_calls deltas → input_json_delta events
+
+### API Routes (`app/api/`)
+
+- `openai_routes.py`: `GET /openai/models` (also `/v1/models`), `POST /openai/chat/completions` (also `/v1/chat/completions`)
+- `anthropic_routes.py`: `POST /anthropic/messages` (also `/v1/messages`) — converts to/from OpenAI internally, including full tool_use conversion
+- `config_routes.py`, `api_key_routes.py`: CRUD for providers/models/keys, writes back to `config.yaml`
+- `usage_routes.py`: Aggregated stats with `group_by` (provider/model/api_key) and drill-down
+- `log_routes.py`: Queries in-memory ring buffer (max 1000 entries)
+- `conversation_routes.py`: Queries conversation log files (multi-file discovery with metadata streaming, on-demand full record fetch)
+
+### Protocol Conversion — Full details in `app/utils/message_converter.py`
+
+- **Request**: `anthropic_to_openai_messages()` — converts system, messages, tools, tool_choice, tool_use/tool_result blocks
+- **Non-stream response**: `_convert_openai_to_anthropic_response()` in `anthropic_routes.py` — converts tool_calls to tool_use content blocks
+- **Stream response**: `openai_stream_to_anthropic()` — handles text deltas and tool_calls deltas with multi-tool index tracking
+
+### Request Tracking (`app/utils/tracking.py`)
+
+Centralized `track_request()` called from both route files after chain_router returns. Records usage stats (via `usage_tracker.record()`) and debug logs (via `add_log_to_buffer()`).
+
+For streaming: `chain_router._execute_chat_stream` populates `_adapter_info` dict with adapter name, latency, and usage. Route handlers read this in the `finally` block after stream consumption and set on `result` before tracking.
+
+### Persistence
+
+- Config: `config.yaml` (hot-reloaded via watchdog)
+- Usage stats: SQLite at `data/usage.db`, batch-flushed every 10s
+- Logs: Rotating file at `logs/gateway.log` + in-memory deque buffer
+
+### Frontend
+
+Single-page app in `web/` (HTML/CSS/JS, no build step). 7 tabs: Providers, Models, API Keys, Queue Monitor, Usage Stats, Debug Logs, Conversations. Login modal requires admin API key. Token persisted in localStorage. Served at `/` and `/web/`.
+
+## Key Patterns
+
+- Route handlers access shared state via `request.app.state` (chain_router, usage_tracker, api_key_service, config)
+- The middleware injects auth info into `scope["state"]` (api_key, api_key_name, api_key_config, api_key_roles), which maps to `request.state` in route handlers
+- Public paths (no auth required): `/`, `/health`, `/metrics`, `/docs`, `/web/*`, `/static/*`
+- Auth-required paths (any valid key): `/v1/*`, `/openai/*`, `/anthropic/*`, `/api/usage`, `/api/logs`, `/api/conversations` — accepts `Authorization: Bearer <key>`, `x-api-key: <key>`, or bare `sk-*` header
+- Admin-required paths (`roles: ["admin"]`): `/api/config/*`, `/api/keys/*`
+- Error format adapts to route: OpenAI-style `{"error": {...}}` for `/openai/*` and `/v1/chat/completions`, Anthropic-style `{"type": "error", "error": {...}}` for `/anthropic/*` and `/v1/messages`
+- Anthropic routes skip conversion when the first adapter is an Anthropic provider (passthrough mode)