claude-flow-novice 1.2.1 → 1.3.1

package/examples/litellm/README.md

@@ -0,0 +1,347 @@
+# LiteLLM Multi-Tenant Gateway for Claude Code
+
+A production-ready, multi-tenant LiteLLM proxy that enables Claude Code to seamlessly route requests across multiple LLM providers (OpenAI, Azure, OpenRouter, Bedrock, Ollama, etc.) with enterprise features.
+
+## 🚀 Quick Start
+
+### 1. Prerequisites
+
+- Docker 20.10+
+- Docker Compose 2.0+
+- 8GB RAM minimum
+- Valid API keys for desired LLM providers
+
+### 2. Installation
+
+```bash
+# Clone the repository
+git clone https://github.com/ruvnet/claude-flow.git
+cd claude-flow/examples/litellm
+
+# Copy environment template
+cp .env.example .env
+
+# Edit .env with your API keys
+nano .env
+
+# Deploy the stack
+./scripts/deploy.sh start
+```
+
+### 3. Configure Claude Code
+
+```bash
+# Set environment variables
+export ANTHROPIC_BASE_URL=http://localhost:4000
+export ANTHROPIC_AUTH_TOKEN=<your-litellm-master-key>
+
+# Use Claude Code with different models
+claude --model codex-mini "Write a Python function"
+claude --model o3-pro "Explain quantum computing"
+claude --model deepseek-coder "Refactor this code"
+```
+
+## 📊 Architecture
+
+```
+┌─────────────┐     ┌──────────────┐     ┌─────────────┐
+│ Claude Code │────▶│ Load Balancer│────▶│  LiteLLM    │
+└─────────────┘     └──────────────┘     │  Proxies    │
+                                         └─────┬───────┘
+                                               │
+                    ┌──────────────────────────┼──────────────────────────┐
+                    │                          │                          │
+              ┌─────▼─────┐          ┌────────▼────────┐        ┌────────▼────────┐
+              │  OpenAI   │          │   OpenRouter   │        │     Azure      │
+              └───────────┘          └────────────────┘        └────────────────┘
+```
+
+## 🎯 Features
+
+### Multi-Provider Support
+- **OpenAI**: GPT-4, GPT-4o, o3 models
+- **OpenRouter**: Qwen3-Coder, DeepSeek, 100+ models
+- **Azure OpenAI**: Enterprise Azure deployments
+- **Amazon Bedrock**: Claude models via AWS
+- **Ollama**: Local models (CodeLlama, Mixtral)
+- **Anthropic**: Direct Claude access
+
+### Enterprise Features
+- 🔐 **Multi-Tenancy**: Isolated configurations per team
+- 💰 **Cost Tracking**: Real-time usage and budget alerts
+- 📊 **Monitoring**: Prometheus + Grafana dashboards
+- 🔄 **High Availability**: Load-balanced proxy instances
+- 🛡️ **Security**: API key management, rate limiting
+- 📝 **Audit Logging**: Complete request/response tracking
+
+## 🔧 Configuration
+
+### Model Aliases
+
+Edit `config/config.yaml` to customize model routing:
+
+```yaml
+model_list:
+  - model_name: "fast-code"
+    litellm_params:
+      model: "openrouter/qwen/qwen-3-coder"
+      max_tokens: 8192
+      temperature: 0.2
+      
+  - model_name: "reasoning"
+    litellm_params:
+      model: "openai/o3-pro"
+      max_tokens: 4096
+      temperature: 0.7
+```
+
+### Tenant Management
+
+Create and manage tenants:
+
+```bash
+# Create a new tenant
+./scripts/manage-tenants.sh create engineering
+
+# List all tenants
+./scripts/manage-tenants.sh list
+
+# Update tenant budget
+./scripts/manage-tenants.sh update engineering budget 200
+
+# View usage statistics
+./scripts/manage-tenants.sh usage engineering
+```
+
+### Fallback Chains
+
+Configure automatic fallbacks in `config/config.yaml`:
+
+```yaml
+fallback_models:
+  code_chain:
+    - codex-mini       # Primary (fast, cheap)
+    - deepseek-coder   # Secondary
+    - local-codellama  # Tertiary (free, local)
+```
+
+## 📈 Monitoring
+
+### Access Dashboards
+
+- **Grafana**: http://localhost:3000
+- **Prometheus**: http://localhost:9090
+- **PgAdmin**: http://localhost:5050
+
+### Key Metrics
+
+- Request latency (p50, p95, p99)
+- Token usage per model/tenant
+- Cost tracking and projections
+- Error rates and fallback triggers
+- Model availability status
+
+## 🛠️ Operations
+
+### Service Management
+
+```bash
+# Start services
+./scripts/deploy.sh start
+
+# Stop services
+./scripts/deploy.sh stop
+
+# View logs
+./scripts/deploy.sh logs
+
+# Check status
+./scripts/deploy.sh status
+
+# Clean everything
+./scripts/deploy.sh clean
+```
+
+### Scaling
+
+Adjust replica count in `docker-compose.yml`:
+
+```yaml
+services:
+  litellm:
+    deploy:
+      replicas: 5  # Increase for higher load
+```
+
+### Backup & Restore
+
+```bash
+# Backup database
+docker-compose exec postgres pg_dump -U litellm > backup.sql
+
+# Restore database
+docker-compose exec -T postgres psql -U litellm < backup.sql
+```
+
+## 🔐 Security
+
+### API Key Rotation
+
+```bash
+# Generate new master key
+export NEW_KEY=$(openssl rand -hex 32)
+echo "LITELLM_MASTER_KEY=sk-litellm-$NEW_KEY" >> .env
+
+# Restart services
+docker-compose restart
+```
+
+### Network Security
+
+- All services run in isolated Docker network
+- PostgreSQL and Redis not exposed externally
+- SSL/TLS support for production deployments
+- Rate limiting per tenant
+
+## 📚 Examples
+
+### Basic Usage
+
+```bash
+# Fast code generation
+claude --model codex-mini "Write a REST API endpoint"
+
+# Complex reasoning
+claude --model o3-pro "Design a distributed system"
+
+# Local processing (no cloud)
+claude --model local-codellama "Refactor this function"
+```
+
+### Cost-Optimized Routing
+
+```bash
+# Use cheapest model for simple tasks
+export ANTHROPIC_MODEL=claude-3-haiku
+claude "Format this JSON"
+
+# Use powerful model for complex tasks
+export ANTHROPIC_MODEL=o3-pro
+claude "Solve this algorithm problem"
+```
+
+### Multi-Tenant Usage
+
+```bash
+# Engineering team
+export ANTHROPIC_AUTH_TOKEN=$TENANT_ENGINEERING_KEY
+claude --model codex-mini "Build feature"
+
+# Research team
+export ANTHROPIC_AUTH_TOKEN=$TENANT_RESEARCH_KEY
+claude --model o3-pro "Analyze dataset"
+```
+
+## 🐛 Troubleshooting
+
+### Common Issues
+
+1. **Connection Refused**
+   ```bash
+   # Check if services are running
+   docker-compose ps
+   
+   # Check LiteLLM logs
+   docker-compose logs litellm-1
+   ```
+
+2. **Authentication Failed**
+   ```bash
+   # Verify master key in .env
+   grep LITELLM_MASTER_KEY .env
+   
+   # Test connection
+   curl http://localhost:4000/health
+   ```
+
+3. **Model Not Found**
+   ```bash
+   # Check available models
+   curl -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
+        http://localhost:4000/models
+   ```
+
+### Debug Mode
+
+Enable debug logging:
+
+```yaml
+# In docker-compose.yml
+environment:
+  - LITELLM_LOG_LEVEL=DEBUG
+  - DEBUG=true
+```
+
+## 📖 Advanced Topics
+
+### Custom Model Providers
+
+Add custom providers in `config/config.yaml`:
+
+```yaml
+model_list:
+  - model_name: "custom-model"
+    litellm_params:
+      model: "custom_provider/model_name"
+      api_base: "https://api.custom.com/v1"
+      api_key: ${CUSTOM_API_KEY}
+      custom_llm_provider: "custom"
+```
+
+### Performance Tuning
+
+```yaml
+# Optimize for high throughput
+router_settings:
+  max_parallel_requests: 100
+  request_timeout: 30
+  enable_caching: true
+  cache_ttl: 3600
+```
+
+### Cost Optimization
+
+```yaml
+# Implement cost controls
+cost_tracking:
+  alert_thresholds:
+    - threshold: 80
+      action: notify
+    - threshold: 100
+      action: block
+```
+
+## 🤝 Contributing
+
+See [CONTRIBUTING.md](../../CONTRIBUTING.md) for guidelines.
+
+## 📄 License
+
+MIT License - see [LICENSE](../../LICENSE) for details.
+
+## 🔗 Resources
+
+- [LiteLLM Documentation](https://docs.litellm.ai)
+- [Claude Code Guide](https://docs.anthropic.com/en/docs/claude-code)
+- [OpenRouter Models](https://openrouter.ai/models)
+- [Docker Compose Reference](https://docs.docker.com/compose/)
+
+## 💬 Support
+
+- GitHub Issues: [Report bugs](https://github.com/ruvnet/claude-flow/issues)
+- Discussions: [Ask questions](https://github.com/ruvnet/claude-flow/discussions)
+- Wiki: [Documentation](https://github.com/ruvnet/claude-flow/wiki)
+
+---
+
+Built with ❤️ by the Claude Flow team

package/examples/litellm/config/basic-config.yaml

@@ -0,0 +1,25 @@
+model_list:
+  # Test model using litellm's test provider
+  - model_name: "test-model"
+    litellm_params:
+      model: "openai/gpt-3.5-turbo"
+      api_key: "sk-test-key"
+      mock_response: "This is a test response from LiteLLM proxy"
+      
+  # Echo model for testing
+  - model_name: "echo-model"
+    litellm_params:
+      model: "openai/gpt-3.5-turbo"
+      api_key: "test-key"
+      mock_response: "Echo test successful"
+
+general_settings:
+  master_key: "sk-1234567890"
+  request_timeout: 60
+  drop_params: false
+  stream: true
+  
+litellm_settings:
+  drop_params: false
+  set_verbose: true
+  json_logs: false

package/examples/litellm/config/config.yaml

@@ -0,0 +1,294 @@
+# LiteLLM Multi-Tenant Gateway Configuration
+# This config enables Claude Code to route to multiple LLM providers
+
+general_settings:
+  master_key: ${LITELLM_MASTER_KEY}
+  database_url: ${DATABASE_URL}
+  redis_url: ${REDIS_URL}
+  request_timeout: 600
+  max_retries: 3
+  fallback_enabled: true
+  cache_enabled: true
+  cache_ttl: 3600
+  telemetry: false
+  custom_auth: true
+  enable_jwt_auth: true
+  multi_tenant: true
+
+# Router settings for intelligent routing
+router_settings:
+  routing_strategy: simple-shuffle  # Avoid latency-based for Anthropic compatibility
+  enable_fallbacks: true
+  retry_policy:
+    max_retries: 3
+    retry_delay: 1
+    exponential_backoff: true
+  rate_limit:
+    enabled: true
+    requests_per_minute: 100
+    requests_per_day: 10000
+
+# Model list with aliases for Claude Code
+model_list:
+  # OpenAI Models
+  - model_name: "codex-mini"
+    litellm_params:
+      model: "openai/gpt-4o-mini"
+      api_key: ${OPENAI_API_KEY}
+      api_base: https://api.openai.com/v1
+      max_tokens: 4096
+      temperature: 0.2
+      timeout: 30
+      stream: true
+    model_info:
+      mode: code
+      cost_per_token: 0.00002
+      
+  - model_name: "o3-pro"
+    litellm_params:
+      model: "openai/o3-2025"
+      api_key: ${OPENAI_API_KEY}
+      api_base: https://api.openai.com/v1
+      max_tokens: 8192
+      temperature: 0.7
+      timeout: 60
+      stream: true
+    model_info:
+      mode: reasoning
+      cost_per_token: 0.00015
+
+  # OpenRouter Models (Qwen3-Coder)
+  - model_name: "claude-3-5-sonnet"
+    litellm_params:
+      model: "openrouter/qwen/qwen-3-coder:free"
+      api_key: ${OPENROUTER_API_KEY}
+      api_base: https://openrouter.ai/api/v1
+      max_tokens: 65536
+      temperature: 0.7
+      timeout: 120
+      stream: true
+      extra_headers:
+        HTTP-Referer: "https://github.com/ruvnet/claude-flow"
+        X-Title: "Claude Code Gateway"
+    model_info:
+      mode: code
+      cost_per_token: 0.00001
+
+  - model_name: "deepseek-coder"
+    litellm_params:
+      model: "openrouter/deepseek/deepseek-coder"
+      api_key: ${OPENROUTER_API_KEY}
+      api_base: https://openrouter.ai/api/v1
+      max_tokens: 32768
+      temperature: 0.5
+      timeout: 90
+      stream: true
+    model_info:
+      mode: code
+      cost_per_token: 0.000005
+
+  # Azure OpenAI
+  - model_name: "azure-gpt4"
+    litellm_params:
+      model: "azure/gpt-4-turbo-2024-08-01"
+      api_key: ${AZURE_API_KEY}
+      api_base: ${AZURE_API_BASE}
+      api_version: "2024-08-01-preview"
+      max_tokens: 8192
+      temperature: 0.7
+      timeout: 60
+      stream: true
+    model_info:
+      mode: general
+      cost_per_token: 0.00012
+
+  # Amazon Bedrock Claude
+  - model_name: "bedrock-claude"
+    litellm_params:
+      model: "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"
+      aws_access_key_id: ${AWS_ACCESS_KEY_ID}
+      aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
+      aws_region_name: ${AWS_REGION}
+      max_tokens: 4096
+      temperature: 0.7
+      timeout: 60
+      stream: true
+    model_info:
+      mode: general
+      cost_per_token: 0.00008
+
+  # Local Ollama Models
+  - model_name: "local-codellama"
+    litellm_params:
+      model: "ollama/codellama:34b"
+      api_base: http://ollama:11434
+      max_tokens: 8192
+      temperature: 0.3
+      timeout: 120
+      stream: true
+    model_info:
+      mode: code
+      cost_per_token: 0
+
+  - model_name: "local-mixtral"
+    litellm_params:
+      model: "ollama/mixtral:8x7b"
+      api_base: http://ollama:11434
+      max_tokens: 4096
+      temperature: 0.7
+      timeout: 90
+      stream: true
+    model_info:
+      mode: general
+      cost_per_token: 0
+
+  # Anthropic Models (Direct)
+  - model_name: "claude-3-opus"
+    litellm_params:
+      model: "anthropic/claude-3-opus-20240229"
+      api_key: ${ANTHROPIC_API_KEY}
+      max_tokens: 4096
+      temperature: 0.7
+      timeout: 60
+      stream: true
+    model_info:
+      mode: reasoning
+      cost_per_token: 0.00015
+
+  - model_name: "claude-3-haiku"
+    litellm_params:
+      model: "anthropic/claude-3-haiku-20240307"
+      api_key: ${ANTHROPIC_API_KEY}
+      max_tokens: 4096
+      temperature: 0.5
+      timeout: 30
+      stream: true
+    model_info:
+      mode: fast
+      cost_per_token: 0.000025
+
+# Fallback chains for reliability
+fallback_models:
+  code_chain:
+    - codex-mini
+    - deepseek-coder
+    - local-codellama
+    - claude-3-haiku
+    
+  reasoning_chain:
+    - o3-pro
+    - azure-gpt4
+    - bedrock-claude
+    - claude-3-opus
+    
+  fast_chain:
+    - claude-3-haiku
+    - codex-mini
+    - local-mixtral
+
+# Tenant-specific configurations
+tenant_configs:
+  engineering:
+    allowed_models:
+      - codex-mini
+      - deepseek-coder
+      - local-codellama
+    budget:
+      daily_limit: 100
+      monthly_limit: 2000
+    priority: high
+    
+  research:
+    allowed_models:
+      - o3-pro
+      - azure-gpt4
+      - claude-3-opus
+    budget:
+      daily_limit: 200
+      monthly_limit: 5000
+    priority: medium
+    
+  operations:
+    allowed_models:
+      - claude-3-haiku
+      - local-mixtral
+      - codex-mini
+    budget:
+      daily_limit: 50
+      monthly_limit: 1000
+    priority: low
+
+# Cost tracking and optimization
+cost_tracking:
+  enabled: true
+  currency: USD
+  alert_thresholds:
+    - threshold: 80
+      action: notify
+    - threshold: 95
+      action: throttle
+    - threshold: 100
+      action: block
+  
+# Logging configuration
+logging:
+  level: INFO
+  format: json
+  destinations:
+    - type: file
+      path: /app/logs/litellm.log
+      rotation: daily
+      retention: 30
+    - type: loki
+      url: http://loki:3100
+      labels:
+        app: litellm
+        env: production
+
+# Monitoring and metrics
+monitoring:
+  prometheus:
+    enabled: true
+    port: 9090
+    path: /metrics
+  custom_metrics:
+    - name: request_duration
+      type: histogram
+      buckets: [0.1, 0.5, 1, 2, 5, 10]
+    - name: model_usage
+      type: counter
+      labels: [model, tenant]
+    - name: cost_per_tenant
+      type: gauge
+      labels: [tenant, model]
+
+# Security settings
+security:
+  api_key_header: X-API-Key
+  jwt_secret: ${JWT_SECRET}
+  allowed_ips:
+    - 10.0.0.0/8
+    - 172.16.0.0/12
+    - 192.168.0.0/16
+  rate_limiting:
+    enabled: true
+    strategy: sliding_window
+    default_limits:
+      - 100 per minute
+      - 1000 per hour
+      - 10000 per day
+
+# Health check configuration
+health_check:
+  enabled: true
+  path: /health
+  checks:
+    - type: database
+      critical: true
+    - type: redis
+      critical: false
+    - type: model_availability
+      critical: true
+      models:
+        - codex-mini
+        - claude-3-haiku