agentic-flow 1.9.0 → 1.9.1

package/agentic-flow/docs/guides/ALTERNATIVE_LLM_MODELS.md

@@ -1,524 +0,0 @@
-# Alternative LLM Models & Optimization Guide
-
-## Agentic Flow - Multi-Model Support & Performance Optimization
-
-Created by: @ruvnet
-Version: 1.0.0
-Date: 2025-10-04
-
----
-
-## Table of Contents
-
-1. [Overview](#overview)
-2. [Supported Providers](#supported-providers)
-3. [OpenRouter Integration](#openrouter-integration)
-4. [ONNX Runtime Support](#onnx-runtime-support)
-5. [Model Routing & Selection](#model-routing--selection)
-6. [Performance Optimization](#performance-optimization)
-7. [Cost Optimization](#cost-optimization)
-8. [Testing & Validation](#testing--validation)
-
----
-
-## Overview
-
-Agentic Flow supports multiple LLM providers through a sophisticated routing system, allowing you to:
-
-- ✅ Use alternative models beyond Claude (GPT-4, Gemini, Llama, Mistral, etc.)
-- ✅ Run local models with ONNX Runtime
-- ✅ Implement intelligent routing based on task complexity
-- ✅ Optimize costs by using cheaper models for simple tasks
-- ✅ Achieve sub-linear performance with local inference
-
----
-
-## Supported Providers
-
-### 1. **Anthropic Claude** (Default)
-- Models: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku
-- Best for: Complex reasoning, coding, long context
-- Configuration: `ANTHROPIC_API_KEY`
-
-### 2. **OpenRouter** (100+ Models)
-- Access to GPT-4, Gemini, Llama 3, Mistral, and more
-- Unified API for multiple providers
-- Pay-per-use pricing
-- Configuration: `OPENROUTER_API_KEY`
-
-### 3. **ONNX Runtime** (Local Inference)
-- Run quantized models locally
-- Zero API costs
-- Privacy-preserving
-- Sub-second inference
-- Models: Phi-3, Phi-4, optimized LLMs
-
----
-
-## OpenRouter Integration
-
-### Configuration
-
-1. **Get API Key:**
-```bash
-# Sign up at https://openrouter.ai
-# Get your API key from dashboard
-```
-
-2. **Add to `.env`:**
-```bash
-OPENROUTER_API_KEY=sk-or-v1-xxxxxxxxxxxxx
-OPENROUTER_SITE_URL=https://github.com/ruvnet/agentic-flow
-OPENROUTER_APP_NAME=agentic-flow
-```
-
-3. **Configure `router.config.json`:**
-```json
-{
-  "providers": {
-    "openrouter": {
-      "apiKey": "${OPENROUTER_API_KEY}",
-      "baseURL": "https://openrouter.ai/api/v1",
-      "models": {
-        "fast": "meta-llama/llama-3.1-8b-instruct",
-        "balanced": "anthropic/claude-3-haiku",
-        "powerful": "openai/gpt-4-turbo",
-        "coding": "deepseek/deepseek-coder-33b-instruct"
-      },
-      "defaultModel": "balanced"
-    }
-  }
-}
-```
-
-### Recommended Models by Use Case
-
-#### **Coding Tasks:**
-```json
-{
-  "model": "deepseek/deepseek-coder-33b-instruct",
-  "description": "Specialized for code generation",
-  "cost": "$0.14/1M tokens",
-  "speed": "Fast"
-}
-```
-
-#### **Fast Simple Tasks:**
-```json
-{
-  "model": "meta-llama/llama-3.1-8b-instruct",
-  "description": "Quick responses, low cost",
-  "cost": "$0.06/1M tokens",
-  "speed": "Very Fast"
-}
-```
-
-#### **Complex Reasoning:**
-```json
-{
-  "model": "openai/gpt-4-turbo",
-  "description": "Best for complex multi-step tasks",
-  "cost": "$10/1M tokens",
-  "speed": "Moderate"
-}
-```
-
-#### **Long Context:**
-```json
-{
-  "model": "google/gemini-pro-1.5",
-  "description": "2M token context window",
-  "cost": "$1.25/1M tokens",
-  "speed": "Fast"
-}
-```
-
-### Usage Examples
-
-```typescript
-import { ModelRouter } from './router/router.js';
-
-const router = new ModelRouter();
-
-// Use OpenRouter for coding task
-const response = await router.chat({
-  provider: 'openrouter',
-  model: 'deepseek/deepseek-coder-33b-instruct',
-  messages: [{
-    role: 'user',
-    content: 'Create a Python REST API with Flask'
-  }]
-});
-```
-
----
-
-## ONNX Runtime Support
-
-### Benefits
-
-- **🚀 Speed:** Sub-second inference (10-100x faster than API calls)
-- **💰 Cost:** Zero API fees
-- **🔒 Privacy:** All processing stays local
-- **📴 Offline:** Works without internet
-- **⚡ Scalable:** No rate limits
-
-### Supported Models
-
-#### **Phi-4 (Microsoft)**
-```json
-{
-  "model": "phi-4-onnx",
-  "size": "14B parameters",
-  "quantization": "INT4",
-  "memory": "~8GB RAM",
-  "speed": "~50 tokens/sec",
-  "quality": "GPT-3.5 level"
-}
-```
-
-#### **Phi-3 Mini**
-```json
-{
-  "model": "phi-3-mini-onnx",
-  "size": "3.8B parameters",
-  "quantization": "INT4",
-  "memory": "~2GB RAM",
-  "speed": "~100 tokens/sec",
-  "quality": "Good for simple tasks"
-}
-```
-
-### Configuration
-
-```json
-{
-  "providers": {
-    "onnx": {
-      "enabled": true,
-      "modelPath": "./models/phi-4-instruct-int4.onnx",
-      "executionProvider": "cpu",
-      "threads": 4,
-      "cache": {
-        "enabled": true,
-        "maxSize": "1GB"
-      }
-    }
-  }
-}
-```
-
-### Installation
-
-```bash
-# Install ONNX Runtime
-npm install onnxruntime-node
-
-# Download quantized model
-mkdir -p models
-wget https://huggingface.co/microsoft/phi-4/resolve/main/onnx/phi-4-instruct-int4.onnx \\
-  -O models/phi-4-instruct-int4.onnx
-```
-
-### Usage
-
-```typescript
-const router = new ModelRouter();
-
-// Use local ONNX model
-const response = await router.chat({
-  provider: 'onnx',
-  messages: [{
-    role: 'user',
-    content: 'Write a hello world in Python'
-  }]
-});
-```
-
----
-
-## Model Routing & Selection
-
-### Intelligent Task-Based Routing
-
-```json
-{
-  "routing": {
-    "rules": [
-      {
-        "condition": "token_count < 500",
-        "provider": "onnx",
-        "model": "phi-3-mini",
-        "reason": "Fast local inference for simple tasks"
-      },
-      {
-        "condition": "task_type == 'coding'",
-        "provider": "openrouter",
-        "model": "deepseek/deepseek-coder-33b-instruct",
-        "reason": "Specialized coding model"
-      },
-      {
-        "condition": "complexity == 'high'",
-        "provider": "anthropic",
-        "model": "claude-3-opus",
-        "reason": "Complex reasoning required"
-      },
-      {
-        "condition": "default",
-        "provider": "openrouter",
-        "model": "meta-llama/llama-3.1-8b-instruct",
-        "reason": "Balanced cost/performance"
-      }
-    ]
-  }
-}
-```
-
-### Fallback Strategy
-
-```json
-{
-  "fallback": {
-    "enabled": true,
-    "chain": [
-      "onnx",
-      "openrouter",
-      "anthropic"
-    ],
-    "retryAttempts": 3,
-    "backoffMs": 1000
-  }
-}
-```
-
----
-
-## Performance Optimization
-
-### 1. **Response Time Optimization**
-
-| Provider | Model | Avg Response Time | Use Case |
-|----------|-------|------------------|----------|
-| ONNX | Phi-3 Mini | 0.5s | Simple queries |
-| ONNX | Phi-4 | 1.2s | Medium complexity |
-| OpenRouter | Llama 3.1 8B | 2.5s | Balanced tasks |
-| OpenRouter | DeepSeek Coder | 3.5s | Code generation |
-| Anthropic | Claude 3 Haiku | 2.0s | Fast reasoning |
-| Anthropic | Claude 3.5 Sonnet | 4.0s | Best quality |
-
-### 2. **Memory Optimization**
-
-```json
-{
-  "optimization": {
-    "onnx": {
-      "quantization": "INT4",
-      "memoryLimit": "8GB",
-      "batchSize": 1
-    },
-    "caching": {
-      "enabled": true,
-      "strategy": "LRU",
-      "maxEntries": 1000
-    }
-  }
-}
-```
-
-### 3. **Parallel Processing**
-
-```typescript
-// Process multiple tasks in parallel with different models
-const tasks = [
-  { task: 'simple', model: 'onnx/phi-3-mini' },
-  { task: 'coding', model: 'openrouter/deepseek-coder' },
-  { task: 'complex', model: 'anthropic/claude-3-opus' }
-];
-
-const results = await Promise.all(
-  tasks.map(t => router.chat({ provider: t.model.split('/')[0], ... }))
-);
-```
-
----
-
-## Cost Optimization
-
-### Monthly Cost Comparison
-
-**Scenario: 1M tokens/month**
-
-| Strategy | Provider Mix | Monthly Cost | Savings |
-|----------|-------------|--------------|---------|
-| All Claude Opus | 100% Anthropic | $15.00 | - |
-| Smart Routing | 50% ONNX + 30% Llama + 20% Claude | $2.50 | 83% |
-| Budget Mode | 80% ONNX + 20% Llama | $0.60 | 96% |
-| Hybrid | 40% ONNX + 40% OpenRouter + 20% Claude | $4.00 | 73% |
-
-### Cost-Optimized Configuration
-
-```json
-{
-  "costOptimization": {
-    "enabled": true,
-    "maxCostPerRequest": 0.01,
-    "preferredProviders": ["onnx", "openrouter", "anthropic"],
-    "budgetLimits": {
-      "daily": 5.00,
-      "monthly": 100.00
-    }
-  }
-}
-```
-
----
-
-## Testing & Validation
-
-### Test Suite
-
-```bash
-# Test OpenRouter integration
-npm run test:router -- --provider=openrouter
-
-# Test ONNX Runtime
-npm run test:onnx
-
-# Benchmark all providers
-npm run benchmark:providers
-```
-
-### Validation Results
-
-**✅ OpenRouter Models Tested:**
-- `meta-llama/llama-3.1-8b-instruct` - Working, fast
-- `deepseek/deepseek-coder-33b-instruct` - Working, excellent for code
-- `google/gemini-pro` - Working, good balance
-- `openai/gpt-4-turbo` - Working, best quality
-
-**✅ ONNX Models Tested:**
-- `phi-3-mini-int4` - Working, 100 tok/s
-- `phi-4-instruct-int4` - Working, 50 tok/s
-
-**✅ Automated Coding Test:**
-- Generated Python hello.py - ✅ Success
-- Generated Flask REST API (3 files) - ✅ Success
-- Code quality - ✅ Production-ready
-
----
-
-## Quick Start Examples
-
-### Example 1: Use OpenRouter for Cost Savings
-
-```bash
-# Set up OpenRouter
-export OPENROUTER_API_KEY=sk-or-v1-xxxxx
-
-# Run with Llama 3.1 (cheap and fast)
-npx agentic-flow --agent coder \\
-  --model openrouter/meta-llama/llama-3.1-8b-instruct \\
-  --task "Create a Python calculator"
-```
-
-### Example 2: Use Local ONNX for Privacy
-
-```bash
-# Run with local Phi-4 (no API needed)
-npx agentic-flow --agent coder \\
-  --model onnx/phi-4 \\
-  --task "Generate unit tests"
-```
-
-### Example 3: Smart Routing
-
-```bash
-# Let the router choose best model
-npx agentic-flow --agent coder \\
-  --auto-route \\
-  --task "Build a complex distributed system"
-# → Routes to Claude for complexity
-```
-
----
-
-## Configuration Files
-
-### Complete Example: `router.config.json`
-
-```json
-{
-  "providers": {
-    "anthropic": {
-      "apiKey": "${ANTHROPIC_API_KEY}",
-      "models": {
-        "fast": "claude-3-haiku-20240307",
-        "balanced": "claude-3-5-sonnet-20241022",
-        "powerful": "claude-3-opus-20240229"
-      },
-      "defaultModel": "balanced"
-    },
-    "openrouter": {
-      "apiKey": "${OPENROUTER_API_KEY}",
-      "baseURL": "https://openrouter.ai/api/v1",
-      "models": {
-        "fast": "meta-llama/llama-3.1-8b-instruct",
-        "coding": "deepseek/deepseek-coder-33b-instruct",
-        "balanced": "google/gemini-pro-1.5",
-        "powerful": "openai/gpt-4-turbo"
-      },
-      "defaultModel": "fast"
-    },
-    "onnx": {
-      "enabled": true,
-      "modelPath": "./models/phi-4-instruct-int4.onnx",
-      "executionProvider": "cpu",
-      "threads": 4
-    }
-  },
-  "routing": {
-    "strategy": "cost-optimized",
-    "fallbackChain": ["onnx", "openrouter", "anthropic"]
-  }
-}
-```
-
----
-
-## Optimization Recommendations
-
-### For Development
-- Use ONNX for rapid iteration (free, fast)
-- Use OpenRouter Llama for testing (cheap)
-
-### For Production
-- Use intelligent routing
-- Cache common queries
-- Monitor costs with budgets
-
-### For Scale
-- Deploy ONNX models on edge
-- Use OpenRouter for burst capacity
-- Reserve Claude for critical tasks
-
----
-
-## Conclusion
-
-Agentic Flow's multi-model support enables:
-
-- **96% cost savings** with smart routing
-- **10-100x faster** responses with ONNX
-- **100+ model choices** via OpenRouter
-- **Production-ready** automated coding
-
-**Next Steps:**
-1. Add OpenRouter key to `.env`
-2. Download ONNX models
-3. Configure `router.config.json`
-4. Test with `npm run test:router`
-
----
-
-**Created by @ruvnet**
-For issues: https://github.com/ruvnet/agentic-flow/issues