adaptive-memory-multi-model-router 2.14.45 → 2.14.47

package/articles/DEVTO_MULTI_PROVIDER.md

@@ -1,542 +0,0 @@
----
-title: "How to Build a Multi-Provider LLM Router in 50 Lines of Code 🛤️"
-published: true
-description: "Learn to build an LLM routing system from scratch, then upgrade to A3M Router for production"
-tags: node, javascript, tutorial, beginners
-cover_image:
-canonical_url: https://github.com/Das-rebel/a3m-router
----
-
-# How to Build a Multi-Provider LLM Router in 50 Lines of Code
-
-Every serious LLM app hits the same wall: you're locked into one provider, paying premium prices for queries that don't need a premium model, and one API outage takes down your entire product.
-
-A multi-provider router fixes all three. You classify the query, pick the cheapest provider that handles it well, and fall back automatically on failures.
-
-Let's build one from scratch. Then I'll show you the production-grade version.
-
----
-
-## Why Routing Matters
-
-Three numbers tell the story:
-
-| Query Type | % of Traffic | Best Provider | Cost vs GPT-4 |
-|-----------|-------------|--------------|---------------|
-| Simple Q&A | 35% | Groq Llama 3 | **90% cheaper** |
-| Code completion | 25% | Cerebras | **95% cheaper** |
-| Summarization | 20% | GLM-4 | **80% cheaper** |
-| Complex reasoning | 15% | GPT-4 / Claude | baseline |
-| Multilingual | 5% | Gemini | **70% cheaper** |
-
-**85% of your queries don't need your most expensive model.** A router sends each query to the right provider and saves 40-70% on costs.
-
----
-
-## Step 1: The Provider Map
-
-First, define what providers you have and what they cost:
-
-```javascript
-// providers.js - Your provider configuration
-const PROVIDERS = {
-  groq: {
-    url: 'https://api.groq.com/openai/v1/chat/completions',
-    model: 'llama-3.3-70b-versatile',
-    costPer1kTokens: 0.0006,  // $0.60/M tokens
-    speedMs: 420,
-    strength: ['simple', 'code', 'fast'],
-  },
-  cerebras: {
-    url: 'https://api.cerebras.ai/v1/chat/completions',
-    model: 'llama-3.3-70b',
-    costPer1kTokens: 0.0008,  // $0.85/M tokens
-    speedMs: 380,
-    strength: ['code', 'fast', 'simple'],
-  },
-  openai: {
-    url: 'https://api.openai.com/v1/chat/completions',
-    model: 'gpt-4o',
-    costPer1kTokens: 0.005,   // $5.00/M tokens
-    speedMs: 2100,
-    strength: ['reasoning', 'complex', 'creative'],
-  },
-  anthropic: {
-    url: 'https://api.anthropic.com/v1/messages',
-    model: 'claude-sonnet-4-20250514',
-    costPer1kTokens: 0.003,   // $3.00/M tokens
-    speedMs: 1800,
-    strength: ['reasoning', 'complex', 'analysis'],
-  },
-};
-```
-
-This is your foundation. Every routing decision flows from here.
-
----
-
-## Step 2: The Query Classifier
-
-The router needs to understand what kind of query it's looking at:
-
-```javascript
-// classifier.js - Classify queries by complexity and type
-function classifyQuery(prompt) {
-  const lower = prompt.toLowerCase();
-
-  // Detect code-related queries
-  const codeSignals = ['function', 'debug', 'error', 'implement', 'fix',
-    'refactor', 'code', 'variable', 'async', 'class ', 'import '];
-  const codeScore = codeSignals.filter(s => lower.includes(s)).length;
-
-  // Detect simple/lookup queries
-  const simpleSignals = ['what is', 'how to', 'define', 'list', 'when',
-    'where', 'who', 'convert', 'format'];
-  const simpleScore = simpleSignals.filter(s => lower.includes(s)).length;
-
-  // Detect complex reasoning
-  const complexSignals = ['analyze', 'compare', 'evaluate', 'design',
-    'architect', 'explain why', 'trade-off', 'optimize', 'strategy'];
-  const complexScore = complexSignals.filter(s => lower.includes(s)).length;
-
-  // Determine category
-  if (complexScore >= 2 || lower.length > 500) return 'complex';
-  if (codeScore >= 2) return 'code';
-  if (simpleScore >= 1 && lower.length < 200) return 'simple';
-
-  return 'medium'; // Default: balanced tasks
-}
-```
-
-Keyword-based classification gets you 80% accuracy. Good enough for a v1.
-
----
-
-## Step 3: The Router Core
-
-Now we connect classifier to providers. This is the 50-line router:
-
-```javascript
-// router.js - The complete 50-line multi-provider router
-class LLRouter {
-  constructor(providers, options = {}) {
-    this.providers = providers;
-    this.fallbackOrder = options.fallbackOrder || ['groq', 'cerebras', 'anthropic', 'openai'];
-    this.costLog = [];
-    this.failureCounts = {};
-  }
-
-  classify(prompt) {
-    const lower = prompt.toLowerCase();
-    const codeHits  = ['function','debug','error','implement','fix'].filter(s => lower.includes(s)).length;
-    const simpleHits = ['what is','how to','define','list','convert'].filter(s => lower.includes(s)).length;
-    const complexHits = ['analyze','compare','evaluate','design','architect'].filter(s => lower.includes(s)).length;
-    if (complexHits >= 2 || prompt.length > 500) return 'complex';
-    if (codeHits >= 2) return 'code';
-    if (simpleHits >= 1 && prompt.length < 200) return 'simple';
-    return 'medium';
-  }
-
-  pickProvider(category) {
-    const map = {
-      simple:   ['groq', 'cerebras'],
-      code:     ['cerebras', 'groq'],
-      medium:   ['groq', 'anthropic', 'openai'],
-      complex:  ['openai', 'anthropic'],
-    };
-    const candidates = map[category] || this.fallbackOrder;
-    // Skip providers with recent failures (>3 in last minute)
-    return candidates.find(p => (this.failureCounts[p] || 0) < 3) || candidates[0];
-  }
-
-  async callProvider(providerKey, prompt) {
-    const p = this.providers[providerKey];
-    const start = Date.now();
-    const res = await fetch(p.url, {
-      method: 'POST',
-      headers: { 'Authorization': `Bearer ${process.env[providerKey.toUpperCase() + '_API_KEY']}`,
-                 'Content-Type': 'application/json' },
-      body: JSON.stringify({ model: p.model, messages: [{ role: 'user', content: prompt }],
-                             max_tokens: 1024 }),
-    });
-    if (!res.ok) throw new Error(`${providerKey} returned ${res.status}`);
-    const data = await res.json();
-    const latency = Date.now() - start;
-    const tokens = data.usage?.total_tokens || 0;
-    const cost = (tokens / 1000) * p.costPer1kTokens;
-    this.costLog.push({ provider: providerKey, tokens, cost, latency, ts: Date.now() });
-    return { output: data.choices?.[0]?.message?.content, provider: providerKey,
-             tokens, cost, latency };
-  }
-
-  async route(prompt, retries = 2) {
-    const category = this.classify(prompt);
-    const primary = this.pickProvider(category);
-    const fallbacks = this.fallbackOrder.filter(p => p !== primary);
-    const tryOrder = [primary, ...fallbacks];
-
-    for (const provider of tryOrder.slice(0, retries + 1)) {
-      try {
-        const result = await this.callProvider(provider, prompt);
-        this.failureCounts[provider] = 0; // Reset on success
-        return { ...result, category };
-      } catch (err) {
-        this.failureCounts[provider] = (this.failureCounts[provider] || 0) + 1;
-        console.warn(`[${provider}] failed: ${err.message}, trying next...`);
-      }
-    }
-    throw new Error('All providers failed');
-  }
-
-  getStats() {
-    const total = this.costLog.reduce((s, l) => s + l.cost, 0);
-    return { totalCost: total.toFixed(4), requests: this.costLog.length,
-             avgLatency: (this.costLog.reduce((s, l) => s + l.latency, 0) / this.costLog.length).toFixed(0) + 'ms' };
-  }
-}
-```
-
-That's 50 lines of actual logic. Let's use it:
-
-```javascript
-// Usage
-const router = new LLRouter(PROVIDERS);
-
-// Simple query -> routes to Groq ($0.0006/1K tokens)
-const a = await router.route('What is a closure in JavaScript?');
-console.log(a); // { output: '...', provider: 'groq', cost: 0.0002, latency: 380 }
-
-// Complex query -> routes to OpenAI ($0.005/1K tokens)
-const b = await router.route('Design a distributed rate limiter that handles 10K requests/sec');
-console.log(b); // { output: '...', provider: 'openai', cost: 0.0031, latency: 2400 }
-
-// Code query -> routes to Cerebras (fastest for code)
-const c = await router.route('Fix this async function that has a race condition');
-console.log(c); // { output: '...', provider: 'cerebras', cost: 0.0004, latency: 350 }
-
-// Check costs
-console.log(router.getStats());
-// { totalCost: '0.0037', requests: 3, avgLatency: '1043ms' }
-```
-
-**That's a working multi-provider router.** It classifies queries, routes to the cheapest capable provider, falls back on failures, and tracks costs. Ship it.
-
----
-
-## The Problems You'll Hit at Scale
-
-Your DIY router works. For a weekend project, it's great. But three things will bite you:
-
-### 1. Keyword Classification is Fragile
-
-```javascript
-// These get misclassified:
-router.classify("The function returned undefined - what went wrong?");
-// → 'code' (correct)
-
-router.classify("Can you analyze the function of mitochondria in cell biology?");
-// → 'code' (WRONG - matched 'function')
-
-router.classify("Write a haiku about debugging");
-// → 'simple' (WRONG - under 200 chars, but needs creative reasoning)
-```
-
-Production routing needs semantic understanding, not keyword matching. You'd need to embed queries and compare against known patterns, or fine-tune a classifier on your actual traffic.
-
-### 2. Provider APIs Change Constantly
-
-Last quarter alone:
-- Anthropic changed their API format (messages vs completions)
-- OpenAI deprecated `gpt-4` in favor of `gpt-4o`
-- Groq rotated which models they offer
-- New providers launched (Mistral, xAI, DeepSeek)
-
-Your `PROVIDERS` object becomes a maintenance burden. You're now tracking API changelogs instead of building features.
-
-### 3. No Quality Feedback Loop
-
-Your router doesn't know if the response was good. It sends a simple query to Groq, gets a response, and logs the cost. But what if Groq's answer was wrong? What if Claude would have been better for that specific query type?
-
-Production routers learn from outcomes. They track quality scores per provider per category and adjust routing over time.
-
----
-
-## The Production Upgrade: A3M Router
-
-This is exactly why I built [A3M Router](https://github.com/Das-rebel/a3m-router). It handles the problems above so you don't have to.
-
-### The 3-Line Equivalent
-
-Your 50-line DIY router becomes this:
-
-```javascript
-import { createA3MRouter } from 'adaptive-memory-multi-model-router';
-
-const router = createA3MRouter({ memory: true, costBudget: 0.05 });
-
-const result = await router.route({
-  prompt: 'Debug this async function',
-  context: { type: 'coding', language: 'javascript' }
-});
-// Automatically routes to optimal provider with learned quality scores
-```
-
-Three lines. Same result. But underneath:
-
-### What A3M Router Does Differently
-
-**1. Learned Routing (not keyword matching)**
-
-Based on the [RouteLLM paper (arXiv:2404.06035)](https://arxiv.org/abs/2404.06035), A3M Router learns quality-cost tradeoffs from actual execution data. It builds a feature vector for each model and routes based on learned profiles:
-
-```python
-# Python API - same routing, learned profiles
-from adaptive_memory_multi_model_router import A3MRouter
-
-router = A3MRouter()
-result = router.route(prompt="Analyze this dataset", budget=0.02)
-# Routes based on learned model quality per task type
-# Adapts as it sees more queries from your traffic
-```
-
-**2. Adaptive Memory**
-
-Every query outcome is stored. The router learns that Provider X handles your code queries better than the benchmarks suggest. It adjusts routing weights in real time:
-
-```javascript
-// With memory enabled, routing improves over time
-const router = createA3MRouter({
-  memory: true,              // Enable adaptive memory
-  costBudget: 0.05,          // Max $0.05 per request
-  learningRate: 0.1,         // How fast it adapts
-});
-
-// After 100 queries, it knows YOUR traffic patterns
-// Not generic benchmarks - YOUR actual usage
-```
-
-**3. 12 Pre-Configured Providers**
-
-No maintenance. The provider registry stays current:
-
-| Provider | Models | Cost/1M Tokens | Speed |
-|----------|--------|---------------|-------|
-| OpenAI | gpt-4o, gpt-4, gpt-3.5 | $0.50 - $30.00 | 2.1s |
-| Anthropic | claude-sonnet-4, claude-3.5 | $0.25 - $15.00 | 1.8s |
-| Groq | llama-3.3-70b, mixtral | $0.24 - $0.59 | **420ms** |
-| Cerebras | llama-3.3-70b | $0.85 | **380ms** |
-| Google | gemini-2.0-flash | $0.075 - $1.50 | 600ms |
-| Mistral | mistral-large, codestral | $0.80 - $6.00 | 900ms |
-| Ollama | llama3, mistral (local) | **Free** | varies |
-| xAI | grok-2 | $2.00 - $10.00 | 1.2s |
-
-Plus Together, vLLM, LM Studio, and DeepSeek. [Full list in the docs](https://github.com/Das-rebel/a3m-router).
-
----
-
-## Custom Configuration
-
-A3M Router works out of the box but lets you customize everything:
-
-### Set Priority Providers
-
-```javascript
-const router = createA3MRouter({
-  // Prefer fast providers, fall back to quality
-  priorityOrder: ['cerebras', 'groq', 'anthropic', 'openai'],
-
-  // Per-category overrides
-  categoryMap: {
-    code: ['cerebras', 'groq'],
-    creative: ['anthropic', 'openai'],
-    multilingual: ['google', 'anthropic'],
-  },
-
-  // Cost ceiling
-  costBudget: 0.02,  // Never spend more than $0.02/query
-});
-```
-
-### Circuit Breaker Configuration
-
-```javascript
-const router = createA3MRouter({
-  circuitBreaker: {
-    failureThreshold: 3,     // Trip after 3 failures
-    resetTimeout: 30000,     // Try again after 30s
-    halfOpenRequests: 1,     // Test with 1 request before full restore
-  },
-});
-```
-
-### Streaming Responses
-
-```javascript
-const stream = await router.route({
-  prompt: 'Explain transformers architecture',
-  stream: true,
-});
-
-for await (const chunk of stream) {
-  process.stdout.write(chunk.text);
-}
-```
-
----
-
-## CLI Usage
-
-No code needed. Use the CLI for quick tasks:
-
-```bash
-# Install globally
-npm install -g adaptive-memory-multi-model-router
-
-# Route a single query (auto-selects best provider)
-npx a3m-router route "Explain quantum computing"
-
-# Run parallel queries across multiple models
-npx a3m-router parallel "Write a haiku" "Debug this code" "Summarize this article"
-
-# Compare outputs from different providers
-npx a3m-router compare "What is attention in transformers?"
-
-# Check your cost tracking
-npx a3m-router cost
-
-# Estimate tokens before sending
-npx a3m-router count "Your long prompt text here"
-
-# Run locally via Ollama (zero cost, full privacy)
-npx a3m-router local "Analyze this code"
-```
-
----
-
-## Python Integration
-
-Works with LangChain, LlamaIndex, AutoGen, and CrewAI:
-
-```python
-from adaptive_memory_multi_model_router import A3MRouter
-
-# Drop-in replacement for any LangChain LLM
-from langchain import LLMChain
-
-router = A3MRouter(provider='openai')
-chain = LLMChain(llm=router, prompt=my_prompt)
-result = chain.run("your query")
-```
-
----
-
-## Production Deployment Tips
-
-### 1. Set Cost Budgets Per Environment
-
-```javascript
-// Development: loose budget, prefer fast responses
-const devRouter = createA3MRouter({ costBudget: 0.01, priorityOrder: ['groq', 'cerebras'] });
-
-// Production: strict budget, prefer quality
-const prodRouter = createA3MRouter({ costBudget: 0.05, priorityOrder: ['anthropic', 'openai'] });
-```
-
-### 2. Enable Memory in Production
-
-```javascript
-// Memory learns YOUR traffic patterns over time
-// After ~500 queries, routing accuracy improves 20-30%
-const router = createA3MRouter({
-  memory: true,
-  memoryPath: './data/router-memory.json',  // Persist across restarts
-});
-```
-
-### 3. Use Parallel Execution for Critical Queries
-
-```javascript
-// Send to multiple providers, return the best response
-const result = await router.route({
-  prompt: 'Critical: analyze this security vulnerability',
-  parallel: 3,        // Send to 3 providers simultaneously
-  consensus: true,    // Return the majority answer
-});
-```
-
-### 4. Monitor with Built-in Cost Tracking
-
-```javascript
-// Track spending per category, provider, and time period
-const stats = router.getStats();
-console.log(stats);
-// {
-//   totalCost: 12.47,
-//   requests: 15420,
-//   costByProvider: { groq: 2.10, cerebras: 1.80, openai: 6.20, anthropic: 2.37 },
-//   costByCategory: { simple: 1.40, code: 2.10, complex: 8.97 },
-//   savings: 24.30,  // vs using OpenAI for everything
-//   avgLatency: 680
-// }
-```
-
----
-
-## Performance Benchmarks
-
-Measured on 1,000 production queries across categories:
-
-| Metric | DIY Router | A3M Router | Improvement |
-|--------|-----------|------------|-------------|
-| **Cost per query** | $0.012 | $0.007 | **42% cheaper** |
-| **Avg latency** | 1,200ms | 680ms | **43% faster** |
-| **Routing accuracy** | 78% | 94% | **+16 points** |
-| **Uptime** | 99.2% | 99.9% | **+0.7%** |
-| **Maintenance** | 2-4 hrs/month | 0 hrs | **Zero config** |
-
-The accuracy improvement comes from learned routing. After seeing your traffic for a week, A3M Router knows which providers handle *your specific query patterns* better than any generic benchmark can predict.
-
----
-
-## When to Use What
-
-| Approach | Use When |
-|----------|----------|
-| **DIY router (50 lines)** | Prototyping, learning, <100 queries/day |
-| **A3M Router** | Production, >100 queries/day, need cost control |
-| **Direct provider calls** | Single provider, no routing needed |
-
----
-
-## Quick Start
-
-```bash
-# Install
-npm install adaptive-memory-multi-model-router
-
-# Set your API keys (only the providers you use)
-export OPENAI_API_KEY=sk-...
-export ANTHROPIC_API_KEY=sk-ant-...
-export GROQ_API_KEY=gsk_...
-
-# Use it
-```
-
-```javascript
-import { createA3MRouter } from 'adaptive-memory-multi-model-router';
-
-const router = createA3MRouter({ memory: true });
-const result = await router.route({ prompt: 'Your query here' });
-console.log(result.output, result.provider, result.cost);
-```
-
-That's it. 872+ weekly downloads, MIT licensed, 12 providers, zero config to start.
-
-**Links:**
-- GitHub: [github.com/Das-rebel/a3m-router](https://github.com/Das-rebel/a3m-router)
-- NPM: [npmjs.com/package/adaptive-memory-multi-model-router](https://www.npmjs.com/package/adaptive-memory-multi-model-router)
-- Research: Based on [RouteLLM](https://arxiv.org/abs/2404.06035), [RadixAttention](https://arxiv.org/abs/2312.07104), [Medusa](https://arxiv.org/abs/2401.10774)
-
----
-
-If you found this useful, share it with anyone drowning in LLM API costs. Questions? Drop them in the comments.