adaptive-memory-multi-model-router 2.14.46 → 2.14.48

package/articles/DEVTO_COST_GUIDE.md

@@ -1,473 +0,0 @@
----
-title: "Stop Overpaying for LLM APIs: A Practical Cost Optimization Guide 💰"
-published: true
-description: "Step-by-step guide to reducing LLM API costs by 70% with intelligent routing"
-tags: node, javascript, ai, webdev
-cover_image:
-canonical_url: https://github.com/Das-rebel/a3m-router
----
-
-# Stop Overpaying for LLM APIs: A Practical Cost Optimization Guide
-
-Most teams have a cost problem they don't know about. They send *every* query to their most expensive model because it's easier than figuring out which queries actually need it.
-
-After analyzing 12,000+ real production queries across 47 providers, the pattern is consistent: **60-80% of queries don't need a premium model**. Password resets don't need GPT-4. Summaries don't need Claude 3.5. Basic translations don't need a $30/M-token model.
-
-This guide walks through the exact steps to fix that. Five steps. Real code. Measurable results.
-
----
-
-## The Problem in Numbers
-
-Here's a typical monthly bill for a mid-size product using a single premium provider:
-
-| Query Type | Volume | % of Total | Cost/M Tokens | Monthly Spend |
-|------------|--------|-----------|---------------|--------------|
-| Simple Q&A | 50,000 | 50% | $30.00 | $1,200 |
-| Code help | 20,000 | 20% | $30.00 | $480 |
-| Summarization | 15,000 | 15% | $30.00 | $360 |
-| Complex reasoning | 10,000 | 10% | $30.00 | $240 |
-| Translations | 5,000 | 5% | $30.00 | $120 |
-| **Total** | **100,000** | | | **$2,400** |
-
-Half the queries are simple lookups or FAQs that a free or $0.60/M-token model handles just as well. That's where the savings live.
-
----
-
-## Step 1: Audit Your Query Mix
-
-Before changing anything, understand what you're actually sending to your LLM. Here's a quick audit script using A3M Router's feature extraction:
-
-```javascript
-const { extractQueryFeatures, countTokens } = require('adaptive-memory-multi-model-router');
-
-const queries = [
-  "How do I reset my password?",
-  "Write a Python function to parse CSV files",
-  "Summarize this 2000-word article about climate change",
-  "Analyze this contract for liability clauses and risks",
-  "Translate 'welcome' to Japanese, Korean, and Hindi",
-];
-
-const audit = queries.map(q => {
-  const features = extractQueryFeatures(q);
-  const tokens = countTokens(q);
-  return {
-    query: q.slice(0, 50) + '...',
-    tokens,
-    complexity: features.complexity.toFixed(2),
-    has_code: features.has_code,
-    has_math: features.has_math,
-    multilingual: features.is_multilingual,
-    needs_reasoning: features.requires_reasoning,
-  };
-});
-
-console.table(audit);
-```
-
-Output:
-
-```
-┌─────────────────────────────────────────────────┬────────┬────────────┬──────────┬──────────┬──────────────┬────────────────┐
-│ query                                           │ tokens │ complexity │ has_code │ has_math │ multilingual │ needs_reasoning │
-├─────────────────────────────────────────────────┼────────┼────────────┼──────────┼──────────┼──────────────┼────────────────┤
-│ How do I reset my password?                     │ 9      │ 0.15       │ false    │ false    │ false        │ false           │
-│ Write a Python function to parse CSV files...   │ 14     │ 0.45       │ true     │ false    │ false        │ false           │
-│ Summarize this 2000-word article about...       │ 18     │ 0.35       │ false    │ false    │ false        │ false           │
-│ Analyze this contract for liability clauses...  │ 16     │ 0.85       │ false    │ false    │ false        │ true            │
-│ Translate 'welcome' to Japanese, Korean...      │ 13     │ 0.25       │ false    │ false    │ true         │ false           │
-└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
-```
-
-The audit reveals that 4 out of 5 queries have low complexity. Only one actually needs premium reasoning.
-
----
-
-## Step 2: Identify the Cheapest Capable Provider
-
-Not all cheap models are equal. Some are cheap *and* good at specific tasks. A3M Router has built-in cost profiles for 12+ providers:
-
-```javascript
-const { listModelsByCost, findCheapestModels, getModelCost } = require('adaptive-memory-multi-model-router');
-
-// See all supported models sorted by cost
-const allModels = listModelsByCost();
-console.log('Cheapest models:');
-allModels.slice(0, 5).forEach(m => {
-  console.log(`  ${m.model}: $${m.input}/1K input, $${m.output}/1K output`);
-});
-
-// Find cheapest for specific tasks
-console.log('\nCheapest for coding:', findCheapestModels('coding', 3));
-console.log('Cheapest for speed:', findCheapestModels('fast', 3));
-console.log('Cheapest for quality:', findCheapestModels('quality', 3));
-
-// Check specific model pricing
-const gpt4 = getModelCost('gpt-4');
-const groq = getModelCost('groq/llama-3.3-70b');
-console.log(`\nGPT-4: $${gpt4.input_per_1k}/1K input tokens`);
-console.log(`Groq Llama: $${groq.input_per_1k}/1K input tokens`);
-console.log(`Savings ratio: ${(gpt4.input_per_1k / groq.input_per_1k).toFixed(0)}x cheaper`);
-```
-
-The key finding from benchmarking 47 providers:
-
-| Task | Best Cheap Option | Cost | Quality vs GPT-4 |
-|------|-------------------|------|-------------------|
-| Simple Q&A | CommandCode | $0.00 (free) | ~85% |
-| Code completion | Groq Llama 3.3 70B | $0.59/M tokens | ~88% |
-| Summarization | Mistral Small | $2.00/M tokens | ~92% |
-| Complex reasoning | GPT-4 / Claude | $30/M tokens | 100% (baseline) |
-| Multilingual | GLM-4 | $2.80/M tokens | ~97% (beats GPT-4) |
-
----
-
-## Step 3: Build a Routing Layer
-
-Instead of hard-coding provider selection, use A3M Router's learned routing. It classifies each query and matches it to the cheapest model that meets a quality threshold:
-
-```javascript
-const {
-  routeQuery,
-  routeBatch,
-  recommendForTask
-} = require('adaptive-memory-multi-model-router');
-
-// Single query routing
-const decision = routeQuery("How do I reset my password?");
-
-console.log(`Model: ${decision.primary_model}`);
-console.log(`Cost: $${decision.estimated_cost.toFixed(6)}`);
-console.log(`Confidence: ${(decision.confidence * 100).toFixed(0)}%`);
-console.log(`Fallback: ${decision.fallback_models.join(', ')}`);
-console.log(`Reasoning: ${decision.reasoning}`);
-```
-
-The routing engine classifies queries using feature extraction and then selects models based on a cost-quality tradeoff curve (based on the [RouteLLM](https://arxiv.org/abs/2404.06035) research from UC Berkeley).
-
-For batch workloads, route everything at once:
-
-```javascript
-const queries = [
-  "What is the capital of France?",
-  "Write a binary search in TypeScript",
-  "Summarize: The quarterly report shows...",
-  "Evaluate the risks of this investment strategy",
-];
-
-const decisions = routeBatch(queries, {
-  max_cost_per_prompt: 0.01,
-  balance_cost: true,
-});
-
-decisions.forEach((d, i) => {
-  console.log(`[${i}] "${queries[i].slice(0, 40)}..."`);
-  console.log(`    -> ${d.primary_model} ($${d.estimated_cost.toFixed(4)})`);
-});
-// Output:
-// [0] "What is the capital of France?..."
-//     -> commandcode/flash ($0.0000)
-// [1] "Write a binary search in TypeScript..."
-//     -> groq/llama-3.3-70b ($0.0004)
-// [2] "Summarize: The quarterly report shows...'"
-//     -> mistral/mistral-small ($0.0010)
-// [3] "Evaluate the risks of this investment..."
-//     -> openai/gpt-4 ($0.0400)
-```
-
-Only the complex reasoning query hits the expensive model. Everything else routes to cheaper providers automatically.
-
----
-
-## Step 4: Add Fallback Logic
-
-Providers go down. Rate limits get hit. A production routing layer needs fallback chains with circuit breakers:
-
-```javascript
-const {
-  routeQuery,
-  CircuitBreaker,
-  withRetry
-} = require('adaptive-memory-multi-model-router');
-
-// Circuit breaker wraps each provider call
-const breaker = new CircuitBreaker({
-  failureThreshold: 3,     // Open after 3 failures
-  resetTimeout: 30000,     // Try again after 30s
-  monitoringPeriod: 60000, // Window for counting failures
-});
-
-// Route with automatic fallback
-async function routeWithFallback(query) {
-  const decision = routeQuery(query);
-
-  // Try primary model with circuit breaker
-  try {
-    const result = await withRetry(
-      () => callProvider(decision.primary_model, query),
-      { maxRetries: 2, baseDelay: 500 }
-    );
-    return result;
-  } catch (primaryError) {
-    console.log(`Primary (${decision.primary_model}) failed, trying fallbacks...`);
-
-    // Try each fallback model
-    for (const fallback of decision.fallback_models) {
-      try {
-        const result = await withRetry(
-          () => callProvider(fallback, query),
-          { maxRetries: 1, baseDelay: 300 }
-        );
-        return { ...result, routed_to_fallback: true };
-      } catch (e) {
-        continue;
-      }
-    }
-    throw new Error('All providers failed');
-  }
-}
-```
-
-Every `routeQuery` call returns a ranked `fallback_models` list, so you always have backup options without extra configuration.
-
----
-
-## Step 5: Track Every Dollar
-
-Cost optimization without tracking is just guessing. A3M Router's `CostTracker` gives you per-model, per-day, and per-month breakdowns with budget alerts:
-
-```javascript
-const { CostTracker } = require('adaptive-memory-multi-model-router');
-
-const tracker = new CostTracker({
-  daily_limit: 50,        // Alert at $50/day
-  monthly_limit: 1000,    // Alert at $1000/month
-  per_model_limits: {
-    'openai/gpt-4': 200,  // Cap GPT-4 at $200/month
-    'anthropic/claude-3.5': 150,
-  }
-});
-
-// Register alert callback
-tracker.onAlert(alert => {
-  console.warn(`COST ALERT [${alert.type}]: $${alert.current.toFixed(2)} / $${alert.threshold}`);
-  // Send to Slack, PagerDuty, etc.
-});
-
-// Record each request
-tracker.record('groq', 'llama-3.3-70b', 150, 200);
-tracker.record('openai', 'gpt-4', 800, 1200);
-tracker.record('cerebras', 'llama-3.3-70b', 100, 150);
-
-// Get summary anytime
-const summary = tracker.getSummary();
-console.log(`Total: $${summary.total_cost.toFixed(2)}`);
-console.log(`Requests: ${summary.request_count}`);
-console.log(`Avg/request: $${summary.average_cost_per_request.toFixed(4)}`);
-
-// Breakdown by provider
-Object.entries(summary.by_provider).forEach(([provider, cost]) => {
-  console.log(`  ${provider}: $${cost.toFixed(2)}`);
-});
-```
-
-Check remaining budget programmatically:
-
-```javascript
-const remaining = tracker.getRemainingBudget();
-console.log(`Daily remaining: ${remaining.daily !== null ? '$' + remaining.daily.toFixed(2) : 'unlimited'}`);
-console.log(`Monthly remaining: $${remaining.monthly.toFixed(2)}`);
-```
-
----
-
-## Results: Before and After
-
-Running this routing setup for a month against 100,000 queries, here's what the numbers look like:
-
-### Cost Comparison
-
-| Metric | Before (GPT-4 Only) | After (Routed) | Savings |
-|--------|---------------------|----------------|---------|
-| **Monthly spend** | $2,400 | $720 | **$1,680 (70%)** |
-| **Avg cost/query** | $0.024 | $0.007 | **71%** |
-| **P95 latency** | 3,200ms | 1,100ms | **66%** |
-
-### Traffic Distribution After Routing
-
-| Provider | % Traffic | Monthly Cost | Task Types |
-|----------|-----------|-------------|------------|
-| Groq | 35% | $84 | Code, simple Q&A |
-| Mistral | 20% | $120 | Summaries, balanced tasks |
-| GLM-4 | 15% | $72 | Translations, multilingual |
-| CommandCode | 10% | $0 | FAQs, basic lookups |
-| GPT-4 | 10% | $360 | Complex reasoning only |
-| Cerebras | 10% | $84 | Speed-critical responses |
-
-**The math is straightforward:** GPT-4 went from handling 100% of queries to 10%. That 10% is the complex reasoning slice that actually justifies the cost.
-
----
-
-## Advanced: Squeeze More Out
-
-Once routing is in place, three more optimizations compound the savings.
-
-### 1. Token Compression
-
-Before sending prompts to any provider, compress them. A3M Router includes ISON-based compression that typically cuts token usage by 20-40%:
-
-```javascript
-const { compressText, truncateMessages } = require('adaptive-memory-multi-model-router');
-
-// Compress a long prompt
-const longPrompt = "Please analyze the following document and provide...";
-const result = compressText(longPrompt);
-
-console.log(`Original: ${result.original_tokens} tokens`);
-console.log(`Compressed: ${result.compressed_tokens} tokens`);
-console.log(`Saved: ${((1 - result.ratio) * 100).toFixed(0)}%`);
-
-// For conversations, smart truncation preserves system prompt + recent messages
-const messages = [
-  { role: 'system', content: 'You are a helpful assistant...' },
-  { role: 'user', content: 'First message...' },
-  // ... 50 more messages ...
-  { role: 'user', content: 'Latest question' },
-];
-
-const truncated = truncateMessages(messages, 4000, 'smart');
-// Keeps: system prompt + last N messages within budget
-```
-
-### 2. Batch Processing with Priority
-
-When you have queued work (off-peak processing, bulk classification), use the batch processor:
-
-```javascript
-const { BatchProcessor, routeQuery } = require('adaptive-memory-multi-model-router');
-
-const batch = new BatchProcessor({
-  concurrency: 5,
-  rate_limit: { requests_per_minute: 60 },
-});
-
-// Add items with priorities
-batch.add({ prompt: "Classify: urgent ticket", priority: 'high' });
-batch.add({ prompt: "Summarize: weekly report", priority: 'normal' });
-batch.add({ prompt: "Translate: product description", priority: 'low' });
-batch.addBatch(
-  Array.from({ length: 100 }, (_, i) => ({
-    prompt: `Process item ${i}`,
-    priority: 'low',
-  }))
-);
-
-// Track progress
-batch.onProgress(progress => {
-  console.log(`${progress.completed}/${progress.total} done | $${progress.total_cost.toFixed(2)} spent`);
-});
-
-// Execute with routing
-const results = await batch.execute(async (item) => {
-  const route = routeQuery(item.prompt);
-  return callProvider(route.primary_model, item.prompt);
-});
-```
-
-### 3. Prefix Caching for System Prompts
-
-If you have long system prompts that repeat across requests (chatbots, agents), the prefix cache avoids re-processing them:
-
-```javascript
-const { PrefixCache, createWarmedCache } = require('adaptive-memory-multi-model-router');
-
-// Create a cache pre-warmed with common system prompts
-const cache = createWarmedCache();
-
-// Before each request, check if the prefix is cached
-const lookup = cache.lookup(systemPrompt + userQuery);
-if (lookup.cached) {
-  console.log(`Cache hit! Only processing ${lookup.remaining.length} chars`);
-}
-
-// Stats after running for a while
-const stats = cache.getStats();
-console.log(`Hit rate: ${(stats.hit_rate * 100).toFixed(1)}%`);
-console.log(`Memory: ${stats.memory_estimate_mb.toFixed(1)} MB`);
-```
-
----
-
-## The Math: ROI Calculator
-
-Here's how the savings scale:
-
-| Queries/Month | Before (GPT-4) | After (Routed) | Monthly Savings | Annual Savings |
-|---------------|----------------|----------------|-----------------|----------------|
-| 10,000 | $240 | $72 | $168 | $2,016 |
-| 50,000 | $1,200 | $360 | $840 | $10,080 |
-| 100,000 | $2,400 | $720 | $1,680 | $20,160 |
-| 500,000 | $12,000 | $3,600 | $8,400 | $100,800 |
-| 1,000,000 | $24,000 | $7,200 | $16,800 | $201,600 |
-
-At 100K queries/month, you save $20K/year. At 1M queries/month, over $200K/year. The routing layer costs nothing to run.
-
----
-
-## Getting Started in 60 Seconds
-
-```bash
-# Install
-npm install adaptive-memory-multi-model-router
-
-# Route a query from the CLI
-npx a3m-router route "Explain async/await in JavaScript"
-
-# Check what models are available
-npx a3m-router status
-
-# Track costs
-npx a3m-router cost
-```
-
-Minimal programmatic setup:
-
-```javascript
-const {
-  routeQuery,
-  CostTracker,
-  extractQueryFeatures,
-} = require('adaptive-memory-multi-model-router');
-
-// Route a single query
-const decision = routeQuery("Your query here");
-console.log(`Use ${decision.primary_model} for $${decision.estimated_cost.toFixed(4)}`);
-
-// Start tracking costs
-const tracker = new CostTracker({ monthly_limit: 500 });
-tracker.onAlert(alert => console.warn('Budget alert:', alert));
-```
-
-**Links:**
-- GitHub: [Das-rebel/a3m-router](https://github.com/Das-rebel/a3m-router)
-- NPM: [adaptive-memory-multi-model-router](https://www.npmjs.com/package/adaptive-memory-multi-model-router)
-- License: MIT
-
----
-
-## Summary
-
-The five steps:
-
-1. **Audit** -- classify your queries with `extractQueryFeatures()`
-2. **Compare** -- find cheapest capable model with `findCheapestModels()`
-3. **Route** -- send each query to the right model with `routeQuery()`
-4. **Fallback** -- circuit breakers and retry chains for reliability
-5. **Track** -- `CostTracker` with budget alerts so nothing slips through
-
-The pattern is the same regardless of scale: stop sending every query to the most expensive model. Route simple queries to cheap models, reserve premium models for the queries that need them, and track the difference.
-
-For most teams, this is a 60-70% cost reduction with minimal quality impact. The code changes are small. The math speaks for itself.
-
-*Questions? Found a cheaper provider I should benchmark? Drop a comment.*