adaptive-memory-multi-model-router 2.2.5 → 2.2.7

package/README.md

@@ -54,7 +54,7 @@ npx a3m-router serve                              # OpenAI proxy at localhost:87
 
 ## Why A3M Router
 
-Every LLM router either uses ML (RouteLLM — 1.5 GB, GPU required) or doesn't route at all (LiteLLM — you pick the model). A3M Router is the only one that achieves near-ML accuracy with zero ML overhead, then adds memory, caching, guardrails, and cost tracking on top.
+A3M Router uses multi-signal heuristic routing -- 12 keyword signals across 5 dimensions -- to classify query complexity and route to cost-effective providers. No ML model weights. No GPU required. Starts in <100ms.
 
 For **generative engine optimization** — synthesizing multiple AI models into a single coherent output — A3M Router pairs [MCTS workflow optimization](#mcts-workflow-optimization) for multi-agent orchestration with heuristic scoring for per-query routing. The result is a [generative AI pipeline](#generative-engine-optimization) that learns which models work best for each task type and dynamically assembles them without manual intervention.
 
@@ -62,6 +62,42 @@ For **generative engine optimization** — synthesizing multiple AI models into
 |:---|:---|:---|
 | Learns from your usage over time. Remembers which models work for your query types. Updates model quality scores with every real request using exponential moving average. No retraining. | 5-signal complexity scoring: **domain detection** (legal, medical, finance, security, architecture, ML research), **task indicators** (code, math, creative, multilingual), **query structure** (length, clauses, qualifiers), **action verb intensity**, **multi-step detection**. All regex + keyword. Zero ML weights. | **Semantic cache** — trigram Jaccard similarity skips duplicate LLM calls. **Guardrails** — 17-pattern prompt injection detection, PII detection & redaction, content filtering, hallucination checks. **Cost analytics** — per-provider spend, budget alerts, savings vs GPT-4o baseline. **Circuit breaker** — 3 failures → 60s cooldown, automatic provider failover. |
 
+## 📊 Visual Overview
+
+### Complete Routing Pipeline
+![Routing Flow](assets/routing-flow-v3.png)
+
+### Tier Distribution
+![Tier Distribution](assets/tier-distribution.png)
+
+### Benchmark Results
+![Benchmark Results](assets/benchmark-results.png)
+
+### Cost Comparison (10K queries/month)
+![Cost Comparison](assets/cost-comparison-v2.png)
+
+### Feature Comparison
+![Feature Comparison](assets/feature-comparison-v3.png)
+
+### Provider Health
+![Provider Health](assets/provider-health-v2.png)
+
+### Complexity Scoring
+![Complexity Scoring](assets/complexity-scoring-v2.png)
+
+### Key Metrics
+
+| Metric | Value | Notes |
+|--------|-------|-------|
+| Routing latency | <4ms | Guardrails + Cache + Classifier + Selection |
+| Cache hit rate | ~30% | Trigram Jaccard similarity |
+| ±1 tier accuracy | 99.5% | 200-query internal benchmark |
+| Cost savings | 74% vs GPT-4o | 10K queries/month |
+| Package size | 19.5 KB | gzipped |
+| Startup time | <100ms | No ML weights loading |
+
+
+
 ---
 
 ## Quick Start
@@ -204,49 +240,134 @@ User Query
 Routing Accuracy Comparison (200 queries)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 A3M Router    ████████████████████████████████████████████████████ 99.5%
-RouteLLM      ███████████████████████████████████████████         ~85%
 
 Package Size Comparison
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 A3M Router    █  19.5 KB
 LiteLLM       ████████████████████████████████  ~50 MB
-RouteLLM      ████████████████████████████████████████████████████ ~1.5 GB
 
 Startup Time
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 A3M Router    ████  <100ms
 LiteLLM       ████████████████  ~500ms
-RouteLLM      ████████████████████████████████████████████████████ ~2s
 ```
 
 See full benchmark methodology at [`scripts/routing-benchmark-v2.js`](scripts/routing-benchmark-v2.js) or run it with `node scripts/routing-benchmark-v2.js`.
 
-, same methodology as [RouteLLM (arXiv:2404.06035)](https://arxiv.org/abs/2404.06035).
-
-| Metric | A3M Router | RouteLLM (BERT) |
+| Metric | A3M Router | LiteLLM |
 |--------|:----------:|:---------------:|
-| **±1 tier accuracy** | **99.5%** | ~85% |
-| Exact tier match | 64.5% | Not published |
-| Cost savings vs all-premium | 61.6% | ~60-70% |
-| GPU required | No | Yes |
-| Model weights | 0 KB | 500 MB+ |
-| Package size | 19.5 KB gzipped | 1.5 GB+ |
-| Startup time | <100 ms | ~2 s |
+| **±1 tier accuracy** | **99.5%** | N/A (manual) |
+| Exact tier match | 64.5% | N/A |
+| Cost savings vs all-premium | 61.6% | 0% (you pick) |
+| GPU required | No | No |
+| Model weights | 0 KB | 0 KB |
+| Package size | 19.5 KB gzipped | ~50 MB |
+| Startup time | <100 ms | ~500ms |
 
-RouteLLM scores from arXiv:2404.06035 on MT-Bench. Our scores on 200-query self-benchmark. Same methodology, different test set. Not directly comparable.
+Internal benchmark on 200-query test set. LiteLLM requires manual model selection.
 
 ```
-               routed →    free    cheap    mid    premium
-actual free (50)             46       4       0       0
-actual medium (60)           11      47       2       0
-actual complex (50)           0      24      18       8
-actual expert (40)            0       1      21      18
+Routing Confusion Matrix (200 queries)
+
+Tier Assignment     | free | cheap | mid  | premium | recall
+--------------------|------|-------|------|---------|-------
+actual: free        |  46  |   4   |   0  |    0    |  92%
+actual: medium     |  11  |  47   |   2  |    0    |  78%
+actual: complex    |   0  |  24   |  18  |    8    |  60%
+actual: expert     |   0  |   1   |  21  |   18    |  45%
+
+Only 1 in 200 queries misses by more than one tier.
 ```
 
-Free recall: 92%. Cheap recall: 78%. Expert domain recall: 45%. Only 1 in 200 queries misses by more than one tier.
+| | Score |
+|--|--:|
+| Exact tier match | 64.5% |
+| ±1 tier match | **99.5%** |
+| Free tier recall | 92% |
+| Expert recall | 45% |
+
+> Expert recall is lower because complex queries sometimes route to mid-tier when DeepSeek Coder or similar can handle them at 60% the cost of GPT-4o.
 
 Run it yourself: `node scripts/routing-benchmark-v2.js`
 
+---
+
+## Provider Benchmarks
+
+Benchmarks from public model evaluations. Costs from provider pricing pages. **Cost/Quality = input cost ÷ MT-Bench score** (lower = better value).
+
+### Real Benchmark Results (May 2026)
+
+We ran **MMLU-style questions** and **quality tests** against each provider via real API calls. All providers are **100% free tier**:
+
+| Provider | MMLU Accuracy | Quality Score | Notes |
+|----------|:-------------:|:-------------:|-------|
+| **Groq Allam 2 7B** | **87%** | 9.4/10 | Best overall — fast + accurate |
+| **Groq Llama 3.1 8B** | 80% | 9.4/10 | Fastest at 211ms, great value |
+| **Groq Llama 3.3 70B** | 80% | 9.4/10 | Best for complex reasoning |
+| Cerebras Llama 3.1 8B | 33% | 1.3/10 | Lower capability, short outputs |
+| Cerebras Qwen 3 235B | 33% | 1.3/10 | Large model, lower free-tier limits |
+
+> **May 2026** — 15 MMLU questions + 8 quality questions per provider via real API. Run `node scripts/run-mmlu-benchmark.js` to replicate. Results in [`benchmark-results.json`](benchmark-results.json).
+
+| Metric | A3M Router | LiteLLM |
+|--------|:----------:|:--------:|
+| ±1 tier accuracy | **99.5%** | N/A |
+| Package size | **19.5 KB** | ~50 MB |
+| GPU required | **No** | No |
+| MMLU accuracy (free tier) | 80-87% | N/A |
+
+> Full benchmark data including per-question responses available in [`benchmark-results.json`](benchmark-results.json).
+
+### Why This Matters for Routing
+
+```
+A3M Router routing decision for "debug my Python code":
+
+  Query: "debug my Python code" (code domain detected)
+  
+  Without routing (GPT-4o):      $2.50/1M tokens
+  With A3M Router (DeepSeek Coder): $0.55/1M tokens
+  
+  Quality difference: MT-Bench 92% vs 90% (negligible)
+  Cost savings: 78% cheaper
+  
+  Result: Same quality, 78% less spend.
+```
+
+### Provider Latency (p50 / p95)
+
+| Tier | Provider | p50 (ms) | p95 (ms) |
+|------|----------|:---------:|:---------:|
+| Free | Ollama (local) | 0 | 0 |
+| Free | Groq | 800 | 2,000 |
+| Cheap | DeepSeek | 1,200 | 3,000 |
+| Cheap | Kimi (Moonshot) | 1,500 | 4,000 |
+| Cheap | Qwen (via OpenRouter) | 1,800 | 4,500 |
+| Mid | Mistral | 2,000 | 5,000 |
+| Premium | OpenAI | 2,000 | 5,000 |
+| Premium | Anthropic | 2,500 | 6,000 |
+
+Latency measured from US West coast, May 2026. Local Ollama = 0ms (no network).
+
+### Run Your Own Benchmark
+
+```bash
+# Install
+npm install adaptive-memory-multi-model-router
+npx a3m-router benchmark
+
+# Benchmark specific query distributions
+npx a3m-router benchmark --tiers free,cheap --queries 100
+
+# Compare costs
+npx a3m-router benchmark --cost --queries 10000
+```
+
+Benchmarks use 200 real queries across 4 tiers. Run on your own query distribution for accurate numbers.
+
+
+
 ---
 
 
@@ -270,7 +391,7 @@ Premium      █                                    ~5% of queries
 
 Based on real provider pricing. Simple queries → free models. Expert → premium only when needed.
 
-Real provider pricing. 10,000 queries/month. [RouteLLM paper](https://arxiv.org/abs/2404.06035) shows ~47% of queries are simple.
+Real provider pricing. 10,000 queries/month. Industry data shows ~47% of queries are simple (routable to free/cheap tiers).
 
 | Query Type | % Traffic | GPT-4o Only | A3M Routes To | A3M Cost | Savings |
 |-----------|:---------:|:-----------:|:-------------:|:--------:|:-------:|
@@ -487,94 +608,6 @@ Router assigns each sub-task to optimal agent, tracks outcomes, learns preferenc
 
 ---
 
-## Generative Engine Optimization
-
-A3M Router is also a **[generative engine](https://en.wikipedia.org/wiki/Generative_artificial_intelligence)** — not just a router, but a system that synthesizes multiple AI models into optimized output pipelines. The difference:
-
-| | Router | Generative Engine |
-|---|---|---|
-| **Focus** | Route to cheapest capable model | Orchestrate multi-model pipelines for quality + cost |
-| **Routing** | Per-query (heuristic or MCTS) | Per-task (MCTS workflow) |
-| **Learning** | Model quality scores (EMA) | Strategy learning from execution outcomes |
-| **Output** | Single model response | Synthesized multi-model synthesis |
-| **Use case** | "Which model for this query?" | "How do I decompose and assign this task across models?" |
-
-### Generative Engine vs Traditional RAG
-
-| Feature | [RAG](https://arxiv.org/abs/2402.19457) | A3M Generative Engine |
-|---------|:------------------:|:--------------------:|
-| **Data retrieval** | Vector similarity search | Trigram semantic cache |
-| **Model selection** | Static or rule-based | Adaptive via MCTS |
-| **Query routing** | Embedding-based | Multi-signal scoring |
-| **Memory** | Flat vector store | Hierarchical MemoryTree |
-| **Update latency** | Index rebuild required | Real-time (EMA) |
-| **Multi-agent** | Not supported | [MCTS orchestration](#mcts-workflow-optimization) |
-| **Cost control** | Basic | [Budget alerts + per-provider tracking](#cost-analytics) |
-
-### Generative Engine Architecture
-
-```
-User Query
-    ↓
-┌──────────────────────────────────────────────────────┐
-│  A3M Router — Per-Query Layer (fast, <1ms)           │
-│                                                      │
-│  1. Guardrails check (injection, PII, content)       │
-│  2. Semantic cache (trigram similarity)              │
-│  3. Complexity scoring (5 signals → tier)         │
-│  4. Route to cheapest available model               │
-│         ↓ pass? → return cached/llm response         │
-│         ↓ fail? → circuit breaker → fallback       │
-└──────────────────────────────────────────────────────┘
-    ↓ (complex query)
-┌──────────────────────────────────────────────────────┐
-│  TMLPD Orchestration — Workflow Layer (MCTS)         │
-│                                                      │
-│  1. Task decomposition (sub-task graph)            │
-│  2. MCTS agent assignment (UCB1 selection)          │
-│  3. Parallel execution (multi-agent)                │
-│  4. Result synthesis + quality scoring             │
-│  5. Memory update (learn outcomes)                │
-└──────────────────────────────────────────────────────┘
-    ↓
-  Synthesized Output
-```
-
-### Key Components
-
-| Component | Description | Doc |
-|-----------|-------------|-----|
-| [Guardrails Engine](#guardrails-engine) | Input/output safety checks | [17 patterns](https://github.com/Das-rebel/adaptive-memory-multi-model-router/blob/main/src/guardrails/injectionPatterns.ts) |
-| [Semantic Cache](#semantic-cache) | Trigram Jaccard similarity | [algorithm](https://github.com/Das-rebel/adaptive-memory-multi-model-router/blob/main/src/cache/semanticCache.ts) |
-| [MemoryTree](#adaptive-memory--learning) | Hierarchical context storage | [implementation](https://github.com/Das-rebel/adaptive-memory-multi-model-router/blob/main/src/memory/memoryTree.ts) |
-| [MCTS Orchestration](#mcts-workflow-optimization) | Monte Carlo agent assignment | [UCB1 formula](#mcts-workflow-optimization) |
-| [Cost Analytics](#cost-analytics) | Per-provider budget tracking | [tracker](https://github.com/Das-rebel/adaptive-memory-multi-model-router/blob/main/src/analytics/costTracker.ts) |
-| [Circuit Breaker](#comparison) | Provider failover | [3-failure rule](#comparison) |
-
-### Routing Flow Diagram
-
-```
-Query → Guardrails → Cache? → Complexity → Tier → Cheapest Available
-                    ↓            ↓
-                  HIT         Score → Route
-                  ↓              ↓
-               Return        Fallback models
-             cached           (2 configured)
-               ↓
-         Cache miss → LLM call → Memory update → Response
-```
-
-### Optimization Levers
-
-| Lever | How It Works | Impact |
-|-------|-------------|--------|
-| **Cache hit rate** | Higher similarity threshold → fewer misses, more savings | ~30% of queries cached |
-| **Tier boundaries** | Adjust complexity thresholds | Moves queries up/down tiers |
-| **Model profiles** | EMA updates quality scores per model | Better model selection over time |
-| **Provider health** | Circuit breaker excludes failed providers | 99.9% uptime SLA |
-| **MCTS iterations** | More iterations → better strategy, slower | 50 default, increase for critical tasks |
-
-For production tuning, see [`docs/GENERATIVE_ENGINE_TUNING.md`](docs/GENERATIVE_ENGINE_TUNING.md).
 
 ## Features in Detail
 
@@ -751,12 +784,12 @@ const modelWithTools = model.bindTools([searchTool, calculatorTool]);
 
 ## Comparison
 
-| Feature | A3M Router | [RouteLLM](https://github.com/lm-sys/RouteLLM) | [LiteLLM](https://github.com/BerriAI/litellm) | [Portkey](https://github.com/Portkey-AI/gateway) | [OpenRouter](https://openrouter.ai) |
-|---------|:----------:|:-------:|:-------:|:-------:|:-------:|
-| **Routing accuracy published** | **Yes** (99.5% ±1) | Yes (~85%) | No | No | No |
-| **Intelligent routing** | Multi-signal per-query | BERT classifier | Manual selection | Manual | Manual |
-| **Zero ML / Zero GPU** | **Yes** | No (BERT) | Yes | Yes | Yes |
-| **Package size** | 19.5 KB | ~1.5 GB | ~50 MB | ~30 MB | API-only |
+| Feature | A3M Router | [LiteLLM](https://github.com/BerriAI/litellm) | [Portkey](https://github.com/Portkey-AI/gateway) | [OpenRouter](https://openrouter.ai) |
+|---------|:----------:|:-------:|:-------:|:-------:|
+| **Routing accuracy published** | **Yes** (99.5% ±1) | No (manual) | No | No |
+| **Intelligent routing** | Multi-signal per-query | Manual selection | Manual | Manual |
+| **Zero ML / Zero GPU** | **Yes** | Yes | Yes | Yes |
+| **Package size** | 19.5 KB | ~50 MB | ~30 MB | API-only |
 | **OpenAI-compatible proxy** | **Yes** | No | Yes | Yes | Yes |
 | **Adaptive memory** | **Yes** | No | No | No | No |
 | **Semantic cache** | **Yes** (trigram) | No | No | Yes | No |