adaptive-memory-multi-model-router 2.5.2 → 2.5.3

package/README.md

@@ -192,171 +192,6 @@ User Query
 | "Analyze economic implications of AI" | — | 0.41 | cheap | groq/llama-3.3-70b |
 | "Review this contract for liability" | legal | 0.87 | premium | anthropic/claude-3.5-sonnet |
 | "Design a clinical trial for oncology" | medical | 1.00 | premium | openai/gpt-4o |
-
----
-
-## Benchmark
-
-200 queries, 4 cost tiers
-### Benchmark Visualized
-
-```
-Routing Accuracy Comparison (200 queries)
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-A3M Router    ████████████████████████████████████████████████████ 99.5%
-
-Package Size Comparison
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-A3M Router    █  19.5 KB
-LiteLLM       ████████████████████████████████  ~50 MB
-
-Startup Time
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-A3M Router    ████  <100ms
-LiteLLM       ████████████████  ~500ms
-```
-
-See full benchmark methodology at [`scripts/routing-benchmark-v2.js`](scripts/routing-benchmark-v2.js) or run it with `node scripts/routing-benchmark-v2.js`.
-
-| Metric | A3M Router | LiteLLM |
-|--------|:----------:|:---------------:|
-| **±1 tier accuracy** | **99.5%** | N/A (manual) |
-| Exact tier match | 64.5% | N/A |
-| Cost savings vs all-premium | 61.6% | 0% (you pick) |
-| GPU required | No | No |
-| Model weights | 0 KB | 0 KB |
-| Package size | 19.5 KB gzipped | ~50 MB |
-| Startup time | <100 ms | ~500ms |
-
-Internal benchmark on 200-query test set. LiteLLM requires manual model selection.
-
-```
-Routing Confusion Matrix (200 queries)
-
-Tier Assignment     | free | cheap | mid  | premium | recall
---------------------|------|-------|------|---------|-------
-actual: free        |  46  |   4   |   0  |    0    |  92%
-actual: medium     |  11  |  47   |   2  |    0    |  78%
-actual: complex    |   0  |  24   |  18  |    8    |  60%
-actual: expert     |   0  |   1   |  21  |   18    |  45%
-
-Only 1 in 200 queries misses by more than one tier.
-```
-
-| | Score |
-|--|--:|
-| Exact tier match | 64.5% |
-| ±1 tier match | **99.5%** |
-| Free tier recall | 92% |
-| Expert recall | 45% |
-
-> Expert recall is lower because complex queries sometimes route to mid-tier when DeepSeek Coder or similar can handle them at 60% the cost of GPT-4o.
-
-Run it yourself: `node scripts/routing-benchmark-v2.js`
-
----
-
-## Provider Benchmarks
-
-Benchmarks from public model evaluations. Costs from provider pricing pages. **Cost/Quality = input cost ÷ MT-Bench score** (lower = better value).
-
-### Real Benchmark Results (May 2026)
-
-We ran **MMLU-style questions** and **quality tests** against each provider via real API calls. All providers are **100% free tier**:
-
-| Provider | MMLU Accuracy | Quality Score | Notes |
-|----------|:-------------:|:-------------:|-------|
-| **Groq Allam 2 7B** | **87%** | 9.4/10 | Best overall — fast + accurate |
-| **Groq Llama 3.1 8B** | 80% | 9.4/10 | Fastest at 211ms, great value |
-| **Groq Llama 3.3 70B** | 80% | 9.4/10 | Best for complex reasoning |
-| Cerebras Llama 3.1 8B | 33% | 1.3/10 | Lower capability, short outputs |
-| Cerebras Qwen 3 235B | 33% | 1.3/10 | Large model, lower free-tier limits |
-
-> **May 2026** — 15 MMLU questions + 8 quality questions per provider via real API. Run `node scripts/run-mmlu-benchmark.js` to replicate. Results in [`benchmark-results.json`](benchmark-results.json).
-
-| Metric | A3M Router | LiteLLM |
-|--------|:----------:|:--------:|
-| ±1 tier accuracy | **99.5%** | N/A |
-| Package size | **19.5 KB** | ~50 MB |
-| GPU required | **No** | No |
-| MMLU accuracy (free tier) | 80-87% | N/A |
-
-> Full benchmark data including per-question responses available in [`benchmark-results.json`](benchmark-results.json).
-
-### Why This Matters for Routing
-
-```
-A3M Router routing decision for "debug my Python code":
-
-  Query: "debug my Python code" (code domain detected)
-  
-  Without routing (GPT-4o):      $2.50/1M tokens
-  With A3M Router (DeepSeek Coder): $0.55/1M tokens
-  
-  Quality difference: MT-Bench 92% vs 90% (negligible)
-  Cost savings: 78% cheaper
-  
-  Result: Same quality, 78% less spend.
-```
-
-### Provider Latency (p50 / p95)
-
-| Tier | Provider | p50 (ms) | p95 (ms) |
-|------|----------|:---------:|:---------:|
-| Free | Ollama (local) | 0 | 0 |
-| Free | Groq | 800 | 2,000 |
-| Cheap | DeepSeek | 1,200 | 3,000 |
-| Cheap | Kimi (Moonshot) | 1,500 | 4,000 |
-| Cheap | Qwen (via OpenRouter) | 1,800 | 4,500 |
-| Mid | Mistral | 2,000 | 5,000 |
-| Premium | OpenAI | 2,000 | 5,000 |
-| Premium | Anthropic | 2,500 | 6,000 |
-
-Latency measured from US West coast, May 2026. Local Ollama = 0ms (no network).
-
-### Run Your Own Benchmark
-
-```bash
-# Install
-npm install adaptive-memory-multi-model-router
-npx a3m-router benchmark
-
-# Benchmark specific query distributions
-npx a3m-router benchmark --tiers free,cheap --queries 100
-
-# Compare costs
-npx a3m-router benchmark --cost --queries 10000
-```
-
-Benchmarks use 200 real queries across 4 tiers. Run on your own query distribution for accurate numbers.
-
-
-
----
-
-
-### 💰 Cost Visualization
-
-```
-Monthly Cost Comparison (100K queries/month)
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-GPT-4o Only    ████████████████████████████████████████████████████ $341
-A3M Router    ████████████                                          $124
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-Your savings  ████████████████████████████████                   $218/mo
-
-Cost by Tier (A3M Router routing 10K queries):
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-Free tier     ████████████████████████████████              ~50% of queries
-Cheap tier   █████████                          ~35% of queries
-Mid tier     ███                                 ~10% of queries
-Premium      █                                    ~5% of queries
-```
-
-Based on real provider pricing. Simple queries → free models. Expert → premium only when needed.
-
-Real provider pricing. 10,000 queries/month. Industry data shows ~47% of queries are simple (routable to free/cheap tiers).
-
 | Query Type | % Traffic | GPT-4o Only | A3M Routes To | A3M Cost | Savings |
 |-----------|:---------:|:-----------:|:-------------:|:--------:|:-------:|
 | Simple Q&A | 47% | $4.94 | CommandCode (free) | $0.00 | 100% |
@@ -879,3 +714,46 @@ Research shows heuristic routing with proper feature engineering achieves compar
 
 ---
 
+
+---
+
+## Benchmark Results (Real API Calls)
+
+### Routing Accuracy (200 queries, May 2026)
+
+| Metric | Score |
+|--------|-------|
+| **±1 Tier Accuracy** | **99.5%** |
+| Exact Tier Match | 64.5% |
+| Free Tier Recall | 92% |
+| Over-routing (wasteful) | 7% |
+| Under-routing (risky) | 28.5% |
+
+### Provider Performance (10 real queries each)
+
+| Provider | Success | Avg Latency | MMLU Accuracy | $/1M |
+|----------|---------|-------------|---------------|------|
+| **Groq Allam 2 7B** | 100% | **156ms** | **87%** | $0 |
+| Groq Llama 3.1 8B | 100% | 318ms | 80% | $0 |
+| Groq Llama 3.3 70B | 100% | 315ms | 67% | $0 |
+| Groq Qwen 3 32B | 100% | 535ms | N/A | $0 |
+| Cerebras Llama 3.1 8B | 100% | ~300ms | 33% | $0 |
+| Cerebras Qwen 3 235B | 100% | ~400ms | 40% | $0 |
+
+### Cost Savings (Real Provider Pricing)
+
+| Tier | Routed To | Cost/1M tokens |
+|------|-----------|:---------------:|
+| Free (~50%) | Groq, DeepSeek, Fireworks | $0 |
+| Cheap (~35%) | Llama, Mistral, Qwen | $0.05-$0.60 |
+| Mid (~10%) | GPT-4o-mini, Claude Haiku | $0.15-$0.80 |
+| Premium (~5%) | GPT-4o, Claude 3.5 | $2.50-$3.00 |
+
+**Real-world savings: 61.6% vs all-premium routing** (benchmark) / **64%** (detailed cost model)
+
+Run benchmarks yourself:
+```bash
+node scripts/routing-benchmark-v2.js  # Routing accuracy
+node scripts/run-mmlu-benchmark.js     # Provider quality
+node scripts/run-provider-benchmark.js  # Latency & throughput
+```

package/benchmark-provider-results.json

@@ -0,0 +1,166 @@
+{
+  "meta": {
+    "date": "2026-05-22T06:48:22.431Z",
+    "questions": 10
+  },
+  "results": {
+    "groq-llama-3.3-70b": {
+      "name": "Groq Llama 3.3 70B",
+      "successRate": 1,
+      "avgLatency": 315,
+      "avgOutputLen": 279,
+      "inputCostPer1M": 0,
+      "questions": [
+        {
+          "id": 1,
+          "success": true,
+          "latency": 173,
+          "content": "2 + 2 = 4.",
+          "status": 200
+        },
+        {
+          "id": 2,
+          "success": true,
+          "latency": 444,
+          "content": "**Checking for Prime Numbers in Python**\n======================================\n\nBelow is a simple Python function that checks if a given number is prime. A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself.\n\n### Code\n\n```python\ndef is_prime(n):\n    \"\"\"\n    Checks if a number is prime.\n\n    Args:\n        n (int): The number to check.\n\n    Returns:\n        bool: True if the number is prime, False otherwise.\n",
+          "status": 200
+        },
+        {
+          "id": 3,
+          "success": true,
+          "latency": 217,
+          "content": "An API, or Application Programming Interface, is a set of defined rules and protocols that allows different software systems, applications, or services to communicate and exchange data with each other in a standardized and secure manner.",
+          "status": 200
+        },
+        {
+          "id": 4,
+          "success": true,
+          "latency": 407,
+          "content": "Climate change is a complex and multifaceted issue, and there are several factors that contribute to it. Some of the main causes of climate change include:\n\n1. **Greenhouse gases**: The burning of fossil fuels such as coal, oil, and gas for energy releases large amounts of carbon dioxide (CO2), methane (CH4), and other greenhouse gases into the atmosphere. These gases trap heat from the sun, leading to a warming effect on the planet.\n2. **Deforestation**: The",
+          "status": 200
+        },
+        {
+          "id": 5,
+          "success": true,
+          "latency": 313,
+          "content": "Code flows like a stream\nLogic dances on the screen\nBeauty in the code",
+          "status": 200
+        },
+        {
+          "id": 6,
+          "success": true,
+          "latency": 403,
+          "content": "The quality and performance of AI models directly depend on the amount and quality of the data used to train them. Here's a summary:\n\n**Key Points:**\n\n1. **Data quantity**: More data generally leads to better model performance, as the model can learn from a larger number of examples.\n2. **Data quality**: High-quality data that is accurate, relevant, and diverse is crucial for training effective AI models.\n3. **Data representation**: The way data is represented and formatted can impact model",
+          "status": 200
+        },
+        {
+          "id": 7,
+          "success": true,
+          "latency": 158,
+          "content": "1. Python\n2. Java\n3. JavaScript",
+          "status": 200
+        },
+        {
+          "id": 8,
+          "success": true,
+          "latency": 480,
+          "content": "**Machine Learning: An Overview**\n=====================================\n\nMachine learning (ML) is a subset of artificial intelligence (AI) that involves the use of algorithms and statistical models to enable machines to learn from data, make predictions, and improve their performance over time.\n\n**Key Aspects of Machine Learning:**\n\n1. **Data**: Machine learning relies on large amounts of data to learn and make predictions.\n2. **Algorithms**: ML algorithms are used to analyze and process the data, extracting patterns and relationships",
+          "status": 200
+        },
+        {
+          "id": 9,
+          "success": true,
+          "latency": 417,
+          "content": "**Reversing a String in Python**\n================================\n\nYou can reverse a string in Python using slicing or the `reversed()` function. Here are examples of both methods:\n\n### Method 1: Slicing\n\n```python\ndef reverse_string_slicing(input_string):\n    \"\"\"Reverse a string using slicing.\"\"\"\n    return input_string[::-1]\n\n# Example usage:\ninput_str = \"Hello, World!\"\nreversed_str = reverse_string_slicing(input_str)\nprint(reversed_str)  # Output",
+          "status": 200
+        },
+        {
+          "id": 10,
+          "success": true,
+          "latency": 138,
+          "content": "The capital of Japan is Tokyo.",
+          "status": 200
+        }
+      ]
+    },
+    "groq-qwen-3-32b": {
+      "name": "Groq Qwen 3 32B",
+      "successRate": 1,
+      "avgLatency": 535,
+      "avgOutputLen": 456,
+      "inputCostPer1M": 0,
+      "questions": [
+        {
+          "id": 1,
+          "success": true,
+          "latency": 488,
+          "content": "<think>\nOkay, so I need to figure out what 2 plus 2 is. Let me start by recalling basic addition. When you add two numbers together, you're combining their quantities. So, 2 plus 2 would be combining two groups of two objects each. If I have two apples and someone gives me two more, how many apples do I have in total? That's 4 apples. \n\nWait, but maybe I should break it down even more. Let's use my",
+          "status": 200
+        },
+        {
+          "id": 2,
+          "success": true,
+          "latency": 622,
+          "content": "<think>\nOkay, I need to write a Python function to check if a number is prime. Hmm, let's think about how to approach this. A prime number is a number greater than 1 that has no divisors other than 1 and itself. So first, I should handle the edge cases. Like, if the number is less than 2, it's definitely not prime. For numbers 2 and up, I need to check divisibility.\n\nWait, for a number n,",
+          "status": 200
+        },
+        {
+          "id": 3,
+          "success": true,
+          "latency": 725,
+          "content": "<think>\nOkay, the user wants a one-sentence explanation of an API. Let me start by recalling what an API is. An API, or Application Programming Interface, allows software applications to communicate with each other. It's a set of protocols and tools that define how different software components should interact.\n\nWait, maybe I should structure it clearly. The key points are: definition, purpose, and function. API acts as an intermediary. It enables different systems to talk to each other. So combining that",
+          "status": 200
+        },
+        {
+          "id": 4,
+          "success": true,
+          "latency": 615,
+          "content": "<think>\nOkay, I need to figure out what causes climate change. Let me start by recalling what I know. Climate change refers to long-term changes in temperature, weather patterns, and other aspects of the Earth's climate. But why does this happen?\n\nFirst, I remember that the greenhouse effect is involved. The greenhouse effect is when certain gases trap heat in the atmosphere. But wait, isn't the greenhouse effect natural? I think so. Without it, Earth would be too cold for life.",
+          "status": 200
+        },
+        {
+          "id": 5,
+          "success": true,
+          "latency": 313,
+          "content": "<think>\nOkay, I need to write a haiku about programming. Let me start by recalling what a haiku is. It's a traditional Japanese poem with three lines. The syllable structure is 5-7-5. So first line 5 syllables, second 7, third 5.\n\nThe topic is programming. So I need to think of programming-related words and concepts. Let me brainstorm some terms: code, debug, logic, syntax, loops, algorithms, debug,",
+          "status": 200
+        },
+        {
+          "id": 6,
+          "success": true,
+          "latency": 518,
+          "content": "<think>\nOkay, the user wants me to summarize \"AI models improve with data\". Let me start by breaking down the main idea.\n\nFirst, I need to explain the relationship between data and AI models. More data usually leads to better performance because the model can learn patterns more effectively. But I should also mention that quality matters as much as quantity. Just having a lot of data isn't enough if it's noisy or biased.\n\nWait, should I mention types of data? Like, maybe structured vs",
+          "status": 200
+        },
+        {
+          "id": 7,
+          "success": true,
+          "latency": 825,
+          "content": "<think>\nOkay, the user is asking for a list of three programming languages. Let me start by recalling the most commonly used ones. Python is definitely up there because of its simplicity and wide range of applications, from web development to data science. Then Java, which is popular for enterprise applications and Android development. JavaScript is another must-include since it's essential for front-end web development and increasingly used for back-end with Node.js. I should mention each briefly to give a quick overview of their uses",
+          "status": 200
+        },
+        {
+          "id": 8,
+          "success": true,
+          "latency": 416,
+          "content": "<think>\nOkay, the user is asking, \"What is machine learning?\" I need to start by breaking this down from the ground up. First, I should define machine learning in simple terms without using too much jargon. Maybe start by mentioning it's a subset of AI. \n\nI should explain how traditional programming versus machine learning works. In traditional programming, we write explicit instructions for a computer. In machine learning, the computer learns patterns from data instead of following exact commands. That's a crucial",
+          "status": 200
+        },
+        {
+          "id": 9,
+          "success": true,
+          "latency": 415,
+          "content": "<think>\nOkay, I need to figure out how to reverse a string in Python. Let me think... Hmm, I remember that strings in Python are immutable, so maybe I can't modify them directly. But there must be ways to create a reversed version. Let me recall.\n\nOh right, slicing! Python has this slicing syntax that can reverse a string. The slice is something like [::-1]. Wait, how does that work exactly? Let me try an example. If I have a string",
+          "status": 200
+        },
+        {
+          "id": 10,
+          "success": true,
+          "latency": 416,
+          "content": "<think>\nOkay, so I need to figure out the capital of Japan. Hmm, I remember that Japan has four main islands, like Honshu, Hokkaido, Kyushu, and Shikoku. But the capital... I think it's Tokyo, right? Wait, but I've also heard about Kyoto. Wasn't Kyoto the capital at some point? Maybe during the Edo period?\n\nLet me think. Japan's history has different capitals over time. They had Kyoto",
+          "status": 200
+        }
+      ]
+    }
+  }
+}

package/benchmark-results.json

@@ -1,6 +1,6 @@
 {
   "meta": {
-    "date": "2026-05-19T19:09:06.916Z",
+    "date": "2026-05-22T06:44:38.534Z",
     "type": "mmlu_quality"
   },
   "results": {
@@ -130,7 +130,7 @@
     },
     "groq-llama-3.3-70b": {
       "name": "Groq Llama 3.3 70B",
-      "mmlu_accuracy": 0.8,
+      "mmlu_accuracy": 0.6666666666666666,
       "quality_score": 9.4,
       "mmlu_detail": [
         {
@@ -145,8 +145,8 @@
         },
         {
           "q": "Time complexity of binary search?",
-          "response": "B",
-          "correct": true
+          "response": "B)",
+          "correct": false
         },
         {
           "q": "In Python, list.append() does what?",
@@ -170,8 +170,8 @@
         },
         {
           "q": "Powerhouse of the cell?",
-          "response": "C",
-          "correct": true
+          "response": "C)",
+          "correct": false
         },
         {
           "q": "DNA stands for?",
@@ -242,7 +242,7 @@
         },
         {
           "prompt": "What is machine learning?",
-          "response": "**Machine Learning: An Overvie",
+          "response": "**Machine Learning Overview**\n",
           "score": 10
         },
         {
@@ -379,7 +379,7 @@
     "cerebras-llama3.1-8b": {
       "name": "Cerebras Llama 3.1 8B",
       "mmlu_accuracy": 0.3333333333333333,
-      "quality_score": 1.3,
+      "quality_score": 0.6,
       "mmlu_detail": [
         {
           "q": "What is the derivative of x²?",
@@ -465,8 +465,8 @@
         },
         {
           "prompt": "Write a Python function that c",
-          "response": "**Palindrome Checker Function*",
-          "score": 10
+          "response": "",
+          "score": 0
         },
         {
           "prompt": "Compare supervised vs unsuperv",
@@ -485,8 +485,8 @@
         },
         {
           "prompt": "All cats are animals. Some ani",
-          "response": "",
-          "score": 0
+          "response": "To determine if some cats are ",
+          "score": 5
         },
         {
           "prompt": "What is machine learning?",
@@ -502,8 +502,8 @@
     },
     "cerebras-qwen-3-235b": {
       "name": "Cerebras Qwen 3 235B",
-      "mmlu_accuracy": 0.3333333333333333,
-      "quality_score": 1.3,
+      "mmlu_accuracy": 0.4,
+      "quality_score": 0,
       "mmlu_detail": [
         {
           "q": "What is the derivative of x²?",
@@ -542,8 +542,8 @@
         },
         {
           "q": "Powerhouse of the cell?",
-          "response": "",
-          "correct": false
+          "response": "C",
+          "correct": true
         },
         {
           "q": "DNA stands for?",
@@ -599,8 +599,8 @@
         },
         {
           "prompt": "What is the capital of France?",
-          "response": "The capital of France is Paris",
-          "score": 10
+          "response": "",
+          "score": 0
         },
         {
           "prompt": "Train travels 300km in 4 hours",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "adaptive-memory-multi-model-router",
-  "version": "2.5.2",
+  "version": "2.5.3",
   "shortName": "A3M Router",
   "displayName": "A3M Router - Adaptive Memory Multi-Model Router",
   "description": "LLM router & AI gateway — 99.5% routing accuracy, 47 providers (DeepSeek, Kimi/Moonshot, Qwen, Zhipu GLM, Yi + more). Semantic cache, guardrails, cost analytics. Built on 30+ arXiv papers (SGLang, Medusa, MemoRAG). Zero ML, 19.5KB. TypeScript + Python SDK. MIT.",