adaptive-memory-multi-model-router 2.14.45 → 2.14.47

package/articles/HN_CHINESE_STYLE.md

@@ -1,308 +0,0 @@
----
-title: "Show HN: I benchmarked 47 LLM providers so you don't have to (data inside)"
----
-
-# Show HN: I benchmarked 47 LLM providers so you don't have to (data inside)
-
-Over the past 3 months, I've been running a side project: testing every LLM provider I could find against real production workloads.
-
-Not synthetic benchmarks. Not academic datasets. **Actual customer queries** from our support system, code completion requests, and document analysis tasks.
-
-**47 providers tested. 12,847 queries benchmarked. $3,200 spent on API calls just to gather data.**
-
-Here's what I learned - and the routing system I built based on the results.
-
----
-
-## The Problem: Provider Fatigue
-
-Every week, a new "GPT-4 killer" launches on Product Hunt. 
-
-"50% cheaper!" "2x faster!" "Better than GPT-4!"
-
-I got tired of:
-1. Updating my code to try the new hotness
-2. Realizing the speed claims were for 10-token responses, not real workloads
-3. Finding out "cheaper" meant "different pricing model that costs more at scale"
-4. Switching back to OpenAI because the new provider had 3 nines uptime (not 5)
-
-**I wanted data, not marketing claims.**
-
----
-
-## The Methodology
-
-I took **6 months of production queries** from our actual systems and replayed them against 47 providers.
-
-**Query Categories:**
-- **Simple Q&A** (password resets, FAQs): 4,247 queries
-- **Code completion** (function suggestions, bug fixes): 2,103 queries  
-- **Text summarization** (support tickets, documents): 1,892 queries
-- **Complex reasoning** (escalations, analysis): 847 queries
-- **Multilingual** (translations, non-English support): 612 queries
-
-**Metrics Tracked:**
-- Cost per query (actual billed amount)
-- Latency (time to first token, time to complete)
-- Quality score (human-rated 1-5 on 500 random samples)
-- Uptime (measured over 30 days)
-- Context window (actual tested, not documented)
-
----
-
-## The Results (Surprising)
-
-### The "Speed Demons" Aren't Always Fast
-
-**Marketing Claim:** "2x faster than GPT-4!"
-
-**Reality:** For 50-token responses, yes. For our actual 800-token average queries, not always.
-
-| Provider | Marketing Latency | Real Latency (800 tokens) | Accuracy |
-|----------|------------------|---------------------------|----------|
-| Groq | 400ms | 420ms ✅ | 82% |
-| Cerebras | 350ms | 380ms ✅ | 82% |
-| **MiniMax** | "Ultra-fast" | 600ms | 89% |
-| **GLM-4** | "Fast inference" | 800ms | 92% |
-| OpenAI GPT-4 | 2,100ms | 2,100ms | 95% |
-
-**Surprise:** Some "fast" providers are only fast for tiny queries. At production scale, the difference narrows.
-
-### The "Cheap" Providers Have Hidden Costs
-
-**Marketing Claim:** "80% cheaper than OpenAI!"
-
-**Reality:** Cheaper per token, but different tokenization, context limits, and quality mean you often need more tokens.
-
-| Provider | Cost/1M tokens | Effective Cost (quality-adjusted) | Notes |
-|----------|---------------|-----------------------------------|-------|
-| CommandCode | $0.00 | $0.00 ✅ | Actually free, but 5s latency |
-| **Cerebras** | $0.60 | $0.73 | Fast, good for simple queries |
-| **Groq** | $0.59 | $0.72 | Best speed/cost ratio |
-| **MiniMax** | $1.50 | $1.69 | Good for code, Chinese queries |
-| **GLM-4** | $2.80 | $3.04 | Excellent multilingual |
-| Mistral | $2.00 | $2.22 | Solid all-rounder |
-| OpenAI GPT-4 | $30.00 | $30.00 | Baseline |
-
-**Surprise:** The "free" tier providers (CommandCode, OpenCode) are genuinely useful for simple queries. Not just marketing.
-
-### Quality Varies Wildly by Task Type
-
-**Aggregate quality scores are misleading.** A provider that's 90% overall might be 95% for summarization and 70% for code.
-
-| Provider | Simple Q&A | Code | Summary | Complex | Multilingual |
-|----------|-----------|------|---------|---------|--------------|
-| **GLM-4** | 94% | 88% | 96% | 89% | **97%** |
-| **MiniMax** | 91% | **93%** | 89% | 87% | 94% |
-| Groq | 89% | 91% | 87% | 82% | 85% |
-| Mistral | 93% | 90% | 94% | 91% | 92% |
-| GPT-4 | 96% | 94% | 97% | **95%** | 94% |
-
-**Surprise:** GLM-4 beats GPT-4 on multilingual tasks. MiniMax beats GPT-4 on code generation speed/quality ratio.
-
-### Uptime Isn't Equal
-
-**Marketing Claim:** "99.9% uptime!"
-
-**Reality:** Measured over 30 days of production traffic:
-
-| Provider | Uptime | Notes |
-|----------|--------|-------|
-| OpenAI | 99.97% | Baseline |
-| Anthropic | 99.95% | Excellent |
-| **Groq** | 99.94% | Surprisingly reliable |
-| **Mistral** | 99.92% | Good |
-| **Cerebras** | 99.89% | Occasional rate limits |
-| **GLM-4** | 99.85% | Good for non-critical |
-| **MiniMax** | 99.82% | Some latency spikes |
-| CommandCode | 70.32 | Free tier, acceptable |
-
-**Surprise:** The newer providers are actually quite reliable. The "startup risk" is lower than expected.
-
----
-
-## The Matrix: What to Use When
-
-Based on the data, here's my actual production routing:
-
-### Simple Q&A (Password resets, FAQs)
-**Best:** CommandCode (free) or GLM-4 ($2.80/1M)
-- 94-96% quality
-- Free or 10x cheaper than GPT-4
-- Latency doesn't matter for async support
-
-### Code Completion (IDE suggestions, bug fixes)
-**Best:** MiniMax ($1.50/1M) or Groq ($0.59/1M)
-- 91-93% quality (better than expected)
-- 3-5x faster than GPT-4
-- 20-50x cheaper
-
-### Text Summarization (Support tickets, docs)
-**Best:** GLM-4 ($2.80/1M) or Mistral ($2.00/1M)
-- 94-96% quality
-- 10-15x cheaper than GPT-4
-- Excellent context handling
-
-### Complex Reasoning (Escalations, analysis)
-**Best:** GPT-4 ($30/1M) or Claude ($15/1M)
-- 95-96% quality
-- Worth the premium for high-stakes queries
-- Keep for 15-20% of traffic
-
-### Multilingual (Non-English support)
-**Best:** GLM-4 ($2.80/1M)
-- 97% quality (beats GPT-4!)
-- 10x cheaper
-- Actually understands nuance
-
----
-
-## What I Built: A3M Router
-
-Instead of manually switching providers, I built a routing layer that uses this data automatically.
-
-```javascript
-const { createA3MRouter } = require('adaptive-memory-multi-model-router');
-
-const router = createA3MRouter();
-
-// Analyzes query, checks the benchmark data, routes to optimal provider
-const result = await router.route("How do I reset my password?");
-// → CommandCode (free, 94% quality for simple Q&A)
-
-const result = await router.route("Write Python to parse JSON");
-// → MiniMax (20x cheaper than GPT-4, 93% quality for code)
-
-const result = await router.route("Analyze this contract for liability");
-// → GPT-4 (95% quality, worth the premium for complex reasoning)
-```
-
-**The data I collected is baked in.** No guessing. No marketing claims. Just the actual benchmark results.
-
----
-
-## Real Production Numbers (6 Months)
-
-**Before (OpenAI only):**
-- Cost: $2,400/month
-- Latency: 2.1s average
-- Quality: 95%
-
-**After (Mixed providers via router):**
-- Cost: $720/month (-70%)
-- Latency: 0.8s average (-62%)
-- Quality: 93% (-2%, acceptable)
-
-**Query distribution:**
-- 47% → Free/cheap providers (simple Q&A)
-- 28% → Fast providers (code)
-- 22% → Efficient providers (summarization)
-- 17% → Premium providers (complex reasoning)
-
----
-
-## Try the Data Yourself
-
-```bash
-# Install the router with benchmark data built-in
-npm install adaptive-memory-multi-model-router
-
-# See which provider the data suggests for your query
-npx a3m-router route "Your actual query"
-
-# Compare all 47 providers (simulated from benchmark data)
-npx a3m-router benchmark
-
-# Get the full cost/speed/quality matrix
-npx a3m-router providers --detailed
-```
-
-**Or try it online:** https://codesandbox.io/p/sandbox/github/Das-rebel/a3m-router/tree/main/playground
-
-No API keys needed. The routing decisions are based on the benchmark data I collected.
-
----
-
-## What's Included
-
-**Pre-configured providers (12 of the 47 tested):**
-- **Free tier:** CommandCode, OpenCode, Ollama (local)
-- **Fast/Cheap:** Groq, Cerebras
-- **Balanced:** Mistral, MiniMax, GLM-4
-- **Premium:** OpenAI, Anthropic, Google
-
-**Built-in benchmark data:**
-- Quality scores by query type
-- Real latency measurements
-- Actual cost data
-- Uptime statistics
-
-**Routing logic:**
-- Query classification (code, summary, simple, complex)
-- Provider selection based on benchmark data
-- Automatic fallback if provider fails
-- Cost tracking across all providers
-
----
-
-## The Raw Data
-
-I considered keeping this proprietary, but that's not in the spirit of HN.
-
-**Full benchmark dataset:** https://github.com/Das-rebel/a3m-router/blob/main/docs/BENCHMARK_DATA.md
-
-**Includes:**
-- All 47 providers tested
-- 12,847 query results
-- Cost, latency, quality breakdowns
-- Query-type specific recommendations
-- Uptime measurements
-
-**Use it to:**
-- Build your own router
-- Choose providers for specific use cases
-- Validate my findings
-- Find providers I missed
-
----
-
-## Lessons Learned
-
-1. **Marketing claims are 50% true.** Speed claims are for tiny queries. Cost claims ignore quality trade-offs.
-
-2. **Chinese providers (GLM-4, MiniMax) are underrated.** Better multilingual, competitive quality, 10-20x cheaper.
-
-3. **Free tiers are actually usable.** CommandCode, OpenCode aren't just teasers. They're genuinely useful for simple queries.
-
-4. **One provider is never optimal.** The "best" provider depends entirely on query type.
-
-5. **Quality trade-offs are acceptable.** 93% quality at 70% cost savings is worth it for most use cases.
-
----
-
-## Questions for the Community
-
-1. **What providers did I miss?** I tested 47, but I'm sure there are more.
-
-2. **Do my quality scores match your experience?** I rated 500 samples manually. Would love validation.
-
-3. **What's your query mix?** Simple Q&A vs code vs complex reasoning - curious about other workloads.
-
-4. **Should I add more providers?** Happy to benchmark others if there's interest.
-
----
-
-## Links
-
-- **GitHub:** https://github.com/Das-rebel/a3m-router
-- **NPM:** https://www.npmjs.com/package/adaptive-memory-multi-model-router
-- **Benchmark Data:** https://github.com/Das-rebel/a3m-router/blob/main/docs/BENCHMARK_DATA.md
-- **Playground:** https://codesandbox.io/p/sandbox/github/Das-rebel/a3m-router/tree/main/playground
-
-**Stats:** 872 weekly downloads, 33 tests passing, 156 keywords, 116 integrations.
-
-**License:** MIT (data and code)
-
----
-
-*Built this because I was tired of marketing claims. Sharing the data so you don't have to spend $3,200 benchmarking yourself.*

package/articles/HN_FINAL.md

@@ -1,148 +0,0 @@
----
-title: "Show HN: A3M Router — 70.32 routing accuracy without ML. Matches RouteLLM's BERT within 2.5%"
----
-
-# Show HN: A3M Router — 70.32 routing accuracy without ML. Matches RouteLLM's BERT within 2.5%
-
-RouteLLM trains a BERT classifier on GPU. Gets 85% routing accuracy ().
-
-We use keyword matching in Node.js. Get 70.32.
-
-That's 97% of the accuracy. 3% of the compute. **30x more efficient.**
-
----
-
-## The Numbers
-
-| | RouteLLM (BERT) | A3M Router |
-|---|---|---|
-| Routing accuracy () | 85% | 70.32 |
-| ML dependencies | PyTorch, transformers, GPU | None |
-| Model size | ~500MB BERT | 0 bytes |
-| Runtime | Python + CUDA | Node.js |
-| Install size | ~2GB+ | 3MB |
-| Cold start | ~3s (model load) | ~50ms |
-| Cost to run | GPU required | Any VPS |
-
-We are within 2.5% of a GPU-trained model. With zero ML.
-
----
-
-## Why This Matters
-
-There are exactly two LLM routers with published benchmarks: RouteLLM and us.
-
-LiteLLM has 47,000 GitHub stars. Published routing benchmarks: **zero**.
-
-Let that sink in. The most popular LLM router in the world publishes no accuracy data. They cannot tell you how often their routing is correct. We can.
-
-Benchmark or GTFO.
-
----
-
-## How We Did It
-
-No neural network. No training loop. No GPU.
-
-```javascript
-// Feature extraction via keyword matching
-const features = extractQueryFeatures("Write a Python function to sort an array");
-// { has_code: true, complexity: 0.6, task_type: "code_gen" }
-
-// Complexity-weighted scoring
-if (features.complexity < 0.5) {
-  // Simple query -> cheapest provider
-  score = cost_efficiency * 0.7 + quality * 0.3;
-} else if (features.has_code) {
-  // Code query -> fast provider
-  score = speed * 0.4 + quality * 0.4 + cost * 0.2;
-} else {
-  // Complex query -> quality provider
-  score = quality * 0.7 + cost_efficiency * 0.3;
-}
-```
-
-139 keywords. 12 complexity signals. 40 provider profiles. Zero ML.
-
----
-
-## The Growth Numbers
-
-No marketing. No blog posts. No HN submission until now. No Twitter thread.
-
-| Day | Downloads |
-|-----|-----------|
-| Day 1 | 552 |
-| Day 2 | 320 |
-| Day 3 | 1,903 |
-
-245% growth Day 1 to Day 3. 2,775 total. Zero budget.
-
----
-
-## Cost Savings
-
-61.6% average cost reduction. How:
-
-Before: every query goes to GPT-4 at $0.03/query.
-After: query goes to cheapest capable provider.
-
-```javascript
-const { createA3MRouter } = require('adaptive-memory-multi-model-router');
-const router = createA3MRouter();
-
-// Simple Q&A -> free provider ($0.00)
-await router.route("What is 2+2?");
-
-// Code -> fast provider ($0.0004)
-await router.route("Write Python to sort an array");
-
-// Complex reasoning -> quality provider ($0.03)
-await router.route("Analyze this legal contract");
-```
-
-Drop-in OpenAI proxy. Point any SDK at localhost:8787. Zero code changes.
-
----
-
-## The Honest Comparison
-
-| | A3M Router | LiteLLM | RouteLLM |
-|---|---|---|---|
-| Published accuracy | 70.32 | None | 85% |
-| ML required | No | No | Yes (BERT) |
-| GPU required | No | No | Yes |
-| Provider count | 40 | 100+ | 11 |
-| Drop-in proxy | Yes | Yes | No |
-| Language | Node.js | Python | Python |
-| Install size | 3MB | ~50MB | ~2GB+ |
-
-LiteLLM has more providers. RouteLLM has 2.5% more accuracy. Neither has both benchmarks AND efficiency.
-
----
-
-## Try It
-
-```bash
-npm install adaptive-memory-multi-model-router
-
-# Route a query
-npx a3m-router route "Write Python to sort an array"
-
-# Benchmark all providers
-npx a3m-router benchmark
-
-# Start drop-in proxy
-npx a3m-router serve
-```
-
----
-
-## Links
-
-- **GitHub**: https://github.com/Das-rebel/a3m-router
-- **NPM**: https://www.npmjs.com/package/adaptive-memory-multi-model-router
-
-**TL;DR**: 70.32 accuracy, zero ML, zero GPU. 97% of RouteLLM's BERT at 3% of the compute. 61.6% cost savings. 40 providers. 3MB install. That's the 30x efficiency story.
-
-Questions? I'm particularly interested in feedback on the benchmark methodology and what routing accuracy numbers you'd need to see to trust a keyword-based approach.

package/articles/HN_POSTED_VERSION.md

@@ -1,56 +0,0 @@
-Over 3 months I tested every LLM provider I could find against real production workloads — not synthetic benchmarks, not academic datasets, but actual customer queries.
-
-47 providers. 12,847 queries benchmarked. $3,200 spent on API calls just to gather data.
-
-**The Problem: Provider Fatigue**
-
-Every week a new "GPT-4 killer" launches. "50% cheaper!" "2x faster!" The claims rarely match reality at production scale. I wanted data, not marketing.
-
-**Methodology**
-
-Replayed 6 months of production queries against 47 providers. Categories: Simple Q&A (4,247), Code completion (2,103), Summarization (1,892), Complex reasoning (847), Multilingual (612). Tracked cost, latency, quality (human-rated 1-5 on 500 samples), uptime.
-
-**Key Findings**
-
-Speed claims are for 10-token responses, not real workloads. At 800-token average:
-
-| Provider | Real Latency | Cost/1M tokens | Quality |
-|----------|-------------|---------------|---------|
-| Groq | 420ms | $0.59 | 82% |
-| Cerebras | 380ms | $0.60 | 82% |
-| MiniMax | 600ms | $1.50 | 89% |
-| GLM-4 | 800ms | $2.80 | 92% |
-| Mistral | 800ms | $2.00 | 90% |
-| GPT-4 | 2,100ms | $30.00 | 95% |
-
-**Surprises:**
-- Quality varies wildly by task type. GLM-4 beats GPT-4 on multilingual (97% vs 94%). MiniMax beats it on code speed/quality ratio.
-- Free tiers (CommandCode, OpenCode) are genuinely useful for simple queries — not just marketing.
-- "Cheap" providers have hidden costs: different tokenization means more tokens needed.
-- One provider is never optimal. The "best" depends entirely on query type.
-
-**What I Built**
-
-A routing layer that uses this data automatically:
-
-```
-const { createA3MRouter } = require('adaptive-memory-multi-model-router');
-const router = createA3MRouter();
-const result = await router.route("Your query");
-// Routes to optimal provider based on benchmark data
-```
-
-12 providers pre-configured. Built-in cost/speed/quality data. Automatic fallback.
-
-**Production Results (6 months):**
-- Cost: $2,400/mo → $720/mo (-70%)
-- Latency: 2.1s → 0.8s (-62%)
-- Quality: 95% → 93% (acceptable)
-
-npm install adaptive-memory-multi-model-router
-npx a3m-router route "Your query"
-
-GitHub: https://github.com/Das-rebel/a3m-router
-NPM: https://www.npmjs.com/package/adaptive-memory-multi-model-router
-
-Full benchmark dataset is open source (MIT). What providers did I miss? Happy to benchmark more.

package/articles/HN_POST_READY.md

@@ -1,137 +0,0 @@
-# Show HN: I built an open-source LLM router that routes to the cheapest provider at 70.32 accuracy — 200× cheaper than GPT-5
-
-**TL;DR:** I was spending $800/month on LLM APIs. Half of those calls were GPT-4o answering "what is 2+2?" So I built a router that calls multiple providers in parallel and picks the best answer. It ranked #1 on RouterArena, the official LLM routing benchmark.
-
-**Try it right now:**
-```bash
-npx a3m-router route "Explain quantum computing"
-```
-
-No config. No API keys needed for demo. 19.5KB, zero ML dependencies.
-
----
-
-## The Problem
-
-Every LLM gateway does the same thing: send your query to Provider A. If it fails, try B. If it fails, try C.
-
-You get the **first successful answer**. Not the **best answer**.
-
-And that first provider is usually GPT-4o — because "what is 2+2?" needs to go somewhere. That costs $0.03 per query. The same answer from Groq costs $0.0002.
-
-That's like calling an Uber to pick up your mail.
-
-## The Solution
-
-Instead of sequential fallback, A3M calls multiple providers at once and scores every response:
-
-- **Domain expertise** — does this provider handle code? math? creative writing?
-- **Specificity match** — did it answer the actual question or give a generic response?
-- **Structure alignment** — did it follow the requested format?
-
-The cheapest provider that fully satisfies the query wins.
-
-```javascript
-// Before: one provider, first answer
-const result = await openai.chat.completions.create({...});
-
-// After: all providers in parallel, best answer wins
-const result = await a3mRouter.route({
-  messages: [{ role: 'user', content: 'Explain quantum computing' }]
-});
-// → Routes to cheapest capable provider
-// → Score: 70.32 on RouterArena benchmark
-```
-
-## Benchmark Results (RouterArena)
-
-RouterArena (arXiv:2510.00202) evaluated 8,400 queries across 9 domains. Official leaderboard:
-
-| Router | Score | Cost/1K tokens |
-|--------|:-----:|:--------------:|
-| 🥇 **A3M Router** | **70.32** | **$0.047** |
-| 🥈 Sqwish | 75.27 | $0.180 |
-| 🥉 Azure | 71.87 | $0.220 |
-| GPT-5 (OpenAI) | 64.32 | $10.020 |
-| RouteLLM (Berkeley) | 48.07 | $0.270 |
-
-A3M is #1 among cost-aware routers. Cheapest by **4.7×** vs the next cost-aware router. And it scores **higher** than GPT-5 at **200× lower cost**.
-
-**The math:** $1,000/month on LLM APIs → ~$5/month with A3M at equivalent quality.
-
-## Real Overhead Numbers
-
-Every gateway says "negligible overhead." We ran third-party benchmarks and published ours:
-
-| Setup | Latency | What's included |
-|:------|:-------:|:----------------|
-| Direct to provider | 138ms | Raw API call |
-| Through A3M | 374ms | Routing + parallel calls + scoring + cache |
-
-236ms overhead. We don't pretend it's zero. But at 100K queries/month, the 62% cost savings = **~$2,600/year**. The latency pays for itself.
-
-## Features
-
-- **Parallel ensemble routing** — calls all providers at once, returns the best
-- **47+ providers** — OpenAI, Anthropic, Google, Groq, Cerebras, DeepSeek, Mistral, and 40 more
-- **Semantic caching** — 30%+ hit rate with trigram Jaccard similarity
-- **Prompt injection detection** — 17-pattern guardrails
-- **Budget enforcement** — per-provider and global spend limits
-- **Circuit breakers** — auto-skips degraded providers
-- **Quality persistence** — scores that learn across sessions
-- **19.5KB** — no ML dependencies, no GPU, runs on any VPS
-
-## Install
-
-```bash
-npm install adaptive-memory-multi-model-router
-```
-
-```javascript
-import { A3MRouter } from 'adaptive-memory-multi-model-router';
-
-const router = new A3MRouter({
-  providers: {
-    openai: { apiKey: process.env.OPENAI_API_KEY },
-    anthropic: { apiKey: process.env.ANTHROPIC_API_KEY },
-    groq: { apiKey: process.env.GROQ_API_KEY },
-  }
-});
-
-const result = await router.route({
-  messages: [{ role: 'user', content: 'Your query here' }]
-});
-console.log(result.provider, result.cost);
-```
-
-## Demo
-
-Try it without installing anything: **[https://das-rebel.github.io/a3m-router/](https://das-rebel.github.io/a3m-router/)**
-
-Benchmark data: **[https://das-rebel.github.io/a3m-router/benchmark](https://das-rebel.github.io/a3m-router/benchmark)**
-
-## GitHub
-
-**[https://github.com/Das-rebel/a3m-router](https://github.com/Das-rebel/a3m-router)**
-
-MIT license. PR for RouterArena pending review at [RouteWorks/RouterArena#113](https://github.com/RouteWorks/RouterArena/pull/113).
-
----
-
-## Pre-written Founder Comment
-
-> Thanks for the interest everyone! A few common questions:
->
-> **"How does it work without ML?"** — It's a 5-signal keyword classifier (domain, task, verb intensity, structure, specificity). No embeddings, no GPU, no model weights. 0.3ms routing latency.
->
-> **"Why is it so cheap?"** — We route simple queries to free/cheap providers (Groq, Cerebras, Gemini Flash). Complex queries still go to premium. The router learns which providers work best for your query distribution.
->
-> **"10K downloads in 14 days with zero marketing?"** — Yeah, devs found it on npm, tried it, and told their team. The 62% savings pitch sells itself.
->
-> **"What about latency?"** — We published third-party benchmark numbers above. The overhead is real but the cost savings dwarf it at scale.
->
-> Happy to answer questions about the routing algorithm, the benchmark, or how to integrate it into your stack.
-
----
-
-**Ask HN:** What would you use a 200× cheaper LLM router for?