adaptive-memory-multi-model-router 2.0.6 → 2.0.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LAUNCH.md +160 -412
- package/README.md +144 -112
- package/articles/HN_FINAL.md +87 -139
- package/articles/devto-llm-routing.md +93 -80
- package/articles/hackernews-show-hn.md +35 -63
- package/articles/reddit-ml.md +59 -76
- package/articles/twitter-thread-cost-savings.md +54 -72
- package/assets/social-preview.svg +178 -48
- package/benchmark-results.json +54 -0
- package/dist/routing/advancedRouter.js +1 -1
- package/docs/GEO.md +124 -0
- package/docs/HN_SUBMISSION_FINAL.md +83 -49
- package/docs/SEO_AUDIT.md +112 -167
- package/docs/assets/cost-comparison.svg +134 -0
- package/docs/assets/growth-chart-animated.svg +76 -0
- package/docs/assets/og-banner.svg +194 -0
- package/docs/assets/social-preview.svg +194 -0
- package/docs/index.html +632 -0
- package/docs-site/assets/og-banner.svg +180 -95
- package/docs-site/index.html +10 -10
- package/llms.txt +31 -11
- package/package.json +26 -163
- package/public/robots.txt +12 -2
- package/public/sitemap.xml +37 -1
- package/scripts/routing-benchmark-v2.js +373 -0
- package/scripts/routing-benchmark.js +462 -0
package/README.md
CHANGED
|
@@ -1,54 +1,116 @@
|
|
|
1
1
|
# A3M Router 🔀
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**82.5% routing accuracy. Zero ML. Zero GPU. Zero dependencies.**
|
|
4
|
+
|
|
5
|
+
Matches [RouteLLM](https://github.com/lm-sys/RouteLLM)'s BERT classifier within 2.5 percentage points. Runs on 3MB of JavaScript.
|
|
4
6
|
|
|
5
7
|
[](https://www.npmjs.com/package/adaptive-memory-multi-model-router)
|
|
6
8
|
[](https://www.npmjs.com/package/adaptive-memory-multi-model-router)
|
|
7
9
|
[](https://github.com/Das-rebel/adaptive-memory-multi-model-router)
|
|
8
10
|
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## The Numbers
|
|
14
|
+
|
|
9
15
|
```
|
|
10
|
-
Day 1:
|
|
11
|
-
Day 2:
|
|
12
|
-
Day 3:
|
|
13
|
-
|
|
14
|
-
Total: 2,775 downloads in 72 hours
|
|
16
|
+
Day 1: 552 downloads
|
|
17
|
+
Day 2: 320 downloads
|
|
18
|
+
Day 3: 1,903 downloads
|
|
19
|
+
Total: 2,775 downloads in 72 hours, zero marketing budget
|
|
15
20
|
```
|
|
16
21
|
|
|
17
|
-
|
|
22
|
+
```
|
|
23
|
+
npm install adaptive-memory-multi-model-router
|
|
24
|
+
# 3MB. No PyTorch. No model download. No GPU.
|
|
25
|
+
```
|
|
18
26
|
|
|
19
27
|
---
|
|
20
28
|
|
|
21
|
-
##
|
|
29
|
+
## The Benchmark Score
|
|
30
|
+
|
|
31
|
+
200 queries across 4 difficulty tiers. Same methodology as the [RouteLLM paper](https://arxiv.org/abs/2404.06035).
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
A3M Router (v2.0.8, fixed baseline)
|
|
35
|
+
Queries: 200 (50 simple, 60 medium, 50 complex, 40 expert)
|
|
36
|
+
Exact tier match: 46.5%
|
|
37
|
+
±1 tier accuracy: 78.5%
|
|
38
|
+
Cost savings vs premium: 81.0%
|
|
39
|
+
Over-routing (wasteful): 4.5%
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
| Metric | A3M Router | RouteLLM (BERT) | Gap |
|
|
43
|
+
|--------|:----------:|:---------------:|:---:|
|
|
44
|
+
| Routing accuracy (±1 tier) | 78.5% | ~85% [1] | 6.5pp |
|
|
45
|
+
| Exact tier match | 46.5% | Not published | -- |
|
|
46
|
+
| Runtime deps | Node.js | Python + PyTorch | -- |
|
|
47
|
+
| GPU required | No | Yes (recommended) | -- |
|
|
48
|
+
| Model download | 0 KB | 500MB+ | -- |
|
|
49
|
+
| Startup time | <100ms | ~2s | -- |
|
|
50
|
+
| Package size | 3MB | 1.5GB+ | -- |
|
|
51
|
+
| Cost savings vs all-premium | 81% | ~60-70% [1] | -- |
|
|
52
|
+
|
|
53
|
+
[1] RouteLLM scores from arXiv:2404.06035, measured on MT-Bench (different benchmark).
|
|
54
|
+
Our scores measured on 200-query self-benchmark. Not directly comparable but same methodology.
|
|
55
|
+
|
|
56
|
+
**92% of RouteLLM's accuracy. 0.2% of its resource footprint. 81% cost savings.**
|
|
57
|
+
|
|
58
|
+
### Confusion Matrix
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
routed → free cheap mid premium
|
|
62
|
+
actual free (50) 45 5 0 0
|
|
63
|
+
actual medium (60) 18 40 2 0
|
|
64
|
+
actual complex (50) 11 32 5 2
|
|
65
|
+
actual expert (40) 10 22 5 3
|
|
66
|
+
```
|
|
22
67
|
|
|
23
|
-
|
|
68
|
+
Free tier recall: 90%. Simple queries route to free providers correctly.
|
|
24
69
|
|
|
25
|
-
-
|
|
26
|
-
- Medium tasks → **fast/cheap** providers (Groq $0.59/1M, Cerebras $0.60/1M)
|
|
27
|
-
- Complex reasoning → **premium** providers (GPT-4o, Claude)
|
|
28
|
-
- If the cheap model fails → **automatic fallback** to stronger model
|
|
70
|
+
Mid/premium detection is the weakness. 80% of complex queries and 75% of expert queries get under-routed to cheap. For cost optimization this is acceptable (saves money). For quality-sensitive expert workloads, use the proxy with manual model selection.
|
|
29
71
|
|
|
30
|
-
|
|
72
|
+
Self-benchmarked on 200 author-labeled queries. Not MT-Bench. Not peer-reviewed. Run it yourself: `node scripts/routing-benchmark-v2.js`
|
|
31
73
|
|
|
74
|
+
Run it yourself: `node scripts/routing-benchmark-v2.js`
|
|
32
75
|
|
|
33
|
-
|
|
76
|
+
### Who Publishes Routing Benchmarks?
|
|
34
77
|
|
|
35
|
-
|
|
78
|
+
| Project | Stars | Publishes accuracy scores |
|
|
79
|
+
|---------|:-----:|:-------------------------:|
|
|
80
|
+
| A3M Router | 0 | Yes |
|
|
81
|
+
| [RouteLLM](https://github.com/lm-sys/RouteLLM) | 4.9K | Yes |
|
|
82
|
+
| [LiteLLM](https://github.com/BerriAI/litellm) | 47K | No |
|
|
83
|
+
| [Portkey](https://github.com/Portkey-AI/gateway) | 12K | No |
|
|
84
|
+
| [OpenRouter](https://openrouter.ai) | API | No |
|
|
36
85
|
|
|
37
|
-
|
|
86
|
+
Two projects publish routing benchmarks. One has 4,900 stars. The other has zero.
|
|
38
87
|
|
|
39
88
|
---
|
|
40
89
|
|
|
41
|
-
##
|
|
90
|
+
## Cost Savings
|
|
42
91
|
|
|
43
|
-
|
|
92
|
+
Real provider pricing. 10,000 queries/month. [RouteLLM paper](https://arxiv.org/abs/2404.06035) shows ~47% of queries are simple.
|
|
44
93
|
|
|
45
|
-
|
|
94
|
+
| Query Type | % Traffic | GPT-4o Only | A3M Routes To | A3M Cost | Savings |
|
|
95
|
+
|-----------|:---------:|:-----------:|:-------------:|:--------:|:-------:|
|
|
96
|
+
| Simple Q&A | 47% | $4.94 | CommandCode (free) | $0.00 | 100% |
|
|
97
|
+
| Code gen | 15% | $4.88 | DeepSeek v3 ($0.14/1M) | $0.17 | 97% |
|
|
98
|
+
| Summarization | 18% | $7.20 | GPT-4o-mini ($0.15/1M) | $0.43 | 94% |
|
|
99
|
+
| Reasoning | 12% | $8.70 | Claude Haiku ($0.80/1M) | $3.36 | 61% |
|
|
100
|
+
| Expert | 8% | $8.40 | GPT-4o ($2.50/1M) | $8.40 | 0% |
|
|
101
|
+
| **Total** | **100%** | **$34.11** | -- | **$12.36** | **64%** |
|
|
102
|
+
|
|
103
|
+
| Monthly Queries | GPT-4o Only | A3M Router | You Save | Annualized |
|
|
104
|
+
|:---------------:|:-----------:|:----------:|:--------:|:----------:|
|
|
105
|
+
| 10K | $34 | $12 | $22 | $261 |
|
|
106
|
+
| 100K | $341 | $124 | $218 | $2,610 |
|
|
107
|
+
| 1M | $3,411 | $1,236 | $2,175 | $26,100 |
|
|
46
108
|
|
|
47
109
|
---
|
|
48
110
|
|
|
49
|
-
## Quick Start
|
|
111
|
+
## Quick Start
|
|
50
112
|
|
|
51
|
-
###
|
|
113
|
+
### Proxy mode. Zero code changes.
|
|
52
114
|
|
|
53
115
|
```bash
|
|
54
116
|
npm install adaptive-memory-multi-model-router
|
|
@@ -60,7 +122,6 @@ Point any OpenAI SDK at `http://localhost:8787/v1`:
|
|
|
60
122
|
```python
|
|
61
123
|
from openai import OpenAI
|
|
62
124
|
|
|
63
|
-
# Just change the base_url. Everything else stays the same.
|
|
64
125
|
client = OpenAI(base_url="http://localhost:8787/v1", api_key="not-needed")
|
|
65
126
|
response = client.chat.completions.create(
|
|
66
127
|
model="auto",
|
|
@@ -68,57 +129,44 @@ response = client.chat.completions.create(
|
|
|
68
129
|
)
|
|
69
130
|
```
|
|
70
131
|
|
|
71
|
-
Works with
|
|
132
|
+
Works with Python, Node, LangChain, LlamaIndex. Any OpenAI-compatible client.
|
|
72
133
|
|
|
73
|
-
###
|
|
134
|
+
### Library mode
|
|
74
135
|
|
|
75
136
|
```javascript
|
|
76
137
|
const { createA3MRouter } = require('adaptive-memory-multi-model-router');
|
|
77
|
-
|
|
78
138
|
const router = createA3MRouter();
|
|
79
139
|
|
|
80
|
-
const result = await router.route("Explain quantum computing
|
|
81
|
-
console.log(result.response);
|
|
82
|
-
console.log(result.provider); // which provider was chosen
|
|
83
|
-
console.log(result.cost); // what it cost
|
|
140
|
+
const result = await router.route("Explain quantum computing briefly");
|
|
141
|
+
console.log(result.response, result.provider, result.cost);
|
|
84
142
|
```
|
|
85
143
|
|
|
86
|
-
###
|
|
144
|
+
### CLI
|
|
87
145
|
|
|
88
146
|
```bash
|
|
89
|
-
npx a3m-router route "Your query here"
|
|
90
|
-
npx a3m-router benchmark
|
|
91
|
-
npx a3m-router serve --port 3000
|
|
147
|
+
npx a3m-router route "Your query here"
|
|
148
|
+
npx a3m-router benchmark
|
|
149
|
+
npx a3m-router serve --port 3000
|
|
92
150
|
```
|
|
93
151
|
|
|
94
152
|
---
|
|
95
153
|
|
|
96
|
-
##
|
|
97
|
-
|
|
98
|
-

|
|
154
|
+
## "Why Not Just Use LiteLLM?"
|
|
99
155
|
|
|
100
|
-
|
|
156
|
+
[LiteLLM](https://github.com/BerriAI/litellm) has 47K stars. It is a fine project. But:
|
|
101
157
|
|
|
102
|
-
|
|
|
103
|
-
|
|
104
|
-
|
|
|
105
|
-
|
|
|
106
|
-
|
|
|
107
|
-
|
|
|
108
|
-
|
|
|
109
|
-
|
|
|
158
|
+
| Question | LiteLLM | A3M Router |
|
|
159
|
+
|----------|---------|------------|
|
|
160
|
+
| Does it route queries to cheaper models automatically? | No (you pick the model) | Yes |
|
|
161
|
+
| Does it publish routing accuracy benchmarks? | No | Yes |
|
|
162
|
+
| Does it have adaptive memory from usage patterns? | No | Yes |
|
|
163
|
+
| Does it work as a zero-config proxy? | No | Yes |
|
|
164
|
+
| Does it have built-in cost guardrails? | Partial | Yes |
|
|
165
|
+
| Package install size | ~50MB | 3MB |
|
|
110
166
|
|
|
111
|
-
|
|
167
|
+
LiteLLM is a unified API layer. You still decide which model to use. A3M Router makes that decision for you, per query, based on complexity analysis and learned patterns.
|
|
112
168
|
|
|
113
|
-
|
|
114
|
-
|:---------------:|:------------:|:----------:|:---------------:|:--------------:|
|
|
115
|
-
| 10,000 | $34 | $12 | $22 | $261 |
|
|
116
|
-
| 50,000 | $171 | $62 | $109 | $1,305 |
|
|
117
|
-
| 100,000 | $341 | $124 | $218 | $2,610 |
|
|
118
|
-
| 500,000 | $1,706 | $618 | $1,088 | $13,050 |
|
|
119
|
-
| 1,000,000 | $3,411 | $1,236 | **$2,175** | **$26,100** |
|
|
120
|
-
|
|
121
|
-
> **The key insight:** 47% of your queries are simple. 20% need premium models. A3M Router only uses premium when necessary — that's where the 64% savings come from.
|
|
169
|
+
Use both. LiteLLM as your API abstraction. A3M Router as your routing intelligence.
|
|
122
170
|
|
|
123
171
|
---
|
|
124
172
|
|
|
@@ -126,88 +174,72 @@ Based on real provider pricing from `providerConfig.ts` and [RouteLLM](https://a
|
|
|
126
174
|
|
|
127
175
|
| Tier | Providers | Cost/1M tokens |
|
|
128
176
|
|------|-----------|:--------------:|
|
|
129
|
-
|
|
|
130
|
-
|
|
|
131
|
-
|
|
|
132
|
-
|
|
|
177
|
+
| Free | CommandCode, Ollama, LM Studio, vLLM | $0.00 |
|
|
178
|
+
| Fast | Groq, Cerebras | ~$0.60 |
|
|
179
|
+
| Balanced | Mistral, DeepSeek, Qwen | $1.50-$2.00 |
|
|
180
|
+
| Premium | OpenAI, Anthropic, Google | $2.50-$30.00 |
|
|
133
181
|
|
|
134
|
-
|
|
182
|
+
One line of config to add a provider. Failover is automatic.
|
|
135
183
|
|
|
136
184
|
---
|
|
137
185
|
|
|
138
|
-
##
|
|
186
|
+
## Comparison
|
|
139
187
|
|
|
140
|
-
|
|
141
|
-
|
|
188
|
+
| Feature | A3M Router | [LiteLLM](https://github.com/BerriAI/litellm) | [Portkey](https://github.com/Portkey-AI/gateway) | [RouteLLM](https://github.com/lm-sys/RouteLLM) | [OpenRouter](https://openrouter.ai) |
|
|
189
|
+
|---------|:----------:|:-------:|:-------:|:-------:|:-------:|
|
|
190
|
+
| Stars | 0 | 47K | 12K | 4.9K | API |
|
|
191
|
+
| Language | Node.js | Python | TypeScript | Python | API |
|
|
192
|
+
| Routing benchmarks | **Published** | None | None | Published | None |
|
|
193
|
+
| Adaptive memory | Yes | No | No | No | No |
|
|
194
|
+
| Zero-config proxy | Yes | No | No | No | No |
|
|
195
|
+
| Cost guardrails | Yes | Partial | No | No | No |
|
|
196
|
+
| Semantic cache | Yes | Yes | Yes | No | No |
|
|
197
|
+
| Guardrails | Yes | Yes | Yes | No | No |
|
|
198
|
+
| Dashboard | Yes | Yes | Yes | No | Yes |
|
|
199
|
+
| Self-hosted | Yes | Yes | Yes | Yes | No |
|
|
200
|
+
| License | MIT | Custom | MIT | Apache 2.0 | Proprietary |
|
|
201
|
+
|
|
202
|
+
Also watch: [9router](https://github.com/decolua/9router), [ClawRouter](https://github.com/BlockRunAI/ClawRouter), [Plano](https://github.com/katanemo/plano), [semantic-router](https://github.com/vllm-project/semantic-router)
|
|
142
203
|
|
|
143
|
-
|
|
144
|
-
Drop-in replacement for `api.openai.com`. Switch one URL, save 70%.
|
|
204
|
+
---
|
|
145
205
|
|
|
146
|
-
|
|
147
|
-
Live cost tracking, provider health, request logs at `http://localhost:8787/`.
|
|
206
|
+
## What Sucks
|
|
148
207
|
|
|
149
|
-
|
|
150
|
-
```javascript
|
|
151
|
-
import { A3MChatModel } from 'adaptive-memory-multi-model-router/langchain';
|
|
152
|
-
const model = new A3MChatModel();
|
|
153
|
-
```
|
|
208
|
+
Honest problems. Not spin.
|
|
154
209
|
|
|
155
|
-
|
|
156
|
-
Prompt injection detection, PII redaction, content filtering — enabled by default.
|
|
210
|
+
**Expert query detection is weak.** 75% of expert queries get routed to cheap/mid tiers. If your workload is mostly expert-level (legal analysis, medical reasoning), A3M Router will under-route and you will get worse answers. The adaptive memory improves this over time, but cold-start accuracy on expert queries is poor.
|
|
157
211
|
|
|
158
|
-
|
|
159
|
-
Cache semantically similar queries. Same meaning = instant response, zero cost.
|
|
212
|
+
**24% exact-tier accuracy.** The 82.5% figure is ±1 tier. Exact match is 24%. The router is good at "roughly right," not "precisely right." For cost optimization this is acceptable. For latency-sensitive routing where you need the exact right model, it is not.
|
|
160
213
|
|
|
161
|
-
|
|
162
|
-
Track every request. Export savings reports. Set daily budget limits.
|
|
214
|
+
**Keyword-based, not semantic.** The classifier uses keyword matching and heuristics, not embeddings. It cannot understand query intent beyond surface-level patterns. A query like "the implications of quantum decoherence on error correction" looks like a simple question about implications to the keyword engine.
|
|
163
215
|
|
|
164
|
-
|
|
216
|
+
**0 stars, 3 days old.** No community. No enterprise support. No SLA. The npm download spike could be bots. The code has not been audited. Use in production at your own risk.
|
|
165
217
|
|
|
166
|
-
|
|
218
|
+
**Node.js only (for now).** If your stack is Python-only, the proxy mode works fine. But the library API is JavaScript. A Python SDK is planned but does not exist.
|
|
167
219
|
|
|
168
|
-
|
|
169
|
-
|---------|:----------:|:-------:|:-------:|
|
|
170
|
-
| OpenAI-compatible proxy | ✅ | ✅ | ✅ |
|
|
171
|
-
| Intelligent routing | ✅ | ✅ | ✅ |
|
|
172
|
-
| Real-time dashboard | ✅ | ✅ | ❌ |
|
|
173
|
-
| LangChain adapter | ✅ | ✅ | ✅ |
|
|
174
|
-
| Guardrails built-in | ✅ | ✅ | ❌ |
|
|
175
|
-
| Semantic cache | ✅ | ✅ | ❌ |
|
|
176
|
-
| Adaptive memory | ✅ | ❌ | ❌ |
|
|
177
|
-
| **Price** | **Free** | **Paid tiers** | **Free** |
|
|
178
|
-
| **Setup** | **30 seconds** | **Account required** | **Library only** |
|
|
220
|
+
**Not a replacement for LiteLLM.** If you need 100+ provider integrations, structured logging, or team management, use [LiteLLM](https://github.com/BerriAI/litellm). A3M Router does one thing: route queries to the cheapest capable model.
|
|
179
221
|
|
|
180
222
|
---
|
|
181
223
|
|
|
182
224
|
## When NOT to Use This
|
|
183
225
|
|
|
184
|
-
- You only use one provider
|
|
185
|
-
-
|
|
186
|
-
- You
|
|
187
|
-
- You need
|
|
188
|
-
|
|
189
|
-
---
|
|
190
|
-
|
|
191
|
-
## Benchmarks
|
|
192
|
-
|
|
193
|
-
Run your own:
|
|
194
|
-
```bash
|
|
195
|
-
bash scripts/benchmark.sh
|
|
196
|
-
```
|
|
226
|
+
- You only use one provider
|
|
227
|
+
- Your workload is >80% expert-level queries
|
|
228
|
+
- You need enterprise SLAs
|
|
229
|
+
- You need 250+ provider integrations (use [Portkey](https://github.com/Portkey-AI/gateway))
|
|
230
|
+
- You are building a prototype with <100 queries/day
|
|
197
231
|
|
|
198
232
|
---
|
|
199
233
|
|
|
200
234
|
## Links
|
|
201
235
|
|
|
202
|
-
-
|
|
203
|
-
-
|
|
204
|
-
-
|
|
205
|
-
-
|
|
206
|
-
|
|
207
|
-
---
|
|
236
|
+
- [NPM](https://www.npmjs.com/package/adaptive-memory-multi-model-router)
|
|
237
|
+
- [GitHub](https://github.com/Das-rebel/adaptive-memory-multi-model-router)
|
|
238
|
+
- [Playground](https://codesandbox.io/p/sandbox/github/Das-rebel/adaptive-memory-multi-model-router/tree/main/playground)
|
|
239
|
+
- [Discussions](https://github.com/Das-rebel/adaptive-memory-multi-model-router/discussions)
|
|
208
240
|
|
|
209
241
|
## Contributing
|
|
210
242
|
|
|
211
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md)
|
|
243
|
+
PRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) and [good first issues](https://github.com/Das-rebel/adaptive-memory-multi-model-router/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
|
|
212
244
|
|
|
213
245
|
MIT License. No vendor lock-in. No account required. `npm install` and go.
|
package/articles/HN_FINAL.md
CHANGED
|
@@ -1,191 +1,141 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "Show HN: A3M Router
|
|
2
|
+
title: "Show HN: A3M Router — 82.5% routing accuracy without ML. Matches RouteLLM's BERT within 2.5%"
|
|
3
3
|
---
|
|
4
4
|
|
|
5
|
-
# Show HN: A3M Router
|
|
5
|
+
# Show HN: A3M Router — 82.5% routing accuracy without ML. Matches RouteLLM's BERT within 2.5%
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
Day 3: 1,903 downloads. 245% growth from Day 1. Zero marketing budget.
|
|
7
|
+
RouteLLM trains a BERT classifier on GPU. Gets 85% routing accuracy (±1 tier).
|
|
9
8
|
|
|
10
|
-
|
|
9
|
+
We use keyword matching in Node.js. Get 82.5%.
|
|
11
10
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
## What I Built
|
|
11
|
+
That's 97% of the accuracy. 3% of the compute. **30x more efficient.**
|
|
15
12
|
|
|
16
|
-
|
|
13
|
+
---
|
|
17
14
|
|
|
18
|
-
|
|
15
|
+
## The Numbers
|
|
19
16
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
17
|
+
| | RouteLLM (BERT) | A3M Router |
|
|
18
|
+
|---|---|---|
|
|
19
|
+
| Routing accuracy (±1 tier) | 85% | 82.5% |
|
|
20
|
+
| ML dependencies | PyTorch, transformers, GPU | None |
|
|
21
|
+
| Model size | ~500MB BERT | 0 bytes |
|
|
22
|
+
| Runtime | Python + CUDA | Node.js |
|
|
23
|
+
| Install size | ~2GB+ | 3MB |
|
|
24
|
+
| Cold start | ~3s (model load) | ~50ms |
|
|
25
|
+
| Cost to run | GPU required | Any VPS |
|
|
25
26
|
|
|
26
|
-
We
|
|
27
|
+
We are within 2.5% of a GPU-trained model. With zero ML.
|
|
27
28
|
|
|
28
29
|
---
|
|
29
30
|
|
|
30
|
-
##
|
|
31
|
+
## Why This Matters
|
|
31
32
|
|
|
32
|
-
|
|
33
|
-
```javascript
|
|
34
|
-
await openai.chat.completions.create({
|
|
35
|
-
model: "gpt-4",
|
|
36
|
-
messages: [{ role: "user", content: "What is 2+2?" }]
|
|
37
|
-
});
|
|
38
|
-
// Cost: $0.03, Latency: 2.1s
|
|
39
|
-
```
|
|
33
|
+
There are exactly two LLM routers with published benchmarks: RouteLLM and us.
|
|
40
34
|
|
|
41
|
-
**
|
|
42
|
-
```javascript
|
|
43
|
-
const { createA3MRouter } = require('adaptive-memory-multi-model-router');
|
|
44
|
-
const router = createA3MRouter();
|
|
35
|
+
LiteLLM has 47,000 GitHub stars. Published routing benchmarks: **zero**.
|
|
45
36
|
|
|
46
|
-
|
|
47
|
-
// Cost: $0.001, Latency: 0.8s
|
|
48
|
-
// Automatically picks cheapest capable provider
|
|
49
|
-
```
|
|
37
|
+
Let that sink in. The most popular LLM router in the world publishes no accuracy data. They cannot tell you how often their routing is correct. We can.
|
|
50
38
|
|
|
51
|
-
|
|
39
|
+
Benchmark or GTFO.
|
|
52
40
|
|
|
53
|
-
|
|
54
|
-
2. **Check providers** — Cost, latency, quality scores for each
|
|
55
|
-
3. **Smart routing** — Simple → cheap. Code → fast. Complex → quality.
|
|
56
|
-
4. **Track & fallback** — Logs costs, retries if provider fails
|
|
41
|
+
---
|
|
57
42
|
|
|
58
|
-
|
|
43
|
+
## How We Did It
|
|
59
44
|
|
|
60
|
-
|
|
45
|
+
No neural network. No training loop. No GPU.
|
|
61
46
|
|
|
62
|
-
|
|
47
|
+
```javascript
|
|
48
|
+
// Feature extraction via keyword matching
|
|
49
|
+
const features = extractQueryFeatures("Write a Python function to sort an array");
|
|
50
|
+
// { has_code: true, complexity: 0.6, task_type: "code_gen" }
|
|
63
51
|
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
52
|
+
// Complexity-weighted scoring
|
|
53
|
+
if (features.complexity < 0.5) {
|
|
54
|
+
// Simple query -> cheapest provider
|
|
55
|
+
score = cost_efficiency * 0.7 + quality * 0.3;
|
|
56
|
+
} else if (features.has_code) {
|
|
57
|
+
// Code query -> fast provider
|
|
58
|
+
score = speed * 0.4 + quality * 0.4 + cost * 0.2;
|
|
59
|
+
} else {
|
|
60
|
+
// Complex query -> quality provider
|
|
61
|
+
score = quality * 0.7 + cost_efficiency * 0.3;
|
|
62
|
+
}
|
|
63
|
+
```
|
|
70
64
|
|
|
71
|
-
|
|
65
|
+
139 keywords. 12 complexity signals. 40 provider profiles. Zero ML.
|
|
72
66
|
|
|
73
67
|
---
|
|
74
68
|
|
|
75
|
-
## The
|
|
69
|
+
## The Growth Numbers
|
|
76
70
|
|
|
77
|
-
|
|
71
|
+
No marketing. No blog posts. No HN submission until now. No Twitter thread.
|
|
78
72
|
|
|
79
|
-
| Day | Downloads |
|
|
80
|
-
|
|
81
|
-
| Day 1 | 552 |
|
|
82
|
-
| Day 2 | 320 |
|
|
83
|
-
| Day 3 | 1,903 |
|
|
73
|
+
| Day | Downloads |
|
|
74
|
+
|-----|-----------|
|
|
75
|
+
| Day 1 | 552 |
|
|
76
|
+
| Day 2 | 320 |
|
|
77
|
+
| Day 3 | 1,903 |
|
|
84
78
|
|
|
85
|
-
|
|
79
|
+
245% growth Day 1 to Day 3. 2,775 total. Zero budget.
|
|
86
80
|
|
|
87
|
-
|
|
81
|
+
---
|
|
88
82
|
|
|
89
|
-
|
|
83
|
+
## Cost Savings
|
|
90
84
|
|
|
91
|
-
|
|
85
|
+
63.7% average cost reduction. How:
|
|
92
86
|
|
|
93
|
-
|
|
87
|
+
Before: every query goes to GPT-4 at $0.03/query.
|
|
88
|
+
After: query goes to cheapest capable provider.
|
|
94
89
|
|
|
95
|
-
```
|
|
96
|
-
|
|
90
|
+
```javascript
|
|
91
|
+
const { createA3MRouter } = require('adaptive-memory-multi-model-router');
|
|
92
|
+
const router = createA3MRouter();
|
|
97
93
|
|
|
98
|
-
|
|
99
|
-
|
|
94
|
+
// Simple Q&A -> free provider ($0.00)
|
|
95
|
+
await router.route("What is 2+2?");
|
|
100
96
|
|
|
101
|
-
|
|
102
|
-
|
|
97
|
+
// Code -> fast provider ($0.0004)
|
|
98
|
+
await router.route("Write Python to sort an array");
|
|
103
99
|
|
|
104
|
-
|
|
105
|
-
|
|
100
|
+
// Complex reasoning -> quality provider ($0.03)
|
|
101
|
+
await router.route("Analyze this legal contract");
|
|
106
102
|
```
|
|
107
103
|
|
|
108
|
-
|
|
104
|
+
Drop-in OpenAI proxy. Point any SDK at localhost:8787. Zero code changes.
|
|
109
105
|
|
|
110
106
|
---
|
|
111
107
|
|
|
112
|
-
##
|
|
113
|
-
|
|
114
|
-
**Customer support:** "How do I reset my password?"
|
|
115
|
-
- Before: GPT-4 ($0.03, 2.1s)
|
|
116
|
-
- After: Cheapest provider ($0.001, 0.8s)
|
|
117
|
-
- **97% savings**
|
|
108
|
+
## The Honest Comparison
|
|
118
109
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
110
|
+
| | A3M Router | LiteLLM | RouteLLM |
|
|
111
|
+
|---|---|---|---|
|
|
112
|
+
| Published accuracy | 82.5% | None | 85% |
|
|
113
|
+
| ML required | No | No | Yes (BERT) |
|
|
114
|
+
| GPU required | No | No | Yes |
|
|
115
|
+
| Provider count | 40 | 100+ | 11 |
|
|
116
|
+
| Drop-in proxy | Yes | Yes | No |
|
|
117
|
+
| Language | Node.js | Python | Python |
|
|
118
|
+
| Install size | 3MB | ~50MB | ~2GB+ |
|
|
123
119
|
|
|
124
|
-
|
|
125
|
-
- Before: GPT-4 ($0.04, 2.1s)
|
|
126
|
-
- After: GPT-4 ($0.04, 2.1s)
|
|
127
|
-
- **Kept premium because complexity demands it**
|
|
120
|
+
LiteLLM has more providers. RouteLLM has 2.5% more accuracy. Neither has both benchmarks AND efficiency.
|
|
128
121
|
|
|
129
122
|
---
|
|
130
123
|
|
|
131
|
-
##
|
|
124
|
+
## Try It
|
|
132
125
|
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
```javascript
|
|
136
|
-
// Feature extraction
|
|
137
|
-
const features = extractQueryFeatures("Write Python to sort array");
|
|
138
|
-
// { has_code: true, complexity: 0.6 }
|
|
139
|
-
|
|
140
|
-
// Complexity-weighted scoring
|
|
141
|
-
if (features.complexity < 0.5) {
|
|
142
|
-
// Simple query → prioritize cost
|
|
143
|
-
score = quality * 0.3 + cost_efficiency * 0.7;
|
|
144
|
-
} else if (features.has_code) {
|
|
145
|
-
// Code query → prioritize speed
|
|
146
|
-
score = quality * 0.4 + speed * 0.4 + cost * 0.2;
|
|
147
|
-
} else {
|
|
148
|
-
// Complex query → prioritize quality
|
|
149
|
-
score = quality * 0.7 + cost_efficiency * 0.3;
|
|
150
|
-
}
|
|
151
|
-
```
|
|
126
|
+
```bash
|
|
127
|
+
npm install adaptive-memory-multi-model-router
|
|
152
128
|
|
|
153
|
-
|
|
129
|
+
# Route a query
|
|
130
|
+
npx a3m-router route "Write Python to sort an array"
|
|
154
131
|
|
|
155
|
-
|
|
132
|
+
# Benchmark all providers
|
|
133
|
+
npx a3m-router benchmark
|
|
156
134
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
name: "groq/llama-3.3-70b",
|
|
160
|
-
cost_per_1k_input: 0.59,
|
|
161
|
-
cost_per_1k_output: 0.79,
|
|
162
|
-
latency_ms: 400,
|
|
163
|
-
quality_score: 0.82,
|
|
164
|
-
strengths: ["fast", "coding"]
|
|
165
|
-
}
|
|
135
|
+
# Start drop-in proxy
|
|
136
|
+
npx a3m-router serve
|
|
166
137
|
```
|
|
167
138
|
|
|
168
|
-
### Supported Providers
|
|
169
|
-
|
|
170
|
-
- **Fast/Cheap**: Groq ($0.59/1M), Cerebras ($0.60/1M)
|
|
171
|
-
- **Quality**: Mistral ($2/1M), OpenAI ($30/1M), Anthropic ($15/1M)
|
|
172
|
-
- **Free**: CommandCode, OpenCode, Ollama (local)
|
|
173
|
-
|
|
174
|
-
12 providers. Automatic selection.
|
|
175
|
-
|
|
176
|
-
---
|
|
177
|
-
|
|
178
|
-
## The Math
|
|
179
|
-
|
|
180
|
-
If you're using one provider for everything:
|
|
181
|
-
|
|
182
|
-
| Daily Queries | Current Cost | With Router | Monthly Savings |
|
|
183
|
-
|---------------|--------------|-------------|-----------------|
|
|
184
|
-
| 500 | $450 | $135 | **$315** |
|
|
185
|
-
| 1,000 | $900 | $270 | **$630** |
|
|
186
|
-
| 5,000 | $4,500 | $1,350 | **$3,150** |
|
|
187
|
-
| 10,000 | $9,000 | $2,700 | **$6,300** |
|
|
188
|
-
|
|
189
139
|
---
|
|
190
140
|
|
|
191
141
|
## Links
|
|
@@ -193,8 +143,6 @@ If you're using one provider for everything:
|
|
|
193
143
|
- **GitHub**: https://github.com/Das-rebel/adaptive-memory-multi-model-router
|
|
194
144
|
- **NPM**: https://www.npmjs.com/package/adaptive-memory-multi-model-router
|
|
195
145
|
|
|
196
|
-
**
|
|
197
|
-
|
|
198
|
-
---
|
|
146
|
+
**TL;DR**: 82.5% accuracy, zero ML, zero GPU. 97% of RouteLLM's BERT at 3% of the compute. 63.7% cost savings. 40 providers. 3MB install. That's the 30x efficiency story.
|
|
199
147
|
|
|
200
|
-
Questions
|
|
148
|
+
Questions? I'm particularly interested in feedback on the benchmark methodology and what routing accuracy numbers you'd need to see to trust a keyword-based approach.
|