cachegate 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +127 -112
- package/README.md +31 -13
- package/cache.js +72 -51
- package/embeddings.js +42 -32
- package/metrics.js +609 -556
- package/package.json +15 -1
- package/redisClient.js +55 -45
- package/router.js +254 -218
- package/semanticCache.js +159 -154
- package/server.js +282 -113
- package/.dockerignore +0 -11
- package/.gitattributes +0 -12
- package/.github/ISSUE_TEMPLATE/bug_report.md +0 -33
- package/.github/ISSUE_TEMPLATE/config.yml +0 -5
- package/.github/ISSUE_TEMPLATE/feature_request.md +0 -29
- package/.github/PULL_REQUEST_TEMPLATE.md +0 -25
- package/.github/workflows/test.yml +0 -63
- package/CODE_OF_CONDUCT.md +0 -66
- package/CONTRIBUTING.md +0 -94
- package/Dockerfile +0 -24
- package/OPEN_SOURCE_ROADMAP.md +0 -855
- package/ROADMAP.md +0 -281
- package/SECURITY.md +0 -39
- package/sync-oss-release.sh +0 -160
- package/test/auth-config.test.js +0 -27
- package/test/cache.test.js +0 -33
- package/test/embeddings.test.js +0 -24
- package/test/env-path.test.js +0 -41
- package/test/failover.test.js +0 -99
- package/test/metrics-postgres.test.js +0 -183
- package/test/metrics.test.js +0 -282
- package/test/router.test.js +0 -195
- package/test/semanticCache.test.js +0 -167
- package/test/server.test.js +0 -357
- package/test/streaming.test.js +0 -248
package/ROADMAP.md
DELETED
|
@@ -1,281 +0,0 @@
|
|
|
1
|
-
# MemoCode Model Router — Roadmap & Review
|
|
2
|
-
|
|
3
|
-
> Review of `210_apps/001_model_router/` as it stands, and the plan to take it
|
|
4
|
-
> from an internal MemoCode utility to a standalone self-hosted product.
|
|
5
|
-
> Originally written by DeepSeek, 2026-08-23, auditing the version that
|
|
6
|
-
> existed on `deepseek/memocode-chat-and-router` at the time. **Corrected the
|
|
7
|
-
> same day**: that audit's "gaps" section was accurate for the version it was
|
|
8
|
-
> looking at, but a second, much more complete implementation had been built
|
|
9
|
-
> in parallel on `claude/memocode-project-review-v0ohnx` and never merged
|
|
10
|
-
> anywhere shared — neither branch's author knew the other's existed. This
|
|
11
|
-
> file now reflects the merged, actual state. See section 0.
|
|
12
|
-
|
|
13
|
-
## 0. What happened, so it doesn't happen again
|
|
14
|
-
|
|
15
|
-
Two independent model-router implementations existed at once: this one
|
|
16
|
-
(minimal - proxy, exact-match cache, provider mapping) on
|
|
17
|
-
`deepseek/memocode-chat-and-router`, and a materially more complete one
|
|
18
|
-
(routing strategies, semantic cache, streaming, JSONL metrics with rotation,
|
|
19
|
-
a cost dashboard, hardened auth, 68 tests) built the same day on
|
|
20
|
-
`claude/memocode-project-review-v0ohnx` - a branch neither this file's
|
|
21
|
-
original audit nor DeepSeek's own semantic-cache attempt knew to check.
|
|
22
|
-
Discovered when the CEO asked to test the semantic cache and DeepSeek,
|
|
23
|
-
checking `master`/`deepseek/memocode-chat-and-router`/
|
|
24
|
-
`claude/session-import-feature`, correctly found it on none of them - but
|
|
25
|
-
hadn't checked the actual branch it lived on. Resolved by replacing this
|
|
26
|
-
directory's contents with the more complete version (a strict superset
|
|
27
|
-
except for this file) and keeping this file's own market/monetization
|
|
28
|
-
analysis, which stands independent of which codebase it's describing.
|
|
29
|
-
**The lesson, not just the fix**: a feature branch that never gets
|
|
30
|
-
merged toward `master` (or at least announced on the coordination board
|
|
31
|
-
with its branch name) is invisible to everyone not already looking at it,
|
|
32
|
-
no matter how complete the work on it is - completeness doesn't substitute
|
|
33
|
-
for discoverability.
|
|
34
|
-
|
|
35
|
-
## 1. What's actually here (audited 2026-08-23, post-merge)
|
|
36
|
-
|
|
37
|
-
**Working, tested, pushed** (`210_apps/001_model_router/`, this directory,
|
|
38
|
-
now matching `claude/memocode-project-review-v0ohnx`):
|
|
39
|
-
- `server.js` — Express, OpenAI-compatible `POST /v1/chat/completions`
|
|
40
|
-
(streaming and non-streaming), internal bearer-key auth (fails CLOSED by
|
|
41
|
-
default - `ALLOW_INSECURE_LOCAL_DEV=true` required to opt into running
|
|
42
|
-
without a key), rate limiting, `/health`, `/stats`, `/dashboard`.
|
|
43
|
-
- `providers/anthropic.js` + `providers/openai.js` — request mapping,
|
|
44
|
-
token/cost estimation, `chatStream()` for SSE.
|
|
45
|
-
- `cache.js` — Redis exact-match cache keyed by a content hash of
|
|
46
|
-
`{model, messages, temperature, max_tokens, tools, tool_choice}`.
|
|
47
|
-
- `semanticCache.js` + `embeddings.js` — near-duplicate matching via
|
|
48
|
-
embeddings + cosine similarity over a bounded per-model Redis list
|
|
49
|
-
(brute-force, not RediSearch/vector-indexed - an honest, documented
|
|
50
|
-
scale limit, not a hidden one). Tool-calling requests excluded. Semantic
|
|
51
|
-
hits tracked separately from exact hits in metrics, never blended into
|
|
52
|
-
one number.
|
|
53
|
-
- `router.js` — three explicit `router:` virtual-model strategies (cost /
|
|
54
|
-
latency / latency-guarded-cost), no invented blended score.
|
|
55
|
-
- `metrics.js` — JSONL logs with day-rotation, bounded reads, opt-in
|
|
56
|
-
pruning.
|
|
57
|
-
- `public/dashboard.html` — a real cost dashboard (KPI tiles, charts,
|
|
58
|
-
auto-refresh, table-view twins), built to this project's own `dataviz`
|
|
59
|
-
skill standard.
|
|
60
|
-
- 68 tests across 8 test files, all passing.
|
|
61
|
-
- Live: dogfooded by `210_apps/000_backend/chat-import-logic.mjs`
|
|
62
|
-
(`localhost:4000/v1/chat/completions` for chat segmentation).
|
|
63
|
-
|
|
64
|
-
**Real remaining gaps** (honest, not "nothing left to do"; updated
|
|
65
|
-
2026-08-29 after re-auditing against the actual code - two of the four
|
|
66
|
-
gaps below turned out to already be closed):
|
|
67
|
-
1. Semantic cache is brute-force cosine over a capped list, not a real
|
|
68
|
-
vector index - fine at self-hosted single-instance volume, not meant to
|
|
69
|
-
scale past `SEMANTIC_CACHE_MAX_CANDIDATES` (default 200) per model.
|
|
70
|
-
2. ~~No provider failover on a 5xx/rate-limit~~ **Closed 2026-08-29.**
|
|
71
|
-
`failover.js` + `server.js`'s non-streaming dispatch path now walk
|
|
72
|
-
`router.js`'s full ranked-candidate list, retrying the next candidate
|
|
73
|
-
when one fails for a reason that isn't the request's own fault (a
|
|
74
|
-
400/404 still fails immediately - retrying elsewhere wouldn't help).
|
|
75
|
-
Streaming is a deliberate exception, documented inline in
|
|
76
|
-
`handleStreamingDispatch` - SSE headers and the first frame commit to
|
|
77
|
-
a model name before a failure could be known, so silent mid-stream
|
|
78
|
-
provider switching is a materially harder problem, left open rather
|
|
79
|
-
than shipped half-working.
|
|
80
|
-
3. ~~No persistence layer under the dashboard~~ **Closed** (already true
|
|
81
|
-
before this pass, just never updated here): `metrics.js` has stored
|
|
82
|
-
metrics in Postgres since the `feat(model-router): persistent metrics
|
|
83
|
-
storage via Postgres` commit, with automatic JSONL fallback when no
|
|
84
|
-
database is configured.
|
|
85
|
-
4. Not yet open-sourced/packaged standalone (Phase 5, section 4 below).
|
|
86
|
-
|
|
87
|
-
**What this actually is today, said plainly**: working, tested internal
|
|
88
|
-
infrastructure with zero users outside this codebase — not an app, not a
|
|
89
|
-
published package, not a running public service, not something anyone has
|
|
90
|
-
paid for or even tried. Everything from here through section 3 (market
|
|
91
|
-
position, monetization tiers, MRR estimates) describes a *hypothetical*
|
|
92
|
-
product this code could become, not a claim about what it is right now.
|
|
93
|
-
Read it as a plan, not a status report - a CEO challenge on 2026-08-23
|
|
94
|
-
("this is not an app, it's not portable, we build nothing") is a fair
|
|
95
|
-
description of today's reality and is what section 4's Phase 5 (open
|
|
96
|
-
source release) exists to close.
|
|
97
|
-
|
|
98
|
-
## 2. Honest market position
|
|
99
|
-
|
|
100
|
-
Do **not** try to out-LiteLLM LiteLLM (140+ providers, Python, huge community)
|
|
101
|
-
or OpenRouter (acquired by Stripe). The winnable niche:
|
|
102
|
-
|
|
103
|
-
- **Self-hosted first** — prompts/PII never leave your infra.
|
|
104
|
-
- **Semantic cache**, not just exact-match (most OSS options still hash-match).
|
|
105
|
-
- **Node.js/TypeScript** — the JS/TS AI-app crowd is underserved.
|
|
106
|
-
- **Embeddable** — usable as a module inside an existing app before it's sold.
|
|
107
|
-
|
|
108
|
-
**Correction, same day, on the semantic-cache claim specifically**: verified
|
|
109
|
-
via WebSearch that LiteLLM shipped a real vector-indexed (Valkey-search +
|
|
110
|
-
HNSW) semantic cache in 2026 - more sophisticated than this router's
|
|
111
|
-
brute-force cosine scan. "Semantic cache, not just exact-match" is no longer
|
|
112
|
-
a differentiator against LiteLLM by name, even though it's still true against
|
|
113
|
-
"most OSS options." The honest remaining edges: small/auditable codebase,
|
|
114
|
-
Node-native, and already dogfooded inside a real app.
|
|
115
|
-
|
|
116
|
-
The sellable, honest claim (do not quote inflated 86–95%): **"Caching alone
|
|
117
|
-
typically saves 20–45%; add routing and it can reach 47–90% on well-tuned
|
|
118
|
-
traffic."** Being the vendor that quotes the real number is the credibility
|
|
119
|
-
edge with the buyer who has been burned by an inflated claim.
|
|
120
|
-
|
|
121
|
-
## 3. How this router makes money (open-core, honest)
|
|
122
|
-
|
|
123
|
-
Self-hosted OSS infrastructure monetizes by **open-core**, not by charging for
|
|
124
|
-
the free thing:
|
|
125
|
-
|
|
126
|
-
| Tier | What | Price | Why someone pays |
|
|
127
|
-
|---|---|---|---|
|
|
128
|
-
| **OSS core (free)** | exact-match cache, 2–3 providers, basic proxy | $0 | top-of-funnel; builds trust |
|
|
129
|
-
| **Pro** | semantic cache, cost-based routing + failover, cost dashboard | $49–99/mo | the "save 40–80% on LLM spend" features; charge ~10–20% of the savings |
|
|
130
|
-
| **Team** | multi-user, SSO, audit log, priority support | $199–499/mo | small teams running production workloads |
|
|
131
|
-
| **Hosted** | we run it for you (no ops) | usage-based, $99–999/mo | teams that don't want to self-host (where LiteLLM makes its money) |
|
|
132
|
-
| **Enterprise** | on-prem, compliance, SLAs | custom | regulated buyers |
|
|
133
|
-
|
|
134
|
-
**Honest caveats:** OSS → paid conversion is typically 1–5%; the money is in
|
|
135
|
-
the small fraction who want the semantic cache + dashboard and don't want to
|
|
136
|
-
self-host. It is a crowded market. Realistic: **$5–20K MRR in 12 months** by
|
|
137
|
-
owning the self-hosted Node.js niche. The metric that makes it work: a buyer
|
|
138
|
-
spending $1,000/mo on LLM APIs saves $400–800/mo with this — paying $50–100/mo
|
|
139
|
-
for the tool is an easy yes.
|
|
140
|
-
|
|
141
|
-
## 4. Phases
|
|
142
|
-
|
|
143
|
-
**Note on an older numbering (added 2026-08-29):** early journal entries
|
|
144
|
-
(2026-08-24) refer to this work as a flat "20-step roadmap" (steps
|
|
145
|
-
13-17 individually named there; 18 marked done but never described; 19
|
|
146
|
-
= the semantic-cache vector-index upgrade, paused; 20 = the standalone
|
|
147
|
-
product). That numbering was never written down as one document - it
|
|
148
|
-
only ever existed as scattered journal references, which made it
|
|
149
|
-
genuinely hard to reconstruct later (confirmed 2026-08-29: step 18's
|
|
150
|
-
actual content couldn't be found anywhere). **The 6 phases below
|
|
151
|
-
supersede that numbering entirely.** If an old "step N" reference ever
|
|
152
|
-
surfaces again, map it here rather than trying to revive the flat list.
|
|
153
|
-
|
|
154
|
-
### Phase 1 — make what exists real — **done**
|
|
155
|
-
Redis verified end-to-end, honest README, tests (68, not the "no test files"
|
|
156
|
-
this section originally reported), the honest cost claim above.
|
|
157
|
-
|
|
158
|
-
### Phase 2 — semantic cache — **done**
|
|
159
|
-
Embeddings + cosine similarity, `SEMANTIC_CACHE_THRESHOLD` default 0.93 (this
|
|
160
|
-
section's own original draft proposed ~0.95 - close, tuned during real
|
|
161
|
-
testing; see `semanticCache.js`'s own comment for the reasoning), tool-calls
|
|
162
|
-
excluded, tracked separately from exact-match hits.
|
|
163
|
-
|
|
164
|
-
### Phase 3 — real routing — **done**
|
|
165
|
-
Three explicit strategies (`router.js`) rather than a single invented
|
|
166
|
-
blended score - cost-ascending, latency-then-cost, and latency-guarded-cost
|
|
167
|
-
(excludes anything too much slower than the fastest known candidate).
|
|
168
|
-
Failover on error for non-streaming requests shipped 2026-08-29
|
|
169
|
-
(`failover.js`) - see section 1, gap 2, for what it does and doesn't cover.
|
|
170
|
-
|
|
171
|
-
### Phase 4 — cost visibility — **done**
|
|
172
|
-
`metrics.js` (JSONL, day-rotation, bounded reads) + `public/dashboard.html`
|
|
173
|
-
(KPI tiles, charts, auto-refresh, table-view twins).
|
|
174
|
-
|
|
175
|
-
### Phase 5 — open-source release — not started
|
|
176
|
-
Public GitHub repo → npm package → Docker image → announce (Hacker News,
|
|
177
|
-
r/LocalLLaMA, r/selfhosted, Dev.to). Still the real next milestone - none of
|
|
178
|
-
phases 1-4 being done changes that this hasn't shipped to anyone outside
|
|
179
|
-
this project yet. **Execution plan, added 2026-08-29:**
|
|
180
|
-
[`OPEN_SOURCE_ROADMAP.md`](./OPEN_SOURCE_ROADMAP.md) - the 20-step
|
|
181
|
-
breakdown of everything this one-liner actually requires, written down
|
|
182
|
-
as one document on purpose (see that file's own note on why).
|
|
183
|
-
|
|
184
|
-
### Phase 6 — hosted tier (only after OSS traction)
|
|
185
|
-
Render/Fly hosted; usage-based pricing.
|
|
186
|
-
|
|
187
|
-
## 5. One honest caveat on dogfooding
|
|
188
|
-
|
|
189
|
-
Chat-import's one segmentation call per session is too low-repeat to prove
|
|
190
|
-
the *cache* saves money — it only proves the *proxy* works. **2026-08-23
|
|
191
|
-
update**: `210_apps/000_backend/ai-providers.mjs`'s "Generate with AI" now
|
|
192
|
-
routes through this router too (app's own shared key only, never a
|
|
193
|
-
signed-in user's BYOK key — the router is one shared-secret proxy today,
|
|
194
|
-
not multi-tenant).
|
|
195
|
-
|
|
196
|
-
**Correction, same day, on how that update was first worded here**: this
|
|
197
|
-
section originally called that traffic "a genuinely more promising source
|
|
198
|
-
of real cache hits" — that was a hunch stated as a finding, and a CEO
|
|
199
|
-
challenge caught it (fair: "what are the odds two people ask the same
|
|
200
|
-
thing" is the right question, and for this app's actual traffic the honest
|
|
201
|
-
answer is "mostly low, and nobody has measured it"). The one real
|
|
202
|
-
structural fact in favor of *some* hits: `ai-providers.mjs` builds that
|
|
203
|
-
prompt as `"Subject: <topic>. Generate about N <label> worth of
|
|
204
|
-
content."` — short and templated, with `topic` the only variable — so two
|
|
205
|
-
different users both studying, say, "the Roman Empire" would produce an
|
|
206
|
-
identical or near-identical request, which chat-import's full conversation
|
|
207
|
-
transcripts basically never do. That's a plausible *mechanism* for a
|
|
208
|
-
non-zero hit rate on a study app whose users cluster around common
|
|
209
|
-
curriculum topics — it is not a measured hit rate, and treating it as one
|
|
210
|
-
would repeat the same mistake. Nobody has looked at MemoCode's actual
|
|
211
|
-
topic distribution. `MODEL_ROUTER_URL` is also still unset in the current
|
|
212
|
-
Render deploy (`render.yaml` declares no router service), so none of this
|
|
213
|
-
runs in production yet regardless.
|
|
214
|
-
|
|
215
|
-
To validate savings for real, you need either that traffic running through
|
|
216
|
-
the router in production with real measurement, or synthetic load testing
|
|
217
|
-
against a realistic topic distribution — not a plausible-sounding argument
|
|
218
|
-
for why it might work, however structurally reasonable that argument is.
|
|
219
|
-
|
|
220
|
-
## 6. Two-version architecture (decision 2026-08-23)
|
|
221
|
-
|
|
222
|
-
One engine (routing + cache + metrics), two wrappers — NOT two codebases:
|
|
223
|
-
|
|
224
|
-
| | Embedded | Standalone |
|
|
225
|
-
|---|---|---|
|
|
226
|
-
| Lives | inside MemoCode (own process) | its own hosted service (Render) |
|
|
227
|
-
| Keys | you bring Anthropic/OpenAI | it holds keys, issues its own key + URL |
|
|
228
|
-
| Login/billing | none | yes (login, payment, multi-tenancy) |
|
|
229
|
-
| Serves | your apps (fallback) | everybody, including your apps |
|
|
230
|
-
| Status | done — needs Redis + production wiring | future project (OpenRouter/LiteLLM competitor) |
|
|
231
|
-
|
|
232
|
-
Fallback: apps point at the standalone and drop back to the embedded
|
|
233
|
-
(localhost or MemoCode's own instance) when the standalone is down. Same job,
|
|
234
|
-
so the switch is invisible to the caller. Do NOT fork the router — wrap it:
|
|
235
|
-
same core logic, two thin deployment shells, so the two can never drift apart
|
|
236
|
-
the way the two parallel implementations did (section 0).
|
|
237
|
-
|
|
238
|
-
The embedded version **stays embedded in each new app** (each app inherits
|
|
239
|
-
this directory as-is) — it is NOT extracted to a shared repo. A shared repo
|
|
240
|
-
means maintaining one router copy per repo plus separate hosting/db, which
|
|
241
|
-
isn't worth it; a new app just copies `210_apps/001_model_router/` into its
|
|
242
|
-
own backend and points `MODEL_ROUTER_URL` at it. The standalone's key+URL are
|
|
243
|
-
left as comments in the app's `.env` as a future reminder, not wired yet.
|
|
244
|
-
|
|
245
|
-
Sequencing: finish the embedded first (cheap, real, no risk), then decide
|
|
246
|
-
whether to build the standalone's billing layer (the hard, deferred part).
|
|
247
|
-
|
|
248
|
-
## 7. Embedded — production checklist
|
|
249
|
-
|
|
250
|
-
1. Hosted Redis (Render Key Value) → set `REDIS_URL`.
|
|
251
|
-
2. Deploy this directory as its own service (or co-locate with the backend).
|
|
252
|
-
The repo `render.yaml` now declares a `memocode-router` service (Node
|
|
253
|
-
runtime, `npm ci` + `npm start`) and a `Dockerfile` is included for
|
|
254
|
-
Docker-based deploys.
|
|
255
|
-
3. Set production env: `MODEL_ROUTER_INTERNAL_KEY` (strong, `openssl rand -hex 32`),
|
|
256
|
-
`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `REDIS_URL`,
|
|
257
|
-
`SEMANTIC_CACHE_ENABLED=false` (low-repeat traffic: semantic cache spends
|
|
258
|
-
an embedding call on every miss for nothing), `ROUTER_STRATEGY=cost`.
|
|
259
|
-
4. Route MemoCode's AI call sites through it via `MODEL_ROUTER_URL`.
|
|
260
|
-
**Corrected 2026-08-29**: this item previously named "image gen" and
|
|
261
|
-
"transcription" as call sites to route through the router - checked
|
|
262
|
-
directly, and that's not actually possible as scoped. The router is an
|
|
263
|
-
OpenAI-*chat-completions*-compatible proxy only; image generation
|
|
264
|
-
(`images.generate`) and Whisper transcription
|
|
265
|
-
(`audio.transcriptions.create`) are different API shapes it doesn't
|
|
266
|
-
speak. What's actually wired today: chat-import's segmentation call
|
|
267
|
-
(`chat-import-logic.mjs`) and "Generate with AI" text generation
|
|
268
|
-
(`ai-providers.mjs`'s `generateStructuredText`), both gated behind
|
|
269
|
-
`MODEL_ROUTER_URL` being set (`shouldRouteToModelRouter()`). Extending
|
|
270
|
-
the router to proxy image/audio calls too would be new scope, not a
|
|
271
|
-
pending item on this checklist.
|
|
272
|
-
5. Verify `/health`, `/dashboard`, `/stats` on production with real traffic.
|
|
273
|
-
|
|
274
|
-
**Note on items 1 and 3 above (2026-08-29):** `render.yaml` declares the
|
|
275
|
-
env var slots for `REDIS_URL`, `MODEL_ROUTER_INTERNAL_KEY`,
|
|
276
|
-
`ANTHROPIC_API_KEY`, and `OPENAI_API_KEY` (all `sync: false` - filled in
|
|
277
|
-
manually on Render's dashboard), and the metrics Postgres connection is
|
|
278
|
-
wired automatically via `fromDatabase`. Whether those manual slots are
|
|
279
|
-
actually populated with real values on the live Render service isn't
|
|
280
|
-
something a sandboxed session can check - that verification needs
|
|
281
|
-
whoever has the Render dashboard.
|
package/SECURITY.md
DELETED
|
@@ -1,39 +0,0 @@
|
|
|
1
|
-
# Security Policy
|
|
2
|
-
|
|
3
|
-
## Reporting a vulnerability
|
|
4
|
-
|
|
5
|
-
**Please do not open a public GitHub issue for a security
|
|
6
|
-
vulnerability.** A public issue is a disclosure before a fix exists.
|
|
7
|
-
|
|
8
|
-
Instead, use GitHub's private vulnerability reporting:
|
|
9
|
-
|
|
10
|
-
1. Go to this repository's **Security** tab.
|
|
11
|
-
2. Click **Report a vulnerability**.
|
|
12
|
-
3. Describe the issue — what it is, how to reproduce it, and its
|
|
13
|
-
likely impact (e.g. "bypasses auth," "leaks another deployment's
|
|
14
|
-
cached data," "exhausts memory regardless of rate limiting").
|
|
15
|
-
|
|
16
|
-
This opens a private conversation with the maintainers, visible only to
|
|
17
|
-
you and them, and lets a fix be prepared and released before the
|
|
18
|
-
vulnerability is public.
|
|
19
|
-
|
|
20
|
-
*(Maintainer note, remove once live: this requires "Private vulnerability
|
|
21
|
-
reporting" to be turned on for the repository — Settings → Security →
|
|
22
|
-
Private vulnerability reporting — as part of step 15's repo setup.)*
|
|
23
|
-
|
|
24
|
-
## What counts as a security issue here
|
|
25
|
-
|
|
26
|
-
Concretely, for this project: anything that lets a request bypass
|
|
27
|
-
`MODEL_ROUTER_INTERNAL_KEY` auth, read or corrupt another deployment's
|
|
28
|
-
cached data or metrics, or exhaust CPU/memory/Redis storage in a way
|
|
29
|
-
`RATE_LIMIT_MAX`/`RATE_LIMIT_WINDOW_MS` doesn't already bound. A
|
|
30
|
-
provider returning an unexpected error, a routing decision you disagree
|
|
31
|
-
with, or a missing feature are regular bugs — open a normal issue for
|
|
32
|
-
those (see `CONTRIBUTING.md`).
|
|
33
|
-
|
|
34
|
-
## Supported versions
|
|
35
|
-
|
|
36
|
-
This project is pre-1.0 (see `OPEN_SOURCE_ROADMAP.md` on the version
|
|
37
|
-
plan) — security fixes go into the latest release only. Once a stable
|
|
38
|
-
1.0 line exists, this section will name which major versions still
|
|
39
|
-
receive fixes.
|
package/sync-oss-release.sh
DELETED
|
@@ -1,160 +0,0 @@
|
|
|
1
|
-
#!/usr/bin/env bash
|
|
2
|
-
#
|
|
3
|
-
# sync-oss-release.sh - mirrors THIS directory's git-tracked files into
|
|
4
|
-
# a checkout of the public cachegate repo, as ONE NEW COMMIT there.
|
|
5
|
-
#
|
|
6
|
-
# See OPEN_SOURCE_ROADMAP.md step 11 for why this direction (this
|
|
7
|
-
# monorepo directory is the source of truth, not the public repo) and
|
|
8
|
-
# step 1 for why the ONE-TIME initial extraction (step 16) uses fresh,
|
|
9
|
-
# curated history. This script is different from that: it's what runs
|
|
10
|
-
# on every sync AFTER the initial extraction, and it does NOT rewrite
|
|
11
|
-
# history - it adds a single normal commit on top of whatever the
|
|
12
|
-
# target repo already has, exactly like any other change to that repo.
|
|
13
|
-
# Rewriting history on every sync would break clones, forks, and
|
|
14
|
-
# in-flight PRs on the public side; that's not what this does.
|
|
15
|
-
#
|
|
16
|
-
# This is a MIRROR, not a merge: after syncing, the target's tracked
|
|
17
|
-
# files exactly match this directory's. A file that exists only in the
|
|
18
|
-
# target (added directly on GitHub, not here) gets REMOVED on sync.
|
|
19
|
-
# That's deliberate - if a file should persist in the public repo, add
|
|
20
|
-
# it here, in the monorepo, since this directory is the source of truth
|
|
21
|
-
# (see step 11). This script will refuse to run against a target that
|
|
22
|
-
# isn't a git repository, specifically so an accidental wipe of some
|
|
23
|
-
# unrelated directory can't happen by pointing this at the wrong path.
|
|
24
|
-
#
|
|
25
|
-
# Usage:
|
|
26
|
-
# ./sync-oss-release.sh <path-to-public-repo-checkout> [--version X.Y.Z]
|
|
27
|
-
#
|
|
28
|
-
# What it does, in order:
|
|
29
|
-
# 1. Refuses to run if the secrets scan (same patterns as the manual
|
|
30
|
-
# step-3 audit: API key shapes, email addresses) finds anything in
|
|
31
|
-
# this directory's tracked files - loud failure, nothing touched,
|
|
32
|
-
# rather than a quiet publish of a leak.
|
|
33
|
-
# 2. If --version is given, bumps THIS directory's own package.json
|
|
34
|
-
# to that version first, so the synced copy carries it too. Omits
|
|
35
|
-
# this by default - the script doesn't invent a version-bump
|
|
36
|
-
# policy on its own (see step 6's deferred semver plan); a
|
|
37
|
-
# no-flag run is a plain resync at whatever version is already
|
|
38
|
-
# set (e.g. reapplying a cherry-picked external PR - see
|
|
39
|
-
# CONTRIBUTING.md's note on that flow).
|
|
40
|
-
# 3. Mirrors every git-tracked file from this directory into the
|
|
41
|
-
# target checkout (removes everything else from the target's
|
|
42
|
-
# working tree first, except its own .git/) - a file removed here
|
|
43
|
-
# also disappears there, never a manual, error-prone diff to keep
|
|
44
|
-
# in sync by hand.
|
|
45
|
-
# 4. Commits in the TARGET repo (one new commit, normal history).
|
|
46
|
-
# Does NOT push - pushing is a deliberate, separate, human/CI step.
|
|
47
|
-
|
|
48
|
-
set -euo pipefail
|
|
49
|
-
|
|
50
|
-
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
51
|
-
cd "$SCRIPT_DIR"
|
|
52
|
-
|
|
53
|
-
TARGET=""
|
|
54
|
-
NEW_VERSION=""
|
|
55
|
-
|
|
56
|
-
while [[ $# -gt 0 ]]; do
|
|
57
|
-
case "$1" in
|
|
58
|
-
--version)
|
|
59
|
-
NEW_VERSION="$2"
|
|
60
|
-
shift 2
|
|
61
|
-
;;
|
|
62
|
-
*)
|
|
63
|
-
if [[ -z "$TARGET" ]]; then
|
|
64
|
-
TARGET="$1"
|
|
65
|
-
shift
|
|
66
|
-
else
|
|
67
|
-
echo "Unexpected extra argument: $1" >&2
|
|
68
|
-
exit 1
|
|
69
|
-
fi
|
|
70
|
-
;;
|
|
71
|
-
esac
|
|
72
|
-
done
|
|
73
|
-
|
|
74
|
-
if [[ -z "$TARGET" ]]; then
|
|
75
|
-
echo "Usage: $0 <path-to-public-repo-checkout> [--version X.Y.Z]" >&2
|
|
76
|
-
exit 1
|
|
77
|
-
fi
|
|
78
|
-
|
|
79
|
-
if [[ ! -d "$TARGET/.git" ]]; then
|
|
80
|
-
echo "❌ Refusing to run: $TARGET is not a git repository (no .git/ found)." >&2
|
|
81
|
-
echo " This is deliberate - pointing this at the wrong path would wipe it." >&2
|
|
82
|
-
exit 1
|
|
83
|
-
fi
|
|
84
|
-
|
|
85
|
-
TARGET="$(cd "$TARGET" && pwd)"
|
|
86
|
-
|
|
87
|
-
echo "🔍 Step 1/4: scanning tracked files for secrets before touching anything..."
|
|
88
|
-
# Same shape of check as step 3's manual audit: API key patterns and
|
|
89
|
-
# email addresses, restricted to git-tracked files only (never
|
|
90
|
-
# node_modules, .env, data/ - those aren't tracked, so git ls-files
|
|
91
|
-
# already excludes them).
|
|
92
|
-
SECRET_HIT=0
|
|
93
|
-
while IFS= read -r -d '' file; do
|
|
94
|
-
if grep -qE "sk-[a-zA-Z0-9_-]{20,}|AIza[0-9A-Za-z_-]{20,}|xai-[a-zA-Z0-9_-]{20,}" "$file" 2>/dev/null; then
|
|
95
|
-
echo " ❌ Possible API key in $file" >&2
|
|
96
|
-
SECRET_HIT=1
|
|
97
|
-
fi
|
|
98
|
-
if grep -qE "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-z]{2,}" "$file" 2>/dev/null; then
|
|
99
|
-
# .env.example intentionally has no real emails; this still flags
|
|
100
|
-
# anything matching the shape so a human confirms it's a placeholder.
|
|
101
|
-
match=$(grep -oE "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-z]{2,}" "$file" | grep -v -E "example\.com|your-|@anthropic\.com" || true)
|
|
102
|
-
if [[ -n "$match" ]]; then
|
|
103
|
-
echo " ⚠️ Email-shaped string in $file: $match (confirm this is a placeholder, not real)" >&2
|
|
104
|
-
SECRET_HIT=1
|
|
105
|
-
fi
|
|
106
|
-
fi
|
|
107
|
-
done < <(git ls-files -z)
|
|
108
|
-
|
|
109
|
-
if [[ "$SECRET_HIT" -eq 1 ]]; then
|
|
110
|
-
echo "❌ Aborting sync - resolve the findings above first. Nothing was copied." >&2
|
|
111
|
-
exit 1
|
|
112
|
-
fi
|
|
113
|
-
echo " ✅ Clean."
|
|
114
|
-
|
|
115
|
-
if [[ -n "$NEW_VERSION" ]]; then
|
|
116
|
-
echo "🔢 Step 2/4: bumping package.json to $NEW_VERSION..."
|
|
117
|
-
node -e "
|
|
118
|
-
const fs = require('fs');
|
|
119
|
-
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf8'));
|
|
120
|
-
pkg.version = process.argv[1];
|
|
121
|
-
fs.writeFileSync('package.json', JSON.stringify(pkg, null, 2) + '\n');
|
|
122
|
-
" "$NEW_VERSION"
|
|
123
|
-
echo " ✅ package.json now at $NEW_VERSION."
|
|
124
|
-
else
|
|
125
|
-
echo "🔢 Step 2/4: no --version given, leaving package.json's version as-is."
|
|
126
|
-
fi
|
|
127
|
-
|
|
128
|
-
echo "📦 Step 3/4: mirroring tracked files into $TARGET..."
|
|
129
|
-
# Wipe the target's working tree except .git/, then copy this
|
|
130
|
-
# directory's tracked files in - guarantees the target ends up an
|
|
131
|
-
# EXACT mirror, not an accumulation of whatever used to be there.
|
|
132
|
-
find "$TARGET" -mindepth 1 -maxdepth 1 -not -name ".git" -exec rm -rf {} +
|
|
133
|
-
|
|
134
|
-
while IFS= read -r -d '' file; do
|
|
135
|
-
dest="$TARGET/$file"
|
|
136
|
-
mkdir -p "$(dirname "$dest")"
|
|
137
|
-
cp "$file" "$dest"
|
|
138
|
-
done < <(git ls-files -z)
|
|
139
|
-
|
|
140
|
-
echo " ✅ Copied $(git ls-files | wc -l | tr -d ' ') tracked files."
|
|
141
|
-
|
|
142
|
-
echo "💾 Step 4/4: committing in the target repo (not pushing)..."
|
|
143
|
-
SOURCE_SHA="$(git rev-parse --short HEAD)"
|
|
144
|
-
(
|
|
145
|
-
cd "$TARGET"
|
|
146
|
-
git add -A
|
|
147
|
-
if git diff --cached --quiet; then
|
|
148
|
-
echo " ℹ️ Nothing changed - target already matches this directory. No commit made."
|
|
149
|
-
else
|
|
150
|
-
git commit -m "Sync from internal monorepo @ ${SOURCE_SHA}
|
|
151
|
-
|
|
152
|
-
Mirrors 210_apps/001_model_router/ as of that commit. This commit was
|
|
153
|
-
generated by sync-oss-release.sh, not written by hand - see
|
|
154
|
-
OPEN_SOURCE_ROADMAP.md step 11 in the source repo for why this
|
|
155
|
-
direction (monorepo -> public repo, not the reverse)."
|
|
156
|
-
echo " ✅ Committed. Review with 'git show' in $TARGET, then push when ready - this script never pushes."
|
|
157
|
-
fi
|
|
158
|
-
)
|
|
159
|
-
|
|
160
|
-
echo "✅ Sync complete."
|
package/test/auth-config.test.js
DELETED
|
@@ -1,27 +0,0 @@
|
|
|
1
|
-
const { test } = require('node:test');
|
|
2
|
-
const assert = require('node:assert/strict');
|
|
3
|
-
|
|
4
|
-
// server.js reads its env vars once at require time, so each case here
|
|
5
|
-
// resets env + require cache before requiring fresh.
|
|
6
|
-
function freshServer(env) {
|
|
7
|
-
delete process.env.MODEL_ROUTER_INTERNAL_KEY;
|
|
8
|
-
delete process.env.ALLOW_INSECURE_LOCAL_DEV;
|
|
9
|
-
Object.assign(process.env, env);
|
|
10
|
-
delete require.cache[require.resolve('../server')];
|
|
11
|
-
return require('../server');
|
|
12
|
-
}
|
|
13
|
-
|
|
14
|
-
test('isAuthConfigured() is false with neither key nor opt-in set', () => {
|
|
15
|
-
const { isAuthConfigured } = freshServer({});
|
|
16
|
-
assert.equal(isAuthConfigured(), false);
|
|
17
|
-
});
|
|
18
|
-
|
|
19
|
-
test('isAuthConfigured() is true once MODEL_ROUTER_INTERNAL_KEY is set', () => {
|
|
20
|
-
const { isAuthConfigured } = freshServer({ MODEL_ROUTER_INTERNAL_KEY: 'some-key' });
|
|
21
|
-
assert.equal(isAuthConfigured(), true);
|
|
22
|
-
});
|
|
23
|
-
|
|
24
|
-
test('isAuthConfigured() is true with the explicit insecure opt-in, even with no key', () => {
|
|
25
|
-
const { isAuthConfigured } = freshServer({ ALLOW_INSECURE_LOCAL_DEV: 'true' });
|
|
26
|
-
assert.equal(isAuthConfigured(), true);
|
|
27
|
-
});
|
package/test/cache.test.js
DELETED
|
@@ -1,33 +0,0 @@
|
|
|
1
|
-
const { test } = require('node:test');
|
|
2
|
-
const assert = require('node:assert/strict');
|
|
3
|
-
const cache = require('../cache');
|
|
4
|
-
|
|
5
|
-
test('buildCacheKey is deterministic for identical payloads', () => {
|
|
6
|
-
const payload = { model: 'gpt-4o-mini', messages: [{ role: 'user', content: 'hi' }] };
|
|
7
|
-
assert.equal(cache.buildCacheKey(payload), cache.buildCacheKey({ ...payload }));
|
|
8
|
-
});
|
|
9
|
-
|
|
10
|
-
test('buildCacheKey differs when messages differ', () => {
|
|
11
|
-
const a = { model: 'gpt-4o-mini', messages: [{ role: 'user', content: 'hi' }] };
|
|
12
|
-
const b = { model: 'gpt-4o-mini', messages: [{ role: 'user', content: 'bye' }] };
|
|
13
|
-
assert.notEqual(cache.buildCacheKey(a), cache.buildCacheKey(b));
|
|
14
|
-
});
|
|
15
|
-
|
|
16
|
-
test('buildCacheKey differs when model differs, same messages', () => {
|
|
17
|
-
const messages = [{ role: 'user', content: 'hi' }];
|
|
18
|
-
const a = { model: 'gpt-4o-mini', messages };
|
|
19
|
-
const b = { model: 'claude-haiku-4-5-20251001', messages };
|
|
20
|
-
assert.notEqual(cache.buildCacheKey(a), cache.buildCacheKey(b));
|
|
21
|
-
});
|
|
22
|
-
|
|
23
|
-
test('buildCacheKey treats an unset temperature the same as 0.0 (documented default)', () => {
|
|
24
|
-
const a = { model: 'gpt-4o-mini', messages: [{ role: 'user', content: 'hi' }] };
|
|
25
|
-
const b = { model: 'gpt-4o-mini', messages: [{ role: 'user', content: 'hi' }], temperature: 0.0 };
|
|
26
|
-
assert.equal(cache.buildCacheKey(a), cache.buildCacheKey(b));
|
|
27
|
-
});
|
|
28
|
-
|
|
29
|
-
test('isConnected() is false with no REDIS_URL configured', () => {
|
|
30
|
-
// This test suite never sets REDIS_URL, matching the documented
|
|
31
|
-
// graceful-degradation path (cache disabled, not crashed).
|
|
32
|
-
assert.equal(cache.isConnected(), false);
|
|
33
|
-
});
|
package/test/embeddings.test.js
DELETED
|
@@ -1,24 +0,0 @@
|
|
|
1
|
-
const { test } = require('node:test');
|
|
2
|
-
const assert = require('node:assert/strict');
|
|
3
|
-
|
|
4
|
-
function freshEmbeddings(env) {
|
|
5
|
-
delete process.env.OPENAI_API_KEY;
|
|
6
|
-
Object.assign(process.env, env);
|
|
7
|
-
delete require.cache[require.resolve('../embeddings')];
|
|
8
|
-
return require('../embeddings');
|
|
9
|
-
}
|
|
10
|
-
|
|
11
|
-
test('isEnabled() is false with no OPENAI_API_KEY', () => {
|
|
12
|
-
const embeddings = freshEmbeddings({});
|
|
13
|
-
assert.equal(embeddings.isEnabled(), false);
|
|
14
|
-
});
|
|
15
|
-
|
|
16
|
-
test('isEnabled() is true once OPENAI_API_KEY is set', () => {
|
|
17
|
-
const embeddings = freshEmbeddings({ OPENAI_API_KEY: 'sk-test-fake' });
|
|
18
|
-
assert.equal(embeddings.isEnabled(), true);
|
|
19
|
-
});
|
|
20
|
-
|
|
21
|
-
test('embed() rejects clearly when disabled, without attempting a network call', async () => {
|
|
22
|
-
const embeddings = freshEmbeddings({});
|
|
23
|
-
await assert.rejects(() => embeddings.embed('hello'), /OPENAI_API_KEY not configured/);
|
|
24
|
-
});
|
package/test/env-path.test.js
DELETED
|
@@ -1,41 +0,0 @@
|
|
|
1
|
-
const { test } = require('node:test');
|
|
2
|
-
const assert = require('node:assert/strict');
|
|
3
|
-
const path = require('path');
|
|
4
|
-
|
|
5
|
-
// server.js reads its env vars once at require time; set a key so
|
|
6
|
-
// requiring it here never trips the fail-closed check for unrelated
|
|
7
|
-
// reasons (resolveEnvPathFromArgv itself doesn't touch env at all).
|
|
8
|
-
process.env.MODEL_ROUTER_INTERNAL_KEY = 'test-internal-key';
|
|
9
|
-
|
|
10
|
-
const { resolveEnvPathFromArgv } = require('../server');
|
|
11
|
-
|
|
12
|
-
test('resolveEnvPathFromArgv returns undefined when --env-path is absent - default cwd behavior unchanged', () => {
|
|
13
|
-
const argv = ['node', 'server.js'];
|
|
14
|
-
assert.equal(resolveEnvPathFromArgv(argv), undefined);
|
|
15
|
-
});
|
|
16
|
-
|
|
17
|
-
test('resolveEnvPathFromArgv resolves a space-separated --env-path value', () => {
|
|
18
|
-
const argv = ['node', 'server.js', '--env-path', './router/.env'];
|
|
19
|
-
assert.equal(resolveEnvPathFromArgv(argv), path.resolve('./router/.env'));
|
|
20
|
-
});
|
|
21
|
-
|
|
22
|
-
test('resolveEnvPathFromArgv resolves an --env-path=value (equals) form', () => {
|
|
23
|
-
const argv = ['node', 'server.js', '--env-path=./router/.env'];
|
|
24
|
-
assert.equal(resolveEnvPathFromArgv(argv), path.resolve('./router/.env'));
|
|
25
|
-
});
|
|
26
|
-
|
|
27
|
-
test('resolveEnvPathFromArgv resolves an absolute path unchanged', () => {
|
|
28
|
-
const absolute = path.resolve('/tmp/some/other/.env');
|
|
29
|
-
const argv = ['node', 'server.js', '--env-path', absolute];
|
|
30
|
-
assert.equal(resolveEnvPathFromArgv(argv), absolute);
|
|
31
|
-
});
|
|
32
|
-
|
|
33
|
-
test('resolveEnvPathFromArgv returns undefined if --env-path is the last arg with no value after it', () => {
|
|
34
|
-
const argv = ['node', 'server.js', '--env-path'];
|
|
35
|
-
assert.equal(resolveEnvPathFromArgv(argv), undefined);
|
|
36
|
-
});
|
|
37
|
-
|
|
38
|
-
test('resolveEnvPathFromArgv is unaffected by other, unrelated flags', () => {
|
|
39
|
-
const argv = ['node', 'server.js', '--some-other-flag', '--env-path', './x/.env'];
|
|
40
|
-
assert.equal(resolveEnvPathFromArgv(argv), path.resolve('./x/.env'));
|
|
41
|
-
});
|