cachegate 1.0.0

package/ROADMAP.md

@@ -0,0 +1,281 @@
+# MemoCode Model Router — Roadmap & Review
+
+> Review of `210_apps/001_model_router/` as it stands, and the plan to take it
+> from an internal MemoCode utility to a standalone self-hosted product.
+> Originally written by DeepSeek, 2026-08-23, auditing the version that
+> existed on `deepseek/memocode-chat-and-router` at the time. **Corrected the
+> same day**: that audit's "gaps" section was accurate for the version it was
+> looking at, but a second, much more complete implementation had been built
+> in parallel on `claude/memocode-project-review-v0ohnx` and never merged
+> anywhere shared — neither branch's author knew the other's existed. This
+> file now reflects the merged, actual state. See section 0.
+
+## 0. What happened, so it doesn't happen again
+
+Two independent model-router implementations existed at once: this one
+(minimal - proxy, exact-match cache, provider mapping) on
+`deepseek/memocode-chat-and-router`, and a materially more complete one
+(routing strategies, semantic cache, streaming, JSONL metrics with rotation,
+a cost dashboard, hardened auth, 68 tests) built the same day on
+`claude/memocode-project-review-v0ohnx` - a branch neither this file's
+original audit nor DeepSeek's own semantic-cache attempt knew to check.
+Discovered when the CEO asked to test the semantic cache and DeepSeek,
+checking `master`/`deepseek/memocode-chat-and-router`/
+`claude/session-import-feature`, correctly found it on none of them - but
+hadn't checked the actual branch it lived on. Resolved by replacing this
+directory's contents with the more complete version (a strict superset
+except for this file) and keeping this file's own market/monetization
+analysis, which stands independent of which codebase it's describing.
+**The lesson, not just the fix**: a feature branch that never gets
+merged toward `master` (or at least announced on the coordination board
+with its branch name) is invisible to everyone not already looking at it,
+no matter how complete the work on it is - completeness doesn't substitute
+for discoverability.
+
+## 1. What's actually here (audited 2026-08-23, post-merge)
+
+**Working, tested, pushed** (`210_apps/001_model_router/`, this directory,
+now matching `claude/memocode-project-review-v0ohnx`):
+- `server.js` — Express, OpenAI-compatible `POST /v1/chat/completions`
+  (streaming and non-streaming), internal bearer-key auth (fails CLOSED by
+  default - `ALLOW_INSECURE_LOCAL_DEV=true` required to opt into running
+  without a key), rate limiting, `/health`, `/stats`, `/dashboard`.
+- `providers/anthropic.js` + `providers/openai.js` — request mapping,
+  token/cost estimation, `chatStream()` for SSE.
+- `cache.js` — Redis exact-match cache keyed by a content hash of
+  `{model, messages, temperature, max_tokens, tools, tool_choice}`.
+- `semanticCache.js` + `embeddings.js` — near-duplicate matching via
+  embeddings + cosine similarity over a bounded per-model Redis list
+  (brute-force, not RediSearch/vector-indexed - an honest, documented
+  scale limit, not a hidden one). Tool-calling requests excluded. Semantic
+  hits tracked separately from exact hits in metrics, never blended into
+  one number.
+- `router.js` — three explicit `router:` virtual-model strategies (cost /
+  latency / latency-guarded-cost), no invented blended score.
+- `metrics.js` — JSONL logs with day-rotation, bounded reads, opt-in
+  pruning.
+- `public/dashboard.html` — a real cost dashboard (KPI tiles, charts,
+  auto-refresh, table-view twins), built to this project's own `dataviz`
+  skill standard.
+- 68 tests across 8 test files, all passing.
+- Live: dogfooded by `210_apps/000_backend/chat-import-logic.mjs`
+  (`localhost:4000/v1/chat/completions` for chat segmentation).
+
+**Real remaining gaps** (honest, not "nothing left to do"; updated
+2026-08-29 after re-auditing against the actual code - two of the four
+gaps below turned out to already be closed):
+1. Semantic cache is brute-force cosine over a capped list, not a real
+   vector index - fine at self-hosted single-instance volume, not meant to
+   scale past `SEMANTIC_CACHE_MAX_CANDIDATES` (default 200) per model.
+2. ~~No provider failover on a 5xx/rate-limit~~ **Closed 2026-08-29.**
+   `failover.js` + `server.js`'s non-streaming dispatch path now walk
+   `router.js`'s full ranked-candidate list, retrying the next candidate
+   when one fails for a reason that isn't the request's own fault (a
+   400/404 still fails immediately - retrying elsewhere wouldn't help).
+   Streaming is a deliberate exception, documented inline in
+   `handleStreamingDispatch` - SSE headers and the first frame commit to
+   a model name before a failure could be known, so silent mid-stream
+   provider switching is a materially harder problem, left open rather
+   than shipped half-working.
+3. ~~No persistence layer under the dashboard~~ **Closed** (already true
+   before this pass, just never updated here): `metrics.js` has stored
+   metrics in Postgres since the `feat(model-router): persistent metrics
+   storage via Postgres` commit, with automatic JSONL fallback when no
+   database is configured.
+4. Not yet open-sourced/packaged standalone (Phase 5, section 4 below).
+
+**What this actually is today, said plainly**: working, tested internal
+infrastructure with zero users outside this codebase — not an app, not a
+published package, not a running public service, not something anyone has
+paid for or even tried. Everything from here through section 3 (market
+position, monetization tiers, MRR estimates) describes a *hypothetical*
+product this code could become, not a claim about what it is right now.
+Read it as a plan, not a status report - a CEO challenge on 2026-08-23
+("this is not an app, it's not portable, we build nothing") is a fair
+description of today's reality and is what section 4's Phase 5 (open
+source release) exists to close.
+
+## 2. Honest market position
+
+Do **not** try to out-LiteLLM LiteLLM (140+ providers, Python, huge community)
+or OpenRouter (acquired by Stripe). The winnable niche:
+
+- **Self-hosted first** — prompts/PII never leave your infra.
+- **Semantic cache**, not just exact-match (most OSS options still hash-match).
+- **Node.js/TypeScript** — the JS/TS AI-app crowd is underserved.
+- **Embeddable** — usable as a module inside an existing app before it's sold.
+
+**Correction, same day, on the semantic-cache claim specifically**: verified
+via WebSearch that LiteLLM shipped a real vector-indexed (Valkey-search +
+HNSW) semantic cache in 2026 - more sophisticated than this router's
+brute-force cosine scan. "Semantic cache, not just exact-match" is no longer
+a differentiator against LiteLLM by name, even though it's still true against
+"most OSS options." The honest remaining edges: small/auditable codebase,
+Node-native, and already dogfooded inside a real app.
+
+The sellable, honest claim (do not quote inflated 86–95%): **"Caching alone
+typically saves 20–45%; add routing and it can reach 47–90% on well-tuned
+traffic."** Being the vendor that quotes the real number is the credibility
+edge with the buyer who has been burned by an inflated claim.
+
+## 3. How this router makes money (open-core, honest)
+
+Self-hosted OSS infrastructure monetizes by **open-core**, not by charging for
+the free thing:
+
+| Tier | What | Price | Why someone pays |
+|---|---|---|---|
+| **OSS core (free)** | exact-match cache, 2–3 providers, basic proxy | $0 | top-of-funnel; builds trust |
+| **Pro** | semantic cache, cost-based routing + failover, cost dashboard | $49–99/mo | the "save 40–80% on LLM spend" features; charge ~10–20% of the savings |
+| **Team** | multi-user, SSO, audit log, priority support | $199–499/mo | small teams running production workloads |
+| **Hosted** | we run it for you (no ops) | usage-based, $99–999/mo | teams that don't want to self-host (where LiteLLM makes its money) |
+| **Enterprise** | on-prem, compliance, SLAs | custom | regulated buyers |
+
+**Honest caveats:** OSS → paid conversion is typically 1–5%; the money is in
+the small fraction who want the semantic cache + dashboard and don't want to
+self-host. It is a crowded market. Realistic: **$5–20K MRR in 12 months** by
+owning the self-hosted Node.js niche. The metric that makes it work: a buyer
+spending $1,000/mo on LLM APIs saves $400–800/mo with this — paying $50–100/mo
+for the tool is an easy yes.
+
+## 4. Phases
+
+**Note on an older numbering (added 2026-08-29):** early journal entries
+(2026-08-24) refer to this work as a flat "20-step roadmap" (steps
+13-17 individually named there; 18 marked done but never described; 19
+= the semantic-cache vector-index upgrade, paused; 20 = the standalone
+product). That numbering was never written down as one document - it
+only ever existed as scattered journal references, which made it
+genuinely hard to reconstruct later (confirmed 2026-08-29: step 18's
+actual content couldn't be found anywhere). **The 6 phases below
+supersede that numbering entirely.** If an old "step N" reference ever
+surfaces again, map it here rather than trying to revive the flat list.
+
+### Phase 1 — make what exists real — **done**
+Redis verified end-to-end, honest README, tests (68, not the "no test files"
+this section originally reported), the honest cost claim above.
+
+### Phase 2 — semantic cache — **done**
+Embeddings + cosine similarity, `SEMANTIC_CACHE_THRESHOLD` default 0.93 (this
+section's own original draft proposed ~0.95 - close, tuned during real
+testing; see `semanticCache.js`'s own comment for the reasoning), tool-calls
+excluded, tracked separately from exact-match hits.
+
+### Phase 3 — real routing — **done**
+Three explicit strategies (`router.js`) rather than a single invented
+blended score - cost-ascending, latency-then-cost, and latency-guarded-cost
+(excludes anything too much slower than the fastest known candidate).
+Failover on error for non-streaming requests shipped 2026-08-29
+(`failover.js`) - see section 1, gap 2, for what it does and doesn't cover.
+
+### Phase 4 — cost visibility — **done**
+`metrics.js` (JSONL, day-rotation, bounded reads) + `public/dashboard.html`
+(KPI tiles, charts, auto-refresh, table-view twins).
+
+### Phase 5 — open-source release — not started
+Public GitHub repo → npm package → Docker image → announce (Hacker News,
+r/LocalLLaMA, r/selfhosted, Dev.to). Still the real next milestone - none of
+phases 1-4 being done changes that this hasn't shipped to anyone outside
+this project yet. **Execution plan, added 2026-08-29:**
+[`OPEN_SOURCE_ROADMAP.md`](./OPEN_SOURCE_ROADMAP.md) - the 20-step
+breakdown of everything this one-liner actually requires, written down
+as one document on purpose (see that file's own note on why).
+
+### Phase 6 — hosted tier (only after OSS traction)
+Render/Fly hosted; usage-based pricing.
+
+## 5. One honest caveat on dogfooding
+
+Chat-import's one segmentation call per session is too low-repeat to prove
+the *cache* saves money — it only proves the *proxy* works. **2026-08-23
+update**: `210_apps/000_backend/ai-providers.mjs`'s "Generate with AI" now
+routes through this router too (app's own shared key only, never a
+signed-in user's BYOK key — the router is one shared-secret proxy today,
+not multi-tenant).
+
+**Correction, same day, on how that update was first worded here**: this
+section originally called that traffic "a genuinely more promising source
+of real cache hits" — that was a hunch stated as a finding, and a CEO
+challenge caught it (fair: "what are the odds two people ask the same
+thing" is the right question, and for this app's actual traffic the honest
+answer is "mostly low, and nobody has measured it"). The one real
+structural fact in favor of *some* hits: `ai-providers.mjs` builds that
+prompt as `"Subject: <topic>. Generate about N <label> worth of
+content."` — short and templated, with `topic` the only variable — so two
+different users both studying, say, "the Roman Empire" would produce an
+identical or near-identical request, which chat-import's full conversation
+transcripts basically never do. That's a plausible *mechanism* for a
+non-zero hit rate on a study app whose users cluster around common
+curriculum topics — it is not a measured hit rate, and treating it as one
+would repeat the same mistake. Nobody has looked at MemoCode's actual
+topic distribution. `MODEL_ROUTER_URL` is also still unset in the current
+Render deploy (`render.yaml` declares no router service), so none of this
+runs in production yet regardless.
+
+To validate savings for real, you need either that traffic running through
+the router in production with real measurement, or synthetic load testing
+against a realistic topic distribution — not a plausible-sounding argument
+for why it might work, however structurally reasonable that argument is.
+
+## 6. Two-version architecture (decision 2026-08-23)
+
+One engine (routing + cache + metrics), two wrappers — NOT two codebases:
+
+| | Embedded | Standalone |
+|---|---|---|
+| Lives | inside MemoCode (own process) | its own hosted service (Render) |
+| Keys | you bring Anthropic/OpenAI | it holds keys, issues its own key + URL |
+| Login/billing | none | yes (login, payment, multi-tenancy) |
+| Serves | your apps (fallback) | everybody, including your apps |
+| Status | done — needs Redis + production wiring | future project (OpenRouter/LiteLLM competitor) |
+
+Fallback: apps point at the standalone and drop back to the embedded
+(localhost or MemoCode's own instance) when the standalone is down. Same job,
+so the switch is invisible to the caller. Do NOT fork the router — wrap it:
+same core logic, two thin deployment shells, so the two can never drift apart
+the way the two parallel implementations did (section 0).
+
+The embedded version **stays embedded in each new app** (each app inherits
+this directory as-is) — it is NOT extracted to a shared repo. A shared repo
+means maintaining one router copy per repo plus separate hosting/db, which
+isn't worth it; a new app just copies `210_apps/001_model_router/` into its
+own backend and points `MODEL_ROUTER_URL` at it. The standalone's key+URL are
+left as comments in the app's `.env` as a future reminder, not wired yet.
+
+Sequencing: finish the embedded first (cheap, real, no risk), then decide
+whether to build the standalone's billing layer (the hard, deferred part).
+
+## 7. Embedded — production checklist
+
+1. Hosted Redis (Render Key Value) → set `REDIS_URL`.
+2. Deploy this directory as its own service (or co-locate with the backend).
+   The repo `render.yaml` now declares a `memocode-router` service (Node
+   runtime, `npm ci` + `npm start`) and a `Dockerfile` is included for
+   Docker-based deploys.
+3. Set production env: `MODEL_ROUTER_INTERNAL_KEY` (strong, `openssl rand -hex 32`),
+   `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `REDIS_URL`,
+   `SEMANTIC_CACHE_ENABLED=false` (low-repeat traffic: semantic cache spends
+   an embedding call on every miss for nothing), `ROUTER_STRATEGY=cost`.
+4. Route MemoCode's AI call sites through it via `MODEL_ROUTER_URL`.
+   **Corrected 2026-08-29**: this item previously named "image gen" and
+   "transcription" as call sites to route through the router - checked
+   directly, and that's not actually possible as scoped. The router is an
+   OpenAI-*chat-completions*-compatible proxy only; image generation
+   (`images.generate`) and Whisper transcription
+   (`audio.transcriptions.create`) are different API shapes it doesn't
+   speak. What's actually wired today: chat-import's segmentation call
+   (`chat-import-logic.mjs`) and "Generate with AI" text generation
+   (`ai-providers.mjs`'s `generateStructuredText`), both gated behind
+   `MODEL_ROUTER_URL` being set (`shouldRouteToModelRouter()`). Extending
+   the router to proxy image/audio calls too would be new scope, not a
+   pending item on this checklist.
+5. Verify `/health`, `/dashboard`, `/stats` on production with real traffic.
+
+**Note on items 1 and 3 above (2026-08-29):** `render.yaml` declares the
+env var slots for `REDIS_URL`, `MODEL_ROUTER_INTERNAL_KEY`,
+`ANTHROPIC_API_KEY`, and `OPENAI_API_KEY` (all `sync: false` - filled in
+manually on Render's dashboard), and the metrics Postgres connection is
+wired automatically via `fromDatabase`. Whether those manual slots are
+actually populated with real values on the live Render service isn't
+something a sandboxed session can check - that verification needs
+whoever has the Render dashboard.

package/SECURITY.md

@@ -0,0 +1,39 @@
+# Security Policy
+
+## Reporting a vulnerability
+
+**Please do not open a public GitHub issue for a security
+vulnerability.** A public issue is a disclosure before a fix exists.
+
+Instead, use GitHub's private vulnerability reporting:
+
+1. Go to this repository's **Security** tab.
+2. Click **Report a vulnerability**.
+3. Describe the issue — what it is, how to reproduce it, and its
+   likely impact (e.g. "bypasses auth," "leaks another deployment's
+   cached data," "exhausts memory regardless of rate limiting").
+
+This opens a private conversation with the maintainers, visible only to
+you and them, and lets a fix be prepared and released before the
+vulnerability is public.
+
+*(Maintainer note, remove once live: this requires "Private vulnerability
+reporting" to be turned on for the repository — Settings → Security →
+Private vulnerability reporting — as part of step 15's repo setup.)*
+
+## What counts as a security issue here
+
+Concretely, for this project: anything that lets a request bypass
+`MODEL_ROUTER_INTERNAL_KEY` auth, read or corrupt another deployment's
+cached data or metrics, or exhaust CPU/memory/Redis storage in a way
+`RATE_LIMIT_MAX`/`RATE_LIMIT_WINDOW_MS` doesn't already bound. A
+provider returning an unexpected error, a routing decision you disagree
+with, or a missing feature are regular bugs — open a normal issue for
+those (see `CONTRIBUTING.md`).
+
+## Supported versions
+
+This project is pre-1.0 (see `OPEN_SOURCE_ROADMAP.md` on the version
+plan) — security fixes go into the latest release only. Once a stable
+1.0 line exists, this section will name which major versions still
+receive fixes.

package/cache.js

@@ -0,0 +1,51 @@
+// model-router/cache.js
+//
+// The exact-match cache: same model + same messages + same params ->
+// same cached response, by content hash. First and free - checked
+// before the semantic cache (semanticCache.js), which is slower (an
+// embedding call) and probabilistic (a similarity threshold, not an
+// exact match). Connection is shared via redisClient.js.
+
+const crypto = require('crypto');
+const redis = require('./redisClient');
+
+function buildCacheKey(payload) {
+  const normalized = JSON.stringify({
+    model: payload.model,
+    messages: payload.messages,
+    temperature: payload.temperature ?? 0.0,
+    max_tokens: payload.max_tokens,
+    tools: payload.tools,
+    tool_choice: payload.tool_choice
+  });
+  const hash = crypto.createHash('sha256').update(normalized).digest('hex');
+  return `ROUTER:${payload.model}:${hash}`;
+}
+
+module.exports = {
+  buildCacheKey,
+
+  isConnected() {
+    return redis.isConnected();
+  },
+
+  async get(payload) {
+    if (!redis.isConnected()) return null;
+    try {
+      const cached = await redis.client.get(buildCacheKey(payload));
+      return cached ? JSON.parse(cached) : null;
+    } catch (err) {
+      return null;
+    }
+  },
+
+  async set(payload, response, ttlSeconds = 3600) {
+    if (!redis.isConnected()) return false;
+    try {
+      await redis.client.set(buildCacheKey(payload), JSON.stringify(response), { EX: ttlSeconds });
+      return true;
+    } catch (err) {
+      return false;
+    }
+  }
+};

package/embeddings.js

@@ -0,0 +1,32 @@
+// model-router/embeddings.js
+//
+// The only embedding backend right now is OpenAI's - which means
+// semantic caching needs an OPENAI_API_KEY configured even for a
+// deployment that only ever talks to Anthropic for chat. That's a real
+// constraint, not hidden: isEnabled() is what semanticCache.js checks
+// before doing anything, and it degrades to "disabled" (not an error)
+// when the key isn't set, same as the Redis cache does when REDIS_URL
+// isn't set.
+
+const { OpenAI } = require('openai');
+
+let client;
+function getClient() {
+  if (!client) client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+  return client;
+}
+
+function isEnabled() {
+  return Boolean(process.env.OPENAI_API_KEY);
+}
+
+async function embed(text) {
+  if (!isEnabled()) {
+    throw new Error('OPENAI_API_KEY not configured - embeddings unavailable');
+  }
+  const model = process.env.EMBEDDING_MODEL || 'text-embedding-3-small';
+  const response = await getClient().embeddings.create({ model, input: text });
+  return response.data[0].embedding;
+}
+
+module.exports = { isEnabled, embed };

package/failover.js

@@ -0,0 +1,76 @@
+// model-router/failover.js
+//
+// Pure control flow for trying a virtual model's ranked candidates in
+// order until one succeeds - isolated from the actual provider-calling
+// code (server.js's dispatchToProvider) so it's directly unit-testable
+// with a fake dispatch function, no live API keys or network calls
+// needed. Same reasoning as providers/*.js's own
+// applyStreamEvent/applyStreamChunk factoring: the trickiest logic
+// shouldn't require a real provider to test.
+//
+// Addresses ROADMAP.md's gap #2 ("no provider failover on a
+// 5xx/rate-limit"): router.js's pickCandidate() already ranks every
+// candidate in a tier by the configured strategy, but server.js used
+// to dispatch to the top-ranked one only - if it failed, the whole
+// request failed, even when a second healthy candidate existed in the
+// same tier. This module is what actually walks that ranked list.
+
+// Whether a failed dispatch attempt is worth retrying against the NEXT
+// candidate, vs failing the request outright. The distinction: is the
+// REQUEST itself broken (retrying elsewhere would fail identically),
+// or did THIS provider fail in a way another provider might not (rate
+// limit, an outage, a bad or expired key)? Bad request (400) and
+// unknown model (404) are the request's own fault, not retried - the
+// Anthropic and OpenAI SDKs both set `.status` on a thrown APIError.
+// A network-level failure with no HTTP response at all (no `.status`)
+// is treated as the provider's fault too, since it's not the
+// request's content that's the problem. An auth failure (401) or
+// missing-key misconfiguration is ALSO treated as retryable on
+// purpose: a different candidate in the tier may use a different
+// provider whose key is fine, so the request can still succeed - the
+// broken key itself still surfaces on the dashboard's Provider alerts
+// table via the metrics.record() call made before moving on (see
+// server.js), so failover keeps requests succeeding without hiding
+// the underlying problem from whoever needs to go fix that key.
+function isRetryableError(err) {
+  const status = err && (err.status || err.statusCode);
+  return status !== 400 && status !== 404;
+}
+
+/**
+ * Calls `dispatch(candidate)` for each candidate in order until one
+ * resolves. On a rejection, calls `onAttemptFailed(candidate, err,
+ * isLastCandidate)` (for logging/metrics only - it has no bearing on
+ * control flow) and, unless the error is non-retryable or this was
+ * the last candidate, moves on to the next one. Rethrows the error
+ * from the LAST attempt if every candidate fails - the caller decides
+ * what HTTP status/response that becomes.
+ *
+ * Resolves to `{ result, candidate, attempts }` on success -
+ * `attempts` is 1 when the first candidate just worked, >1 when
+ * failover actually happened (worth logging distinctly - see
+ * server.js's caller).
+ *
+ * @param {Array<{provider: string, model: string}>} candidates ranked
+ *   order, e.g. router.js's pickCandidate().rankedCandidates
+ * @param {(candidate) => Promise<any>} dispatch
+ * @param {(candidate, err, isLastCandidate) => void} [onAttemptFailed]
+ */
+async function dispatchWithFailover(candidates, dispatch, onAttemptFailed) {
+  let lastErr;
+  for (let i = 0; i < candidates.length; i++) {
+    const candidate = candidates[i];
+    const isLastCandidate = i === candidates.length - 1;
+    try {
+      const result = await dispatch(candidate);
+      return { result, candidate, attempts: i + 1 };
+    } catch (err) {
+      lastErr = err;
+      if (onAttemptFailed) onAttemptFailed(candidate, err, isLastCandidate);
+      if (!isRetryableError(err) || isLastCandidate) throw err;
+    }
+  }
+  throw lastErr; // unreachable when candidates.length > 0; kept honest for an empty list
+}
+
+module.exports = { isRetryableError, dispatchWithFailover };