cachegate 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +127 -112
- package/README.md +31 -13
- package/cache.js +72 -51
- package/embeddings.js +42 -32
- package/metrics.js +609 -556
- package/package.json +15 -1
- package/redisClient.js +55 -45
- package/router.js +254 -218
- package/semanticCache.js +159 -154
- package/server.js +282 -113
- package/.dockerignore +0 -11
- package/.gitattributes +0 -12
- package/.github/ISSUE_TEMPLATE/bug_report.md +0 -33
- package/.github/ISSUE_TEMPLATE/config.yml +0 -5
- package/.github/ISSUE_TEMPLATE/feature_request.md +0 -29
- package/.github/PULL_REQUEST_TEMPLATE.md +0 -25
- package/.github/workflows/test.yml +0 -63
- package/CODE_OF_CONDUCT.md +0 -66
- package/CONTRIBUTING.md +0 -94
- package/Dockerfile +0 -24
- package/OPEN_SOURCE_ROADMAP.md +0 -855
- package/ROADMAP.md +0 -281
- package/SECURITY.md +0 -39
- package/sync-oss-release.sh +0 -160
- package/test/auth-config.test.js +0 -27
- package/test/cache.test.js +0 -33
- package/test/embeddings.test.js +0 -24
- package/test/env-path.test.js +0 -41
- package/test/failover.test.js +0 -99
- package/test/metrics-postgres.test.js +0 -183
- package/test/metrics.test.js +0 -282
- package/test/router.test.js +0 -195
- package/test/semanticCache.test.js +0 -167
- package/test/server.test.js +0 -357
- package/test/streaming.test.js +0 -248
|
@@ -1,33 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: Bug report
|
|
3
|
-
about: Something isn't working the way it should
|
|
4
|
-
title: ''
|
|
5
|
-
labels: bug
|
|
6
|
-
assignees: ''
|
|
7
|
-
---
|
|
8
|
-
|
|
9
|
-
**What happened**
|
|
10
|
-
A clear description of the actual behavior.
|
|
11
|
-
|
|
12
|
-
**What you expected instead**
|
|
13
|
-
|
|
14
|
-
**Steps to reproduce**
|
|
15
|
-
1.
|
|
16
|
-
2.
|
|
17
|
-
3.
|
|
18
|
-
|
|
19
|
-
**Environment**
|
|
20
|
-
- cachegate version:
|
|
21
|
-
- Node version:
|
|
22
|
-
- Deployment: standalone / embedded in another app / Docker
|
|
23
|
-
- Redis: version, and whether it's configured at all (some features
|
|
24
|
-
degrade cleanly without it — see README's "Features")
|
|
25
|
-
|
|
26
|
-
**Relevant config**
|
|
27
|
-
Which env vars are set (names only — **never paste real API keys, the
|
|
28
|
-
internal key, or a database URL here**): e.g. `ROUTER_STRATEGY=cost`,
|
|
29
|
-
`SEMANTIC_CACHE_ENABLED=true`, `DATABASE_URL` set (Postgres) vs. unset
|
|
30
|
-
(JSONL).
|
|
31
|
-
|
|
32
|
-
**Logs**
|
|
33
|
-
Relevant server console output, with any secret values redacted.
|
|
@@ -1,29 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: Feature request
|
|
3
|
-
about: Suggest something this project should do
|
|
4
|
-
title: ''
|
|
5
|
-
labels: enhancement
|
|
6
|
-
assignees: ''
|
|
7
|
-
---
|
|
8
|
-
|
|
9
|
-
**Before filing — is this in scope?**
|
|
10
|
-
Read `README.md`'s "What this is NOT" section first. Login, billing,
|
|
11
|
-
multi-tenant key custody, or a hosted-service feature will be closed as
|
|
12
|
-
out of scope regardless of how it's proposed — that functionality
|
|
13
|
-
belongs to a separate closed product built on this engine, not to this
|
|
14
|
-
engine. Everything else (routing, caching, provider support, the
|
|
15
|
-
dashboard, operational tooling) is fair game.
|
|
16
|
-
|
|
17
|
-
**What problem does this solve?**
|
|
18
|
-
Describe the actual pain, not just the feature - "I need X because Y
|
|
19
|
-
currently forces me to Z."
|
|
20
|
-
|
|
21
|
-
**What would the ideal solution look like?**
|
|
22
|
-
A sketch is fine - an API shape, a new env var, a CLI flag.
|
|
23
|
-
|
|
24
|
-
**Alternatives considered**
|
|
25
|
-
What are you doing today without this? Is there a workaround?
|
|
26
|
-
|
|
27
|
-
**Additional context**
|
|
28
|
-
Anything else - a link to how a similar tool does this, a real
|
|
29
|
-
production scenario this came up in.
|
|
@@ -1,25 +0,0 @@
|
|
|
1
|
-
## What changed and why
|
|
2
|
-
|
|
3
|
-
<!-- The problem this solves, not just the diff. If it fixes a bug,
|
|
4
|
-
say how you found and verified it. -->
|
|
5
|
-
|
|
6
|
-
## Scope check
|
|
7
|
-
|
|
8
|
-
- [ ] I've read `README.md`'s "What this is NOT" section and confirmed
|
|
9
|
-
this doesn't add login, billing, multi-tenant key custody, or a
|
|
10
|
-
hosted-service feature.
|
|
11
|
-
|
|
12
|
-
## Testing
|
|
13
|
-
|
|
14
|
-
<!-- What you ran, and what it proved. "npm test passes" is fine for a
|
|
15
|
-
small change; describe manual verification for anything that couldn't
|
|
16
|
-
be covered by the existing suite (e.g. a real provider call). -->
|
|
17
|
-
|
|
18
|
-
- [ ] `npm test` passes locally
|
|
19
|
-
- [ ] Added/updated tests for the behavior this changes, or explained
|
|
20
|
-
why that's not practical (e.g. it needs a live provider call)
|
|
21
|
-
|
|
22
|
-
## Anything reviewers should look at closely
|
|
23
|
-
|
|
24
|
-
<!-- Optional - a tricky edge case, a tradeoff you're not fully sure
|
|
25
|
-
about, a place you deviated from an existing pattern in the codebase. -->
|
|
@@ -1,63 +0,0 @@
|
|
|
1
|
-
name: Tests
|
|
2
|
-
|
|
3
|
-
# Runs on every push and PR - unlike this monorepo's own root
|
|
4
|
-
# .github/workflows/e2e.yml (currently workflow_dispatch-only), this
|
|
5
|
-
# is a public OSS repo once extracted: a green check on every PR IS
|
|
6
|
-
# the trust signal step 8 of OPEN_SOURCE_ROADMAP.md exists for, so
|
|
7
|
-
# automatic triggers are the whole point here, not something to dial
|
|
8
|
-
# back to manual.
|
|
9
|
-
on:
|
|
10
|
-
push:
|
|
11
|
-
branches: [main]
|
|
12
|
-
pull_request:
|
|
13
|
-
|
|
14
|
-
jobs:
|
|
15
|
-
test:
|
|
16
|
-
runs-on: ubuntu-latest
|
|
17
|
-
|
|
18
|
-
# metrics-postgres.test.js needs a REAL reachable Postgres - it
|
|
19
|
-
# fails loudly (ECONNREFUSED) rather than skipping if one isn't
|
|
20
|
-
# running, by design (see that file's own header comment). Values
|
|
21
|
-
# here match its own hardcoded fallback connection string exactly
|
|
22
|
-
# (MEMOCODE_ROUTER_TEST_DATABASE_URL's default), so no extra env
|
|
23
|
-
# wiring is needed below.
|
|
24
|
-
services:
|
|
25
|
-
postgres:
|
|
26
|
-
image: postgres:16
|
|
27
|
-
env:
|
|
28
|
-
POSTGRES_USER: postgres
|
|
29
|
-
POSTGRES_PASSWORD: dryrun
|
|
30
|
-
POSTGRES_DB: router_metrics_dryrun
|
|
31
|
-
ports:
|
|
32
|
-
- 5432:5432
|
|
33
|
-
options: >-
|
|
34
|
-
--health-cmd pg_isready
|
|
35
|
-
--health-interval 10s
|
|
36
|
-
--health-timeout 5s
|
|
37
|
-
--health-retries 5
|
|
38
|
-
|
|
39
|
-
strategy:
|
|
40
|
-
# Multiple Node versions, not just one: package.json declares
|
|
41
|
-
# "engines": {"node": ">=18.0.0"} - running the suite against all
|
|
42
|
-
# three actually backs that claim instead of just asserting it.
|
|
43
|
-
matrix:
|
|
44
|
-
node-version: ['18.x', '20.x', '22.x']
|
|
45
|
-
|
|
46
|
-
steps:
|
|
47
|
-
- uses: actions/checkout@v4
|
|
48
|
-
|
|
49
|
-
- uses: actions/setup-node@v4
|
|
50
|
-
with:
|
|
51
|
-
node-version: ${{ matrix.node-version }}
|
|
52
|
-
|
|
53
|
-
# semanticCache.test.js spawns its own throwaway redis-server
|
|
54
|
-
# process directly (a real instance, not a mock) - ubuntu-latest
|
|
55
|
-
# doesn't ship the binary, so it has to be installed explicitly.
|
|
56
|
-
# This is NOT the same thing as the Postgres service above: that
|
|
57
|
-
# test connects to an already-running external service, this one
|
|
58
|
-
# starts and stops its own process on a random port.
|
|
59
|
-
- name: Install redis-server
|
|
60
|
-
run: sudo apt-get update && sudo apt-get install -y redis-server
|
|
61
|
-
|
|
62
|
-
- run: npm ci
|
|
63
|
-
- run: npm test
|
package/CODE_OF_CONDUCT.md
DELETED
|
@@ -1,66 +0,0 @@
|
|
|
1
|
-
# Contributor Covenant Code of Conduct
|
|
2
|
-
|
|
3
|
-
## Our Pledge
|
|
4
|
-
|
|
5
|
-
We as members, contributors, and leaders pledge to make participation in
|
|
6
|
-
our community a harassment-free experience for everyone, regardless of
|
|
7
|
-
age, body size, visible or invisible disability, ethnicity, sex
|
|
8
|
-
characteristics, gender identity and expression, level of experience,
|
|
9
|
-
education, socio-economic status, nationality, personal appearance,
|
|
10
|
-
race, religion, or sexual identity and orientation.
|
|
11
|
-
|
|
12
|
-
We pledge to act and interact in ways that contribute to an open,
|
|
13
|
-
welcoming, diverse, inclusive, and healthy community.
|
|
14
|
-
|
|
15
|
-
## Our Standards
|
|
16
|
-
|
|
17
|
-
Examples of behavior that contributes to a positive environment:
|
|
18
|
-
|
|
19
|
-
* Demonstrating empathy and kindness toward other people
|
|
20
|
-
* Being respectful of differing opinions, viewpoints, and experiences
|
|
21
|
-
* Giving and gracefully accepting constructive feedback
|
|
22
|
-
* Accepting responsibility and apologizing to those affected by our
|
|
23
|
-
mistakes, and learning from the experience
|
|
24
|
-
* Focusing on what is best not just for us as individuals, but for the
|
|
25
|
-
overall community
|
|
26
|
-
|
|
27
|
-
Examples of unacceptable behavior:
|
|
28
|
-
|
|
29
|
-
* The use of sexualized language or imagery, and sexual attention or
|
|
30
|
-
advances of any kind
|
|
31
|
-
* Trolling, insulting or derogatory comments, and personal or political
|
|
32
|
-
attacks
|
|
33
|
-
* Public or private harassment
|
|
34
|
-
* Publishing others' private information, such as a physical or email
|
|
35
|
-
address, without their explicit permission
|
|
36
|
-
* Other conduct which could reasonably be considered inappropriate in a
|
|
37
|
-
professional setting
|
|
38
|
-
|
|
39
|
-
## Enforcement Responsibilities
|
|
40
|
-
|
|
41
|
-
Project maintainers are responsible for clarifying and enforcing our
|
|
42
|
-
standards of acceptable behavior and will take appropriate and fair
|
|
43
|
-
corrective action in response to any behavior that they deem
|
|
44
|
-
inappropriate, threatening, offensive, or harmful.
|
|
45
|
-
|
|
46
|
-
## Scope
|
|
47
|
-
|
|
48
|
-
This Code of Conduct applies within all community spaces (issues, pull
|
|
49
|
-
requests, discussions), and also applies when an individual is
|
|
50
|
-
officially representing the community in public spaces.
|
|
51
|
-
|
|
52
|
-
## Enforcement
|
|
53
|
-
|
|
54
|
-
Instances of abusive, harassing, or otherwise unacceptable behavior may
|
|
55
|
-
be reported to the maintainers via the contact method listed in
|
|
56
|
-
`SECURITY.md`. All complaints will be reviewed and investigated
|
|
57
|
-
promptly and fairly.
|
|
58
|
-
|
|
59
|
-
## Attribution
|
|
60
|
-
|
|
61
|
-
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
|
62
|
-
version 2.1, available at
|
|
63
|
-
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
|
|
64
|
-
|
|
65
|
-
[homepage]: https://www.contributor-covenant.org
|
|
66
|
-
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
|
package/CONTRIBUTING.md
DELETED
|
@@ -1,94 +0,0 @@
|
|
|
1
|
-
# Contributing to cachegate
|
|
2
|
-
|
|
3
|
-
Thanks for considering it. This is a small, deliberately focused
|
|
4
|
-
project — read the scope note below before writing code, it'll save
|
|
5
|
-
you a round-trip.
|
|
6
|
-
|
|
7
|
-
## Scope — read this first
|
|
8
|
-
|
|
9
|
-
`README.md`'s "What this is NOT" section is the actual contract, not
|
|
10
|
-
just a marketing caveat: **no login, no billing, no multi-tenant key
|
|
11
|
-
custody, no hosted service** live in this repository, and a PR adding
|
|
12
|
-
any of those will be closed regardless of how well it's built — that
|
|
13
|
-
functionality belongs to a separate, closed product built on top of
|
|
14
|
-
this engine, not this engine itself. This isn't a licensing
|
|
15
|
-
restriction (MIT permits building any of that — see `LICENSE`); it's
|
|
16
|
-
that this repository specifically isn't going to grow into a hosted
|
|
17
|
-
competitor to its own paid product, so a PR heading that direction
|
|
18
|
-
gets closed here regardless of quality, not merged and then diverged
|
|
19
|
-
from later.
|
|
20
|
-
|
|
21
|
-
Everything else — routing strategies, cache behavior, provider
|
|
22
|
-
support, the dashboard, bug fixes — is fair game.
|
|
23
|
-
|
|
24
|
-
## Running it locally
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
git clone <this-repo-url>
|
|
28
|
-
cd cachegate
|
|
29
|
-
npm install
|
|
30
|
-
cp .env.example .env # then fill in a real key for at least one provider
|
|
31
|
-
npm start
|
|
32
|
-
```
|
|
33
|
-
|
|
34
|
-
## Running the tests
|
|
35
|
-
|
|
36
|
-
```bash
|
|
37
|
-
npm test
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
This runs Node's built-in test runner (`node --test`) across every file
|
|
41
|
-
in `test/`. A few things worth knowing before you add to it:
|
|
42
|
-
|
|
43
|
-
- **No test here calls a real provider API.** That needs live keys and
|
|
44
|
-
real spend, which isn't reasonable to require of a contributor or a
|
|
45
|
-
CI run. Provider-facing logic (`providers/anthropic.js`,
|
|
46
|
-
`providers/openai.js`) is tested by factoring the pure,
|
|
47
|
-
non-network parts (see `applyStreamEvent`/`applyStreamChunk`) out
|
|
48
|
-
into directly-testable functions with canned input — follow that
|
|
49
|
-
pattern for new provider logic rather than trying to mock the HTTP
|
|
50
|
-
layer.
|
|
51
|
-
- **Tests need to be genuinely isolated.** Several existing tests reset
|
|
52
|
-
`METRICS_LOG_PATH` to a fresh temp file per test (see
|
|
53
|
-
`router.test.js`'s `freshModules` helper) specifically because
|
|
54
|
-
`metrics.js` reads its config once at require time. If your test
|
|
55
|
-
writes metrics, don't assume a clean slate — either isolate it the
|
|
56
|
-
same way, or place it deliberately last in its file if it needs to
|
|
57
|
-
run after everything else that depends on a clean state (see
|
|
58
|
-
`server.test.js`'s own comment on this for a worked example).
|
|
59
|
-
- All tests must pass before a PR is reviewed. If a test is failing for
|
|
60
|
-
a reason unrelated to your change, say so explicitly in the PR rather
|
|
61
|
-
than silently working around it — that's a real bug worth its own
|
|
62
|
-
issue.
|
|
63
|
-
|
|
64
|
-
## Opening a PR
|
|
65
|
-
|
|
66
|
-
- Explain the *why*, not just the *what* — a one-line "fixes X" is
|
|
67
|
-
fine for a trivial fix, but anything behavioral should say what
|
|
68
|
-
problem it solves and how you verified the fix, the same standard
|
|
69
|
-
this codebase's own commit history holds itself to.
|
|
70
|
-
- If you're touching `router.js`'s scoring logic, `failover.js`'s
|
|
71
|
-
retry logic, or anything else with existing inline documentation
|
|
72
|
-
explaining a past decision (search the file for "why" before
|
|
73
|
-
changing something that looks arbitrary) — it's very likely not
|
|
74
|
-
arbitrary. If you disagree with the reasoning, say so in the PR;
|
|
75
|
-
don't just silently remove it.
|
|
76
|
-
- Small, focused PRs review faster than large ones bundling several
|
|
77
|
-
unrelated changes.
|
|
78
|
-
|
|
79
|
-
**How merges actually work here, stated plainly:** this repository is
|
|
80
|
-
mirrored out from a private internal monorepo where day-to-day
|
|
81
|
-
development happens, rather than the other way around. Your PR gets
|
|
82
|
-
reviewed and, once approved, merged here on GitHub like normal — but
|
|
83
|
-
it's also manually reapplied on the internal side afterward, through
|
|
84
|
-
that project's own review process, rather than auto-syncing. In
|
|
85
|
-
practice this means a short delay between "merged here" and "in the
|
|
86
|
-
next internal release," not a rejection — you'll see it land in a
|
|
87
|
-
tagged release once that happens.
|
|
88
|
-
|
|
89
|
-
## Reporting a bug vs. reporting a security issue
|
|
90
|
-
|
|
91
|
-
Regular bugs: open a GitHub issue. Security vulnerabilities (anything
|
|
92
|
-
that could let a request bypass auth, leak another deployment's data,
|
|
93
|
-
or exhaust resources in a way rate limiting doesn't already cover): see
|
|
94
|
-
`SECURITY.md` instead — please don't file those as public issues.
|
package/Dockerfile
DELETED
|
@@ -1,24 +0,0 @@
|
|
|
1
|
-
FROM node:20-slim
|
|
2
|
-
|
|
3
|
-
# Standard OCI labels - GHCR reads image.source automatically on every push
|
|
4
|
-
# and uses it to link this package to the repo (the alternative to clicking
|
|
5
|
-
# "Connect Repository" by hand once, which doesn't survive a re-publish).
|
|
6
|
-
LABEL org.opencontainers.image.source="https://github.com/iDebunk/cachegate" \
|
|
7
|
-
org.opencontainers.image.description="Self-hostable, OpenAI-compatible LLM proxy: routes to the cheapest healthy provider, caches responses exactly and semantically, tracks cost and latency per call." \
|
|
8
|
-
org.opencontainers.image.licenses="MIT"
|
|
9
|
-
|
|
10
|
-
WORKDIR /app
|
|
11
|
-
COPY package*.json ./
|
|
12
|
-
RUN npm ci --omit=dev
|
|
13
|
-
COPY . .
|
|
14
|
-
# The official Node images already ship a non-root "node" user (uid 1000) -
|
|
15
|
-
# use it instead of running as root, standard practice for a public image.
|
|
16
|
-
RUN chown -R node:node /app
|
|
17
|
-
USER node
|
|
18
|
-
EXPOSE 4000
|
|
19
|
-
# No curl/wget in the slim base image - Node's own http module does the
|
|
20
|
-
# check instead. Uses the real GET /health endpoint (public, no auth,
|
|
21
|
-
# exists specifically for this).
|
|
22
|
-
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
|
23
|
-
CMD node -e "require('http').get('http://localhost:'+(process.env.PORT||4000)+'/health',(r)=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))"
|
|
24
|
-
CMD ["node", "server.js"]
|