@starci/ai-router 0.4.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +29 -0
- package/README.md +133 -40
- package/dist/ai-router.service.d.ts +2 -2
- package/dist/ai-router.service.js +4 -2
- package/dist/attempt.d.ts +26 -0
- package/dist/attempt.js +29 -0
- package/dist/classify.d.ts +40 -0
- package/dist/classify.js +145 -0
- package/dist/config.d.ts +30 -5
- package/dist/config.js +184 -24
- package/dist/env.d.ts +70 -0
- package/dist/env.js +152 -0
- package/dist/errors.d.ts +1 -3
- package/dist/errors.js +4 -23
- package/dist/fake.d.ts +624 -5
- package/dist/fake.js +177 -21
- package/dist/gateway/config-file.d.ts +8 -3
- package/dist/gateway/config-file.js +53 -8
- package/dist/gateway/gateway.d.ts +2 -0
- package/dist/gateway/gateway.js +19 -3
- package/dist/gateway/index.d.ts +10 -1
- package/dist/gateway/index.js +5 -0
- package/dist/gateway/serve.js +7 -2
- package/dist/headers.d.ts +1 -1
- package/dist/headers.js +12 -6
- package/dist/pool.d.ts +37 -16
- package/dist/pool.js +173 -48
- package/dist/providers.d.ts +34 -0
- package/dist/providers.js +22 -0
- package/dist/router.js +108 -105
- package/dist/secrets.d.ts +8 -0
- package/dist/secrets.js +22 -0
- package/dist/shapes.d.ts +11 -0
- package/dist/shapes.js +124 -0
- package/dist/state-stores.d.ts +38 -0
- package/dist/state-stores.js +101 -0
- package/dist/store.d.ts +26 -7
- package/dist/store.js +64 -2
- package/dist/testing.d.ts +2 -2
- package/dist/testing.js +1 -1
- package/dist/types.d.ts +126 -38
- package/dist/wire/anthropic.d.ts +21 -0
- package/dist/wire/anthropic.js +322 -0
- package/dist/wire/index.d.ts +5 -0
- package/dist/wire/index.js +7 -0
- package/dist/wire/wire.d.ts +42 -0
- package/dist/wire/wire.js +35 -0
- package/docs/design.md +279 -0
- package/package.json +12 -3
package/docs/design.md
ADDED
|
@@ -0,0 +1,279 @@
|
|
|
1
|
+
# @starci/ai-router design: many providers below, one virtual key above
|
|
2
|
+
|
|
3
|
+
This is the design of `@starci/ai-router` 0.5: how a pool of upstream providers, each with an array of keys, is
|
|
4
|
+
load-balanced behind one OpenAI-compatible gateway, how a key whose quota is spent or whose secret is refused is
|
|
5
|
+
blacklisted, and how an app declares all of it from env. The README is the reference of every option; this document
|
|
6
|
+
says why the pieces are shaped as they are, and what was read to shape them.
|
|
7
|
+
|
|
8
|
+
## The shape
|
|
9
|
+
|
|
10
|
+
```
|
|
11
|
+
clients (above) gateway (in process) providers (below)
|
|
12
|
+
─────────────── ──────────────────── ─────────────────
|
|
13
|
+
app LangChain client ─┐ ┌── anthropic [a-1, a-2] Messages API
|
|
14
|
+
agent runtime ────────┼─ OpenAI HTTP ─▶ virtual key ─▶ profile ─▶ router ──┼── openrouter [or-1..or-9] OpenAI
|
|
15
|
+
│ model: "work" STARCI_AI_KEY work lanes ├── qwen [dashscope-1] OpenAI
|
|
16
|
+
│ model: "embed" quotas, usage embed blacklist ├── deepseek [deepseek-1] OpenAI
|
|
17
|
+
└─────────────────────────────────────────── breakers ├── openai [openai-1..] OpenAI
|
|
18
|
+
budgets ├── gemini [gemini-1] OpenAI compat
|
|
19
|
+
└── groq, ollama, vllm, ... OpenAI
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
- **Above:** every client of an app speaks OpenAI chat-completions and embeddings to the gateway with one virtual key
|
|
23
|
+
per app and environment (`STARCI_AI_KEY`, `sk-starci-<label>-...`), mounted from a sealed secret
|
|
24
|
+
(`virtualKeyFromEnv`). It names a profile as its `model`. No app uses the admin token or the admin API; both stay in
|
|
25
|
+
the package for an operator who runs the standalone gateway.
|
|
26
|
+
- **Below:** each provider is declared with an array of keys (`<PREFIX>_API_KEY`, `_2`, ...). A lane is one
|
|
27
|
+
(provider, key, model). The router picks a lane per attempt by health, weight and headroom, and translates the
|
|
28
|
+
request into the provider's protocol.
|
|
29
|
+
|
|
30
|
+
## Provider matrix
|
|
31
|
+
|
|
32
|
+
Read from each provider's official documentation on 2026-10-31. Model ids move fast; they are an app's config, never the
|
|
33
|
+
package's. The table is what `PROVIDER_PRESETS` (`src/providers.ts`) encodes.
|
|
34
|
+
|
|
35
|
+
| kind | default base URL | protocol | auth header | embeddings | env prefix | docs read |
|
|
36
|
+
|---|---|---|---|---|---|---|
|
|
37
|
+
| `anthropic` | `https://api.anthropic.com/v1` (`/messages`) | Anthropic Messages, through the adapter | `x-api-key`, `anthropic-version: 2023-06-01` | none | `ANTHROPIC` | platform.claude.com/docs/en/api/errors, /api/rate-limits, /api/messages, /build-with-claude/streaming, /build-with-claude/handling-stop-reasons, /build-with-claude/structured-outputs, /models/overview |
|
|
38
|
+
| `openrouter` | `https://openrouter.ai/api/v1` | OpenAI | `Authorization: Bearer` (optional `HTTP-Referer`, `X-OpenRouter-Title` as provider `headers`) | yes (`/embeddings`) | `OPENROUTER` | openrouter.ai/docs/api/reference/overview, /errors-and-debugging, /limits, /embeddings |
|
|
39
|
+
| `qwen` | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` (Singapore; `DASHSCOPE_BASE_URL` for `dashscope.aliyuncs.com` or a workspace host `{WorkspaceId}.<region>.maas.aliyuncs.com`) | OpenAI (DashScope compatible mode) | `Authorization: Bearer` | yes (`text-embedding-v4`, default 1024 dims) | `DASHSCOPE` | alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope, /error-code, /embedding, /models |
|
|
40
|
+
| `deepseek` | `https://api.deepseek.com` | OpenAI | `Authorization: Bearer` | none | `DEEPSEEK` | api-docs.deepseek.com, /quick_start/error_codes, /quick_start/rate_limit |
|
|
41
|
+
| `openai` | `https://api.openai.com/v1` | OpenAI | `Authorization: Bearer` | yes (`text-embedding-3-small` 1536, `-large` 3072, `dimensions` accepted) | `OPENAI` | developers.openai.com/api/docs/guides/error-codes, /rate-limits, /embeddings |
|
|
42
|
+
| `gemini` | `https://generativelanguage.googleapis.com/v1beta/openai` | OpenAI compatibility (beta) | `Authorization: Bearer` | yes (`gemini-embedding-001`, default 3072) | `GEMINI` | ai.google.dev/gemini-api/docs/openai, /api-errors, /rate-limits, /embeddings, /troubleshooting |
|
|
43
|
+
| `groq` | `https://api.groq.com/openai/v1` | OpenAI | `Authorization: Bearer` | not declared (unverified) | `GROQ` | console.groq.com/docs/openai, /errors, /rate-limits |
|
|
44
|
+
| `openai-compatible` | required | OpenAI | `Authorization: Bearer` | yes | the app names one | docs.ollama.com/api/openai-compatibility, docs.vllm.ai/en/latest/serving/online_serving/ |
|
|
45
|
+
|
|
46
|
+
Facts that shaped the code:
|
|
47
|
+
|
|
48
|
+
- A Qwen key works only in the region it was created in; a key sent to another region answers 401 `invalid_api_key`,
|
|
49
|
+
which is a refused secret here. Give each region its own provider entry and base URL.
|
|
50
|
+
- Gemini's limits are per Google Cloud project, not per key, so several keys of one project share one bucket; spread
|
|
51
|
+
keys over projects for real headroom.
|
|
52
|
+
- DeepSeek's concurrency is per account; two keys of one account do not double it.
|
|
53
|
+
- OpenRouter can answer an error inside a 200 body, and mid-stream as a chunk with `error` and `finish_reason: "error"`;
|
|
54
|
+
both are classified like an error status.
|
|
55
|
+
- DeepSeek sends empty lines on a slow non-streaming answer and `: keep-alive` comments in a stream; JSON parsing and the
|
|
56
|
+
SSE parser skip both.
|
|
57
|
+
- Anthropic's own OpenAI-compatible endpoint is documented for testing, not production (no prompt caching, `json_object`
|
|
58
|
+
refused), so the package speaks the Messages API through its own adapter instead.
|
|
59
|
+
|
|
60
|
+
What the docs did not show and is marked unverified: the literal JSON error bodies of DashScope compatible mode and
|
|
61
|
+
DeepSeek (status codes are documented, bodies are assumed OpenAI-style; the classifier also reads their messages),
|
|
62
|
+
Gemini's list-wrapped error form (both forms are read), and Groq's embeddings.
|
|
63
|
+
|
|
64
|
+
## Error classification
|
|
65
|
+
|
|
66
|
+
`classifyFailure(kind, status, body)` (`src/classify.ts`) reads any of the shapes (OpenAI `{ error: { message, type,
|
|
67
|
+
code, param } }`, Anthropic `{ type: "error", error: { type, message, details } }`, OpenRouter `{ error: { code: <status>,
|
|
68
|
+
message, metadata } }`, Gemini object or `[{ error: { code, message, status } }]`, DashScope `{ code, message }`) and
|
|
69
|
+
decides in this order: a refused secret, a spent quota (some providers send it as 429 or 400), a rate limit, then the
|
|
70
|
+
status alone.
|
|
71
|
+
|
|
72
|
+
| class | effect on the key | retried |
|
|
73
|
+
|---|---|---|
|
|
74
|
+
| `rate_limit` | the lane cools down until `retry-after` (else `rateLimitCooldownMs`) | at once, another lane |
|
|
75
|
+
| `quota` | the key is blacklisted until its window ends or its secret changes | at once, another key |
|
|
76
|
+
| `auth` | the key is blacklisted until its secret changes (or `authDisableMs`) | at once, another key |
|
|
77
|
+
| `transient` | the key's and provider's breakers count it | after backoff, another lane |
|
|
78
|
+
| `invalid_request` | none | no: the caller's request is wrong |
|
|
79
|
+
|
|
80
|
+
| provider | rate_limit | quota (window) | auth | transient | invalid_request |
|
|
81
|
+
|---|---|---|---|---|---|
|
|
82
|
+
| Anthropic | 429 `rate_limit_error` | 402 `billing_error` (TTL); 429 with `details.error_code: enforced_spend_limit_reached` (next UTC month); 400 "You have reached your specified API usage limits" (TTL) | 401 `authentication_error`, 403 `permission_error` | 500 `api_error`, 504 `timeout_error`, 529 `overloaded_error`, SSE `event: error` | other 400, 404, 413 |
|
|
83
|
+
| OpenRouter | 429; 402 `metadata.limit_source: openrouter_in_flight_budget` | 402 `openrouter_credits`, `openrouter_key_limit` (TTL); 429 "free-models-per-day" (next 00:00 UTC) | 401 | 408, 502, 503, a 200 with `error` | 400, 403 (moderation) |
|
|
84
|
+
| Qwen (DashScope) | 429 `Throttling`, `Throttling.RateQuota`, `Throttling.AllocationQuota`, `Throttling.BurstRate` | 400 `Arrearage`, 403 `AllocationQuota.FreeTierOnly` (TTL) | 401 `InvalidApiKey` / `invalid_api_key`, 403 `AccessDenied`, `Model.AccessDenied` | 500 `InternalError` | 400 `InvalidParameter`, `DataInspectionFailed`, 404 `ModelNotFound` |
|
|
85
|
+
| DeepSeek | 429 | 402 "Insufficient Balance" (TTL) | 401 | 500, 503 | 400, 422 |
|
|
86
|
+
| OpenAI | 429 `rate_limit_exceeded`, `slow_down` | 429 `insufficient_quota`, `credit_balance_exhausted` (TTL); `organization_usage_limit_exceeded`, `*_spend_limit_exceeded` (next UTC month) | 401 `invalid_api_key`, 403 (region) | 500, 503 `server_is_overloaded` | 400, 404 |
|
|
87
|
+
| Gemini | 429 per-minute (`RESOURCE_EXHAUSTED`, `rate_limit_exceeded`, `too_many_requests`) | 429 `quota_exceeded` or a per-day limit (next midnight America/Los_Angeles); 402 `payment_required`; 400 `failed_precondition` naming billing | 401 `authentication`, "API key not valid", 403 `permission_denied`, "reported as leaked" | 500, 503, 504 | 400, 404 |
|
|
88
|
+
| Groq | 429 tokens per minute | 429 on requests per day (until `x-ratelimit-reset-requests`) | 401 | 498 (flex capacity), 500, 502, 503 | 400, 413, 422 |
|
|
89
|
+
| openai-compatible | 429 | 402 | 401, 403 | 5xx, connection refused | other 4xx |
|
|
90
|
+
|
|
91
|
+
Gemini words its per-minute 429 like a spent quota ("You exceeded your current quota"); only a per-day limit or the
|
|
92
|
+
`quota_exceeded` code is treated as one, else a short cooldown would become a day-long blacklist. The fake provider's
|
|
93
|
+
`PROVIDER_ERRORS` holds every row as the provider answers it, and `test/classify.test.ts` holds the table to it.
|
|
94
|
+
|
|
95
|
+
## Blacklist lifecycle
|
|
96
|
+
|
|
97
|
+
```
|
|
98
|
+
401/403, invalid key ─────────────▶ blacklisted (auth) ── until the secret changes, or authDisableMs
|
|
99
|
+
healthy ── 402, quota/credit/billing codes ──▶ blacklisted (quota) ── until its window ends, or the secret changes
|
|
100
|
+
no secret (unset env, resolver threw) ▶ blacklisted (missing) ── until a secret is set
|
|
101
|
+
429 ──────────────────────────────▶ cooling down ── until retry-after (short)
|
|
102
|
+
5xx, timeout, network ────────────▶ breaker counts ── closed / open / half-open probe
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
- **Where it lives.** The key's `HealthRecord` in the pool's `StateStore`: `disabledUntil`, `disabledReason`
|
|
106
|
+
(`auth`, `missing`, `quota`), `disabledCode` (the provider's code) and `fingerprint`. With `openSqliteStateStore`
|
|
107
|
+
(one host) or `redisStateStore` (several) a restart reads it back, so a dead key is not hammered again; the default
|
|
108
|
+
memory store forgets it. A record written by 0.4 reads with the new fields empty.
|
|
109
|
+
- **Until when.** A quota uses, in order: the window the answer names (next UTC month for Anthropic's spend cap and
|
|
110
|
+
OpenAI's monthly limits, next Pacific midnight for Gemini's daily quota, next UTC midnight for OpenRouter's free daily
|
|
111
|
+
cap), Groq's `x-ratelimit-reset-requests` when the request limit is at 0, else `resilience.quotaBlacklistMs` (default 6
|
|
112
|
+
hours); a later `retry-after` wins. A refused secret lasts until the secret changes (`authDisableMs: null`, the
|
|
113
|
+
default). Expiry is lazy: a past `disabledUntil` is simply usable.
|
|
114
|
+
- **A new secret lifts it.** The record keeps a 64-bit fingerprint (namespaced SHA-256, truncated) of the secret it
|
|
115
|
+
refused, never the secret. Each pick resolves the secret of a blacklisted key and compares: a rotated env variable,
|
|
116
|
+
a re-sealed `_FILE` or a resolver's new value brings the key back at once, with a `key-restored` event.
|
|
117
|
+
- **Observable.** `key-disabled` carries the alias, the reason, the provider's code and `until`; `key-restored` the
|
|
118
|
+
alias and what it was out for; `health()` shows `disabledReason` and `disabledUntil` per key. No surface carries a
|
|
119
|
+
secret.
|
|
120
|
+
- **Everything blacklisted.** When every lane is out because its key is blacklisted (or its provider is past its
|
|
121
|
+
budget) and none returns before the call's deadline, the call fails at once with class `quota` (any spent quota or
|
|
122
|
+
budget) or `auth`, naming the keys and the earliest return as `retryAt`. The gateway answers 503
|
|
123
|
+
`pool_quota_exhausted` or 503 `pool_auth_failed`, with `Retry-After` at that return, or 300 seconds when none is
|
|
124
|
+
known. A blacklist shorter than the deadline is waited out, like a cooldown.
|
|
125
|
+
|
|
126
|
+
## Selection
|
|
127
|
+
|
|
128
|
+
Unchanged in spirit from 0.4: a lane is out while blacklisted, breaker-open, cooling down, out of headroom or at its
|
|
129
|
+
concurrency cap (`key.maxConcurrency`, `provider.maxConcurrency`, per process). Tiers: untried lane, candidate tier,
|
|
130
|
+
untried key, untried provider, headroom; then fewest in flight, latency EWMA within tolerance (after three samples),
|
|
131
|
+
smooth weighted round-robin by `provider.weight * key.weight * model.weight * candidate.weight`. New: a provider may
|
|
132
|
+
carry a `budget` (`costPerDay`, `costPerMonth`, in the models' cost unit), spent from usage and kept in the store; past
|
|
133
|
+
it the provider is out until the next UTC day or month. Budget writes are read-modify-write, so two processes may
|
|
134
|
+
overshoot by one call each.
|
|
135
|
+
|
|
136
|
+
## Profiles
|
|
137
|
+
|
|
138
|
+
A profile is a named policy. `work` takes ordered tiers of `(provider, model)` candidates; a tier that is a list shares
|
|
139
|
+
its load by weight, and the router falls to the next tier when the tier has no usable lane:
|
|
140
|
+
|
|
141
|
+
```ts
|
|
142
|
+
work: {
|
|
143
|
+
candidates: [
|
|
144
|
+
[{ provider: "anthropic", model: "claude-sonnet-5-5", weight: 3 }, { provider: "openrouter", model: "anthropic/claude-sonnet-5-5" }],
|
|
145
|
+
{ provider: "deepseek", model: "deepseek-v4-pro" },
|
|
146
|
+
{ provider: "qwen", model: "qwen3.8-max" },
|
|
147
|
+
],
|
|
148
|
+
},
|
|
149
|
+
embed: { pinned: true, candidates: [[{ provider: "openai", model: "text-embedding-3-small" }, { provider: "openrouter", model: "openai/text-embedding-3-small" }]] },
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
`embed` is pinned: every candidate must resolve to models of one `identity` (`openai/text-embedding-3-small` at
|
|
153
|
+
OpenRouter declares `identity: "text-embedding-3-small"`) and one `dimensions`, or the pool is refused at startup; a call
|
|
154
|
+
cannot name another model; a vector of another length fails its lane and is never returned. Beyond pinned profiles,
|
|
155
|
+
every embeddings call stays on the identity of its first lane, and the gateway gives a direct embeddings model no
|
|
156
|
+
fallback: another model is another vector space, and an index holds one.
|
|
157
|
+
|
|
158
|
+
`model` and `fallbacks` keep working as in 0.4; a policy gives `candidates` or them, not both.
|
|
159
|
+
|
|
160
|
+
## Key scopes
|
|
161
|
+
|
|
162
|
+
A virtual key carries the profiles it may call (`work`, `embed`; more named profiles such as `fast`, `reason` or
|
|
163
|
+
`vision` later) and, optionally, an allowlist of exact catalog model ids, empty by default. A client sends
|
|
164
|
+
`model: "<profile>"`; the profile's candidates live only in the router config.
|
|
165
|
+
|
|
166
|
+
- A request naming neither one of the key's profiles nor a model on its allowlist is refused 403 `model_not_allowed`.
|
|
167
|
+
It is never remapped to a profile silently.
|
|
168
|
+
- An allowlisted model goes through the same pool, keys, blacklist and budgets as a profile, and the key's quotas and
|
|
169
|
+
cost caps count it the same: they are per virtual key, whatever the model.
|
|
170
|
+
- An embeddings request is served by the pinned `embed` profile's model only. A key cannot allowlist another embedding
|
|
171
|
+
model into `embed`; a second embeddings space is a separately named profile with its own index.
|
|
172
|
+
- `GET /v1/models` lists only what the calling key may use.
|
|
173
|
+
|
|
174
|
+
Why profiles and not a free model choice: a profile keeps cost, failover and model upgrades in one place. The model
|
|
175
|
+
behind `work` is swapped in the router config, without touching an app. A free model choice would bypass the budget
|
|
176
|
+
planning and the fallback chains, and pin apps to provider ids that change every few months. The allowlist covers the
|
|
177
|
+
rare caller that genuinely needs one specific model, by name and on purpose.
|
|
178
|
+
|
|
179
|
+
## The Anthropic adapter
|
|
180
|
+
|
|
181
|
+
`src/wire/anthropic.ts` translates OpenAI chat-completions to the Messages API and back. The router keeps every
|
|
182
|
+
other concern (lanes, retries, breakers, blacklist) protocol-free through the `Wire` interface (`src/wire/wire.ts`):
|
|
183
|
+
auth headers, request body, unary answer and stream events.
|
|
184
|
+
|
|
185
|
+
| OpenAI request | Anthropic request |
|
|
186
|
+
|---|---|
|
|
187
|
+
| `system` and `developer` messages | top-level `system`, joined by a blank line |
|
|
188
|
+
| `user` content string or parts; `image_url` (data URL or URL) | `text` blocks; `image` with a `base64` or `url` source; empty text left out |
|
|
189
|
+
| `assistant` content and `tool_calls` | `text` blocks then `tool_use` blocks (`arguments` parsed; not JSON is refused) |
|
|
190
|
+
| `tool` messages | `tool_result` blocks in a user turn; consecutive turns of one role merge |
|
|
191
|
+
| `max_completion_tokens` / `max_tokens` | `max_tokens` (required: else the provider's `defaultMaxTokens`, else 4096) |
|
|
192
|
+
| `stop` | `stop_sequences` |
|
|
193
|
+
| `tools` (`function`) | `{ name, description, input_schema, strict }`; other tool types pass as they are |
|
|
194
|
+
| `tool_choice` `auto` / `none` / `required` / `{ function }`, `parallel_tool_calls: false` | `auto` / `none` / `any` / `{ type: "tool", name }`, `disable_parallel_tool_use` |
|
|
195
|
+
| `response_format` `json_schema` | `output_config.format` `json_schema` (structured outputs) |
|
|
196
|
+
| `response_format` `json_object` | a system instruction (Anthropic has no JSON mode) |
|
|
197
|
+
| `user` | `metadata.user_id` |
|
|
198
|
+
| `temperature`, `top_p`, and `top_k`, `thinking`, `output_config`, `service_tier` | as they are |
|
|
199
|
+
| `n` above 1 | refused, `invalid_request` |
|
|
200
|
+
| `seed`, penalties, `logprobs`, `stream_options` | dropped |
|
|
201
|
+
|
|
202
|
+
Answers: text blocks join into `content`, `tool_use` blocks become `tool_calls`, thinking blocks are dropped;
|
|
203
|
+
`stop_reason` maps `end_turn`/`stop_sequence`/`pause_turn` to `stop`, `max_tokens`/`model_context_window_exceeded` to
|
|
204
|
+
`length`, `tool_use` to `tool_calls`, `refusal` to `content_filter`; usage counts cache reads and writes into
|
|
205
|
+
`prompt_tokens` and reports cache reads as `prompt_tokens_details.cached_tokens`.
|
|
206
|
+
|
|
207
|
+
Streams: `message_start` becomes the role chunk; `text_delta` content chunks; a `tool_use` block start a `tool_calls`
|
|
208
|
+
delta with id and name, and its `input_json_delta` pieces argument deltas at the tool call's own index; `message_delta`
|
|
209
|
+
the finish reason (its usage is cumulative); `message_stop` a final usage chunk. `ping`, `content_block_stop`, thinking
|
|
210
|
+
deltas and event types Anthropic adds later are skipped, as its docs ask. An `error` event is classified like an error
|
|
211
|
+
status; a stream that ends before `message_stop` is a failed attempt. Failover is before the first token only, as for
|
|
212
|
+
every provider.
|
|
213
|
+
|
|
214
|
+
Current models need care the adapter does not take for you: forced tool use (`tool_choice` `any`/`tool`) is refused on
|
|
215
|
+
Opus 5.5 and Sonnet 5.5, a `temperature` below 1 on 4.7 and later, and assistant prefill on 4.6 and later. These answer 400
|
|
216
|
+
and are not retried.
|
|
217
|
+
|
|
218
|
+
## Config from env
|
|
219
|
+
|
|
220
|
+
`poolFromEnv(spec, env)` (`src/env.ts`) is what an app calls instead of hand-rolling `<PROVIDER>_API_KEY_2...`:
|
|
221
|
+
|
|
222
|
+
| variable | meaning |
|
|
223
|
+
|---|---|
|
|
224
|
+
| `<PREFIX>_API_KEY`, `<PREFIX>_API_KEY_2`, ... `_<n>` | one key each, aliases `<id>-1`, `<id>-2`, ... (gaps allowed, no upper bound) |
|
|
225
|
+
| `<PREFIX>_API_KEY_FILE`, `<PREFIX>_API_KEY_<n>_FILE` | the same key as a mounted file (first line); both forms for one number is refused |
|
|
226
|
+
| `<PREFIX>_API_KEYS` | a list in one variable, split at commas and whitespace, aliases `<id>-list-<i>` |
|
|
227
|
+
| `<PREFIX>_BASE_URL` | moves the endpoint (a Qwen region or workspace host, an Ollama host) |
|
|
228
|
+
| `STARCI_AI_KEY`, `STARCI_AI_KEY_FILE` | the app's virtual key, `virtualKeyFromEnv` |
|
|
229
|
+
|
|
230
|
+
Prefixes: `ANTHROPIC`, `OPENROUTER`, `DASHSCOPE` (Qwen, the name Alibaba's own SDKs read), `DEEPSEEK`, `OPENAI`,
|
|
231
|
+
`GEMINI`, `GROQ`; an `openai-compatible` provider names its own (`OLLAMA`, `VLLM`, `TOGETHER`). Keys are references
|
|
232
|
+
read at each attempt; the helper never copies a value. A provider with no key in env is left out, and every profile is
|
|
233
|
+
cut down to the candidates the present providers serve, so one declaration deploys with whatever keys an environment
|
|
234
|
+
has; a profile left with nothing is refused at startup naming the variables that would serve it. The config file takes
|
|
235
|
+
`keysFromEnv: <PREFIX>` on a provider for the same discovery.
|
|
236
|
+
|
|
237
|
+
```ts
|
|
238
|
+
const pool = poolFromEnv({
|
|
239
|
+
providers: [
|
|
240
|
+
{ kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input: 3, output: 15 } }] },
|
|
241
|
+
{ kind: "openrouter", models: [{ id: "anthropic/claude-sonnet-5-5" }, { id: "openai/text-embedding-3-small", identity: "text-embedding-3-small", dimensions: 1536, capabilities: ["embeddings"] }] },
|
|
242
|
+
{ kind: "deepseek", models: [{ id: "deepseek-v4-pro" }] },
|
|
243
|
+
{ kind: "qwen", models: [{ id: "qwen3.8-max" }] },
|
|
244
|
+
{ kind: "openai", models: [{ id: "text-embedding-3-small", dimensions: 1536, capabilities: ["embeddings"] }] },
|
|
245
|
+
],
|
|
246
|
+
profiles: { work: { candidates: [...] }, embed: { pinned: true, candidates: [...] } },
|
|
247
|
+
store: await openSqliteStateStore("/data/ai-router-state.sqlite"),
|
|
248
|
+
})
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
## Constraints
|
|
252
|
+
|
|
253
|
+
- Breakers, blacklists, headroom and budgets are per process unless the `store` is shared; in-flight counts, latency
|
|
254
|
+
EWMAs and round-robin positions are always per process.
|
|
255
|
+
- One gateway host per app unless the store is Redis: the SQLite store is shared only by processes on one host.
|
|
256
|
+
- A stream fails over only before its first token; after it, the error reaches the caller with `afterFirstToken`.
|
|
257
|
+
- Store updates are read-modify-write: two processes failing at once may count one failure, and a budget may be
|
|
258
|
+
overshot by one call per process.
|
|
259
|
+
- A blacklisted key's secret is resolved at each pick to notice a new one: a resolver that is slow or rate-limited (a
|
|
260
|
+
vault) is called that often.
|
|
261
|
+
|
|
262
|
+
## Migration from 0.4
|
|
263
|
+
|
|
264
|
+
- `ProviderConfig.baseUrl` is optional for a kind with a default; `kind` defaults to `openai-compatible`, so a 0.4 pool
|
|
265
|
+
routes exactly as before.
|
|
266
|
+
- `ErrorClass` gains `quota`. A spent quota was a 429 `rate_limit` with a 10-second cooldown; it is now a blacklist.
|
|
267
|
+
Code that switches over `errorClass` adds the case.
|
|
268
|
+
- `KeyDisabledEvent.reason` gains `quota` and the event may carry `code`; `KeyHealth` gains `disabledUntil`;
|
|
269
|
+
`ProviderHealth` gains `kind` and `spend`; `CatalogProfile` gains `pinned` and `serves`; `RouterEvents` gains
|
|
270
|
+
`key-restored`.
|
|
271
|
+
- A key refused for its secret comes back when the secret changes, without `reset()`; a missing secret comes back when
|
|
272
|
+
it is set.
|
|
273
|
+
- When every key is blacklisted, the router fails with `quota` or `auth` rather than `no_candidate`, and the gateway
|
|
274
|
+
answers 503 (`pool_auth_failed` was 502) with `Retry-After`.
|
|
275
|
+
- Embeddings never fall back to another model identity, and the gateway gives a direct embeddings model no fallback.
|
|
276
|
+
- `HealthRecord` gains fields; a custom `StateStore` adapter stores them as it stores the rest (records it wrote before
|
|
277
|
+
read with the new fields empty).
|
|
278
|
+
- Apps replace hand-rolled key lists (`OPENROUTER_API_KEY` plus `_2..._9`) with `poolFromEnv`, keep `STARCI_AI_KEY` as
|
|
279
|
+
their one virtual key (`virtualKeyFromEnv`), pass no admin token, and pass a durable `store`.
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@starci/ai-router",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "A router for OpenAI-compatible calls across a
|
|
3
|
+
"version": "0.5.0",
|
|
4
|
+
"description": "A router for OpenAI-compatible calls across many providers (Anthropic through a protocol adapter, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible endpoint), each with an array of keys, behind one virtual key: an OpenAI-compatible gateway with virtual keys, quotas and usage, a NestJS module (AiRouterModule, AiRouterService, AiRouterGatewayModule), and createRouter for plain Node, with health-aware selection, a persisted key blacklist for spent quotas and refused secrets, circuit breakers, rate-limit awareness, failover across providers and models by profile, pinned embeddings, config from env and redacted telemetry.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai",
|
|
7
7
|
"llm",
|
|
@@ -18,7 +18,15 @@
|
|
|
18
18
|
"gateway",
|
|
19
19
|
"proxy",
|
|
20
20
|
"virtual-keys",
|
|
21
|
-
"langchain"
|
|
21
|
+
"langchain",
|
|
22
|
+
"anthropic",
|
|
23
|
+
"claude",
|
|
24
|
+
"openrouter",
|
|
25
|
+
"qwen",
|
|
26
|
+
"deepseek",
|
|
27
|
+
"gemini",
|
|
28
|
+
"quota",
|
|
29
|
+
"blacklist"
|
|
22
30
|
],
|
|
23
31
|
"license": "MIT",
|
|
24
32
|
"author": "starci183",
|
|
@@ -56,6 +64,7 @@
|
|
|
56
64
|
},
|
|
57
65
|
"files": [
|
|
58
66
|
"dist",
|
|
67
|
+
"docs",
|
|
59
68
|
"LICENSE",
|
|
60
69
|
"README.md",
|
|
61
70
|
"CHANGELOG.md"
|