@starci/ai-router 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/README.md +170 -30
- package/dist/ai-router.service.d.ts +5 -1
- package/dist/ai-router.service.js +8 -0
- package/dist/cache.d.ts +35 -0
- package/dist/cache.js +76 -0
- package/dist/config.js +31 -2
- package/dist/fake.d.ts +27 -0
- package/dist/fake.js +84 -15
- package/dist/gateway/config-file.js +9 -1
- package/dist/gateway/gateway.js +189 -17
- package/dist/langchain/index.d.ts +137 -0
- package/dist/langchain/index.js +182 -0
- package/dist/model-request.d.ts +7 -0
- package/dist/model-request.js +54 -0
- package/dist/pool.d.ts +5 -1
- package/dist/pool.js +17 -4
- package/dist/router.js +84 -21
- package/dist/testing.d.ts +1 -1
- package/dist/types.d.ts +130 -3
- package/dist/wire/anthropic.d.ts +18 -8
- package/dist/wire/anthropic.js +238 -58
- package/dist/wire/native.d.ts +29 -0
- package/dist/wire/native.js +56 -0
- package/dist/wire/thinking-memory.d.ts +32 -0
- package/dist/wire/thinking-memory.js +61 -0
- package/dist/wire/wire.d.ts +12 -7
- package/dist/wire/wire.js +10 -1
- package/docs/design.md +189 -19
- package/package.json +26 -6
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,38 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.7.0 - 2026-11-13
|
|
4
|
+
|
|
5
|
+
One virtual key, two protocols: the gateway speaks Anthropic Messages beside OpenAI, a LangChain app asks for the client
|
|
6
|
+
of a name, and prompt caching and thinking cross both. [docs/design.md](docs/design.md) has the sections "Prompt
|
|
7
|
+
caching", "Thinking with tools", "A model's own request shape", "The native Anthropic route", "Who served, and what it
|
|
8
|
+
cost" and "The LangChain entry" (Anthropic's docs read 2026-11-10; LangChain's packages read at the versions below).
|
|
9
|
+
|
|
10
|
+
- Added: `@starci/ai-router/langchain`. `createRoutedModels({ gatewayUrl, virtualKey, fetch?, timeoutMs?, headers? })` gives `chat(name)`, a discriminated union by protocol (`{ kind: "openai", model: ChatOpenAI }`, `{ kind: "anthropic", model: ChatAnthropic }`, with `name` and `fallbacks`), `embeddings(name)` (`{ kind: "openai", embeddings: OpenAIEmbeddings, dimensions }`) and `refresh()`; `usageOf(reply)` reads one usage shape for both kinds (`inputTokens`, `cacheReadTokens`, `cacheWriteTokens`, `outputTokens`, `reasoningTokens`, `model`, `provider`, `cost`); `RoutedChatKinds` and `registerChatKind` add a kind without touching the core; `RoutedModelsError`. The clients never retry (`maxRetries: 0`). `@langchain/openai` (^1.8.2), `@langchain/anthropic` (^1.6.2) and `@langchain/core` (^1.3.5) are optional peer dependencies, imported when a kind is first built.
|
|
11
|
+
- Added: the gateway's `POST /v1/messages`, the Anthropic Messages protocol under the same virtual key (`x-api-key` or a bearer token): the request reaches an Anthropic provider untranslated, the answer and stream events come back as sent, errors are in Anthropic's shape, `anthropic-beta` is forwarded, and a name no Anthropic provider serves is refused 400 `model_not_supported`. In process: `router.messages()` and `router.messagesStream()` (`AnthropicMessagesRequest`, `AnthropicMessageResult`, `AnthropicStream`, `AnthropicStreamEvent`), also on `AiRouterService`.
|
|
12
|
+
- Added: `GET /v1/models` lists `kind`, `kinds`, `serves` and `dimensions` per name; the catalog has `providers` (`CatalogProvider`), `CatalogModel.protocols` and `dimensions`, `CatalogProfile.protocols` and `dimensions`. A request header `x-ai-router-kind: openai | anthropic` keeps a chat call on that protocol's providers.
|
|
13
|
+
- Added: prompt-cache breakpoints in the OpenAI shape: `cache_control` (`{ type: "ephemeral", ttl? }`, `CacheControl`) on a content part, a message, a tool or the top level. The Anthropic adapter sends them as blocks, tool and top-level breakpoints (a marked system prompt becomes system blocks) and refuses more than four; `anthropic/*` on OpenRouter keeps part and top-level breakpoints; every other lane receives none. `ModelConfig.cacheControl` overrides per model.
|
|
14
|
+
- Added: usage reports the cache and thinking apart, in one shape for every provider: `prompt_tokens_details.cached_tokens`, `cache_write_tokens`, `cache_write_1h_tokens` and `completion_tokens_details.reasoning_tokens` (`PromptTokensDetails`, `CompletionTokensDetails`); `ModelCost` takes `cacheRead`, `cacheWrite` and `cacheWrite1h` and a call's cost prices each part; `LaneStats` counts `cachedTokens`, `cacheWriteTokens` and `reasoningTokens`.
|
|
15
|
+
- Added: a model's own request shape, applied per attempt on both routes: `ModelConfig.params` (default request fields), `omit` (fields never sent) and `forcedToolUse: false` (a forced `tool_choice` becomes `auto` plus an instruction, for models that refuse one).
|
|
16
|
+
- Added: `ProviderConfig.thinking` on an `anthropic` provider: `drop` (the default) never sends thinking blocks back; `replay` keeps each tool-calling turn in process and sends it back exactly as received while the conversation before it is unchanged. Thinking text, when the request asks to see it, is `reasoning_content`.
|
|
17
|
+
- Added: every answer names its lane inside its usage, `usage.served = { provider, model, attempts, cost? }`, on both routes, unary and streamed; response headers `x-ai-router-provider` and `x-ai-router-cost`; the gateway takes a client's well-formed `x-request-id` as the call's id.
|
|
18
|
+
- Added: the fake provider scripts `thinking`, `cache` and `reasoningTokens`, gives each answer its own id (`FakeCall.answerId`), and records where each request marked cache breakpoints (`FakeCall.cacheBreakpoints`); `FakeThinking`, `FakeCacheUsage`.
|
|
19
|
+
- Changed: the `max_tokens` an Anthropic provider is sent when nothing sets one is 16000 (was 4096): thinking counts toward it on current models. `defaultMaxTokens` or a model's `params` sets another.
|
|
20
|
+
- Changed: a provider stream that shows it is at work before its first token (Anthropic's pings and hidden thinking) restarts the attempt's idle timeout instead of being timed out at `timeoutMs`.
|
|
21
|
+
- Changed: Anthropic usage adds `cache_write_tokens` to `prompt_tokens_details` (and `cache_write_1h_tokens` when the answer splits writes); a refusal sets `message.refusal`.
|
|
22
|
+
- Fixed: the README's and the design's example price of `claude-sonnet-5-5` (2 and 10 per million tokens, cache read 0.1, writes 2.5 and 4).
|
|
23
|
+
|
|
24
|
+
### Migration from 0.6
|
|
25
|
+
|
|
26
|
+
- A request that carried `cache_control` to a provider other than Anthropic or `anthropic/*` on OpenRouter no longer sends it; a model whose provider takes breakpoints (Gemini on OpenRouter, Qwen) sets `cacheControl: true`.
|
|
27
|
+
- A pool with an Anthropic model whose output limit is below 16000 sets `defaultMaxTokens` (or the model's `params.max_tokens`).
|
|
28
|
+
- A custom `Wire` takes the lane's `ModelConfig` in `chat` (was the model id) and may return and receive a `context`; `WireEvent` adds `alive`.
|
|
29
|
+
- Code that builds a `RouterCatalog`, a `CatalogModel` or a `CatalogProfile` adds `providers` and `protocols`; code that implements `AiRouter` adds `messages` and `messagesStream`; a `LaneStats` literal adds the three token counters.
|
|
30
|
+
- Fake answers carry `msg_fake_<n>` and `fake-<n>` ids, no longer `msg_fake` and `fake-stream`.
|
|
31
|
+
|
|
32
|
+
## 0.6.1 - 2026-11-13
|
|
33
|
+
|
|
34
|
+
- Changed: the README's examples use neutral profile, label and owner names. No code changes.
|
|
35
|
+
|
|
3
36
|
## 0.6.0 - 2026-11-08
|
|
4
37
|
|
|
5
38
|
- Added: `AiRouterError`, `KeyAdminError` and `MountedKeyError` extend `DomainError` of `@starci/common` (a runtime dependency), so an app and its canon recognise them as coded errors. `AiRouterError.code` is `AI_ROUTER_<CLASS>` (`AI_ROUTER_ERROR_CODES`, `AiRouterErrorCode`, exported); the two key errors keep their code values. Class names, fields and messages are unchanged.
|
package/README.md
CHANGED
|
@@ -1,13 +1,15 @@
|
|
|
1
1
|
# @starci/ai-router
|
|
2
2
|
|
|
3
|
-
A router for
|
|
4
|
-
It load-balances below (Anthropic, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible
|
|
5
|
-
blacklists a key whose quota is spent or whose secret is refused, and exposes one
|
|
6
|
-
usable three ways over one engine:
|
|
7
|
-
|
|
8
|
-
- **as a gateway service** (`run.js serve`): an
|
|
9
|
-
included, `/v1/embeddings`, `/v1/models`)
|
|
10
|
-
app (LangChain, an OpenAI SDK, curl, a non-Node
|
|
3
|
+
A router for chat and embeddings calls across many providers, each with an array of keys, behind one virtual key per
|
|
4
|
+
app. It load-balances below (Anthropic, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible
|
|
5
|
+
endpoint), blacklists a key whose quota is spent or whose secret is refused, and exposes one gateway above that speaks
|
|
6
|
+
the OpenAI and the Anthropic protocols. It is usable three ways over one engine:
|
|
7
|
+
|
|
8
|
+
- **as a gateway service** (`run.js serve`): an HTTP endpoint in the OpenAI shape (`/v1/chat/completions`, streaming
|
|
9
|
+
included, `/v1/embeddings`, `/v1/models`) and in the Anthropic Messages shape (`/v1/messages`) with its own virtual
|
|
10
|
+
API keys, per-key quotas and usage accounting, so any app (LangChain, an OpenAI or Anthropic SDK, curl, a non-Node
|
|
11
|
+
service) uses the pool by pointing its base URL at it; `@starci/ai-router/langchain` hands a LangChain app the right
|
|
12
|
+
client for a name;
|
|
11
13
|
- **as a NestJS module**: register the pool once at the app root, inject `AiRouterService`, and call `chat`, `stream`
|
|
12
14
|
and `embeddings`; `AiRouterGatewayModule` hosts the gateway in the same process;
|
|
13
15
|
- **as a plain Node library**: `createRouter(config)` from `@starci/ai-router/gateway`, with no Nest at all.
|
|
@@ -16,11 +18,13 @@ The router picks a healthy key for each attempt, backs off what is failing, hono
|
|
|
16
18
|
quota or credit is spent until its window ends (or its secret changes), retries on a different key or provider, falls
|
|
17
19
|
back across providers and models by profile, keeps embeddings on one model, and tells you what it did without ever
|
|
18
20
|
exposing a key or a prompt. [docs/design.md](docs/design.md) is the design: the provider matrix, the error
|
|
19
|
-
classification per provider, the blacklist lifecycle, profiles and key scopes, the Anthropic adapter
|
|
21
|
+
classification per provider, the blacklist lifecycle, profiles and key scopes, the Anthropic adapter, prompt caching,
|
|
22
|
+
thinking with tools, the native Anthropic route, the LangChain entry and the env naming.
|
|
20
23
|
|
|
21
24
|
- ESM, typed, Node 20 or newer (the durable key store needs Node 22.13 or newer for `node:sqlite`). One runtime
|
|
22
25
|
dependency, `yaml`, for the config file. `@nestjs/common`, `@nestjs/core` (11 or 12) and `reflect-metadata` are
|
|
23
26
|
optional peer dependencies: the root entry needs them, `@starci/ai-router/gateway`, `@starci/ai-router/cli` and the `run.js` process entry do not.
|
|
27
|
+
`@langchain/openai`, `@langchain/anthropic` and `@langchain/core` are optional peers of `@starci/ai-router/langchain` alone.
|
|
24
28
|
- OpenAI chat-completions and embeddings shapes in and out. A provider's `kind` sets its protocol, auth header, default
|
|
25
29
|
base URL and error classification: `anthropic` (the Messages API, through a protocol adapter for chat, streaming,
|
|
26
30
|
tools and JSON), `openrouter`, `qwen` (DashScope compatible mode), `deepseek`, `openai`, `gemini` (its OpenAI
|
|
@@ -57,7 +61,7 @@ AiRouterModule.registerAsync({
|
|
|
57
61
|
useFactory: async () =>
|
|
58
62
|
poolFromEnv({
|
|
59
63
|
providers: [
|
|
60
|
-
{ kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input:
|
|
64
|
+
{ kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 } }] },
|
|
61
65
|
{ kind: "openrouter", models: [{ id: "anthropic/claude-sonnet-5-5" }, { id: "openai/text-embedding-3-small", identity: "text-embedding-3-small", dimensions: 1536, capabilities: ["embeddings"] }] },
|
|
62
66
|
{ kind: "deepseek", models: [{ id: "deepseek-v4-pro" }] },
|
|
63
67
|
{ kind: "qwen", models: [{ id: "qwen3.8-max" }] },
|
|
@@ -138,7 +142,7 @@ providers:
|
|
|
138
142
|
models: [{ id: openai/gpt-4o-mini, aliases: [fast] }]
|
|
139
143
|
defaults: { fallbacks: [smart], deadlineMs: 60000 }
|
|
140
144
|
profiles:
|
|
141
|
-
|
|
145
|
+
chat: { model: fast, fallbacks: [smart] }
|
|
142
146
|
grading: { model: smart, allowFallback: false, maxAttempts: 3 }
|
|
143
147
|
```
|
|
144
148
|
|
|
@@ -148,15 +152,20 @@ profiles:
|
|
|
148
152
|
|---|---|---|
|
|
149
153
|
| `POST /v1/chat/completions` | virtual key | a chat completion; `stream: true` answers OpenAI `chat.completion.chunk` events over SSE ending with `data: [DONE]` |
|
|
150
154
|
| `POST /v1/embeddings` | virtual key | an embeddings response |
|
|
151
|
-
| `
|
|
155
|
+
| `POST /v1/messages` | virtual key (as `x-api-key`, or a bearer token) | an Anthropic message, untranslated, from an Anthropic provider; `stream: true` answers Anthropic's own events |
|
|
156
|
+
| `GET /v1/models`, `GET /v1/models/{id}` | virtual key | the profiles and pool models (ids and aliases) this key may call, each with `kind` (the protocol of its first candidate), `kinds`, `serves` and, for embeddings, `dimensions` |
|
|
152
157
|
| `GET /health` | none; the admin token adds the pool's breakers and keys | `{ "status": "ok" }`, or `degraded` when no provider has a usable key |
|
|
153
158
|
| `/admin/keys...` | admin token | the [admin API](#admin-api) |
|
|
154
159
|
|
|
155
|
-
A request's `model` names a profile (`
|
|
160
|
+
A request's `model` names a profile (`chat`) or a pool model by id or alias (`gpt-4o-mini`, `fast`); a profile wins
|
|
156
161
|
when a name is both. A profile call routes by the profile's policy; a direct model call falls back only to the pool
|
|
157
|
-
defaults' fallbacks that the key may call itself. The answer is the provider's
|
|
158
|
-
|
|
159
|
-
|
|
162
|
+
defaults' fallbacks that the key may call itself. The answer is the provider's body without the router's `routing`
|
|
163
|
+
(which names a key); the lane that answered is inside its usage, `usage.served = { provider, model, attempts, cost? }`,
|
|
164
|
+
on every route, unary and streamed (the final usage chunk; the `message_start` event of an Anthropic stream). The
|
|
165
|
+
response carries `x-request-id` (the client's own when it sends a well-formed one no call in flight uses, so an app in
|
|
166
|
+
the router's process can match the call to the router's events; it must be unique per call), and a unary one
|
|
167
|
+
`x-ai-router-provider`, `x-ai-router-model`, `x-ai-router-attempts` and `x-ai-router-cost`. A request header
|
|
168
|
+
`x-ai-router-kind: openai` (or `anthropic`) keeps a chat call on the providers of that protocol. For a stream the gateway always asks the provider for usage (for accounting) and forwards the
|
|
160
169
|
usage chunk only when the client set `stream_options.include_usage`. Nothing is sent before the first chunk, so a pool
|
|
161
170
|
failure before the first token is an ordinary error response; after it, the stream ends with an OpenAI error event.
|
|
162
171
|
|
|
@@ -216,7 +225,7 @@ AiRouterGatewayModule.registerAsync({
|
|
|
216
225
|
keys: await openSqliteKeyStore("/data/keys.sqlite"),
|
|
217
226
|
mountedKeys: [
|
|
218
227
|
// The secret is sk-starci-<label>- and at least 32 letters, digits, "_" or "-": openssl rand -hex 24
|
|
219
|
-
{ label: "
|
|
228
|
+
{ label: "app_one", secret: () => readFileSync("/run/secrets/app_one_ai_key", "utf8").trim(), profiles: ["chat"], owner: "app-one", quota: { costPerMonth: 20 } },
|
|
220
229
|
],
|
|
221
230
|
listen: true,
|
|
222
231
|
}),
|
|
@@ -254,7 +263,7 @@ Guarded by `Authorization: Bearer <admin token>` (403 `admin_disabled` when no t
|
|
|
254
263
|
|
|
255
264
|
```sh
|
|
256
265
|
RUN="node node_modules/@starci/ai-router/dist/gateway/run.js"
|
|
257
|
-
$RUN keys create --label
|
|
266
|
+
$RUN keys create --label chat_app --profiles chat --rpm 60 --cost-per-month 20 --config pool.yaml
|
|
258
267
|
$RUN keys list --config pool.yaml
|
|
259
268
|
$RUN keys usage key_1a2b3c4d5e6f7a8b --from 2026-10-01 --config pool.yaml
|
|
260
269
|
$RUN keys rotate key_1a2b3c4d5e6f7a8b --url http://gateway:8080 # admin token from AI_ROUTER_ADMIN_TOKEN
|
|
@@ -273,7 +282,7 @@ LangChain (JS):
|
|
|
273
282
|
import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai"
|
|
274
283
|
|
|
275
284
|
const llm = new ChatOpenAI({
|
|
276
|
-
model: "
|
|
285
|
+
model: "chat", // a profile, or a pool model such as "gpt-4o-mini"
|
|
277
286
|
apiKey: process.env.STARCI_AI_KEY, // sk-starci-...
|
|
278
287
|
configuration: { baseURL: "http://127.0.0.1:3002/v1" },
|
|
279
288
|
})
|
|
@@ -283,7 +292,7 @@ for await (const chunk of await llm.stream("Count to five.")) process.stdout.wri
|
|
|
283
292
|
const embeddings = new OpenAIEmbeddings({ model: "text-embedding-3-small", apiKey: process.env.STARCI_AI_KEY, configuration: { baseURL: "http://127.0.0.1:3002/v1" } })
|
|
284
293
|
```
|
|
285
294
|
|
|
286
|
-
LangChain (Python): `ChatOpenAI(model="
|
|
295
|
+
LangChain (Python): `ChatOpenAI(model="chat", api_key=..., base_url="http://127.0.0.1:3002/v1")`.
|
|
287
296
|
|
|
288
297
|
The OpenAI SDK, or any compatible client:
|
|
289
298
|
|
|
@@ -291,12 +300,69 @@ The OpenAI SDK, or any compatible client:
|
|
|
291
300
|
import OpenAI from "openai"
|
|
292
301
|
|
|
293
302
|
const client = new OpenAI({ apiKey: process.env.STARCI_AI_KEY, baseURL: "http://127.0.0.1:3002/v1" })
|
|
294
|
-
const completion = await client.chat.completions.create({ model: "
|
|
303
|
+
const completion = await client.chat.completions.create({ model: "chat", messages: [{ role: "user", content: "hi" }] })
|
|
295
304
|
```
|
|
296
305
|
|
|
297
306
|
```sh
|
|
298
307
|
curl http://127.0.0.1:3002/v1/chat/completions -H "Authorization: Bearer $STARCI_AI_KEY" \
|
|
299
|
-
-H "content-type: application/json" -d '{"model":"
|
|
308
|
+
-H "content-type: application/json" -d '{"model":"chat","stream":true,"messages":[{"role":"user","content":"hi"}]}'
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
### LangChain clients by name
|
|
312
|
+
|
|
313
|
+
An app that uses LangChain asks `@starci/ai-router/langchain` for the client of a name and never builds a provider
|
|
314
|
+
client itself. `kind` is the wire protocol the gateway serves that name through (`openai` for every OpenAI-compatible
|
|
315
|
+
provider, `anthropic` for an Anthropic one); the common path uses `model` and never reads it.
|
|
316
|
+
|
|
317
|
+
```ts
|
|
318
|
+
import { createRoutedModels, usageOf } from "@starci/ai-router/langchain"
|
|
319
|
+
import { createAgent } from "langchain"
|
|
320
|
+
|
|
321
|
+
const ai = createRoutedModels({ gatewayUrl: "http://127.0.0.1:3002", virtualKey: process.env.STARCI_AI_KEY ?? "", fetch: platformFetch, timeoutMs: 120_000 })
|
|
322
|
+
|
|
323
|
+
const picked = await ai.chat("auto-heavy") // a pool model id, an alias or a profile
|
|
324
|
+
const agent = createAgent({ model: picked.model, tools }) // a ChatAnthropic or a ChatOpenAI: the agent does not care
|
|
325
|
+
const reply = await picked.model.invoke(messages)
|
|
326
|
+
usageOf(reply) // { inputTokens, cacheReadTokens, cacheWriteTokens, outputTokens, reasoningTokens, model, provider, cost }
|
|
327
|
+
|
|
328
|
+
if (picked.kind === "anthropic") {
|
|
329
|
+
// a provider's own feature, on its own client: automatic prompt caching as a call option
|
|
330
|
+
await picked.model.invoke(messages, { cache_control: { type: "ephemeral" } })
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
const next = picked.fallbacks[0] // the same name through its other protocol, for the app's own retry
|
|
334
|
+
const { embeddings, dimensions } = await ai.embeddings("embed")
|
|
335
|
+
```
|
|
336
|
+
|
|
337
|
+
- `ai.chat(name, { timeoutMs?, maxTokens?, fields? })` reads the gateway's `GET /v1/models` once (again when a name is
|
|
338
|
+
unknown, or on `ai.refresh()`) and builds the client: `maxRetries: 0` (the router fails over between keys and
|
|
339
|
+
providers; the app owns its tries), usage on streams, the injected `fetch`, the timeout. `fields.openai` and
|
|
340
|
+
`fields.anthropic` are extra constructor fields of that kind's client.
|
|
341
|
+
- A profile whose candidates span two protocols is the client of its first candidate's protocol; `fallbacks` holds the
|
|
342
|
+
other protocols' clients in the order the pool tries them, each kept on its own protocol's providers. Moving to one
|
|
343
|
+
is the app's retry.
|
|
344
|
+
- `usageOf(reply)` has one shape for both kinds, unary and streamed (the chunks added up): `inputTokens` is the prompt
|
|
345
|
+
billed at the full price, beside the cache reads and writes; `model` and `provider` are the pool's ids of the lane
|
|
346
|
+
that answered.
|
|
347
|
+
- `@langchain/openai`, `@langchain/anthropic` and `@langchain/core` are optional peer dependencies, imported when a kind
|
|
348
|
+
is first built: install the ones the pool's protocols need. A refusal is a `RoutedModelsError` with a code
|
|
349
|
+
(`ROUTED_NAME_UNKNOWN`, `ROUTED_CATALOG_UNAVAILABLE`, `ROUTED_KIND_UNSUPPORTED`, `ROUTED_NOT_CHAT`,
|
|
350
|
+
`ROUTED_NOT_EMBEDDINGS`, `ROUTED_PACKAGE_MISSING`).
|
|
351
|
+
- A new protocol is one interface member and one factory: `declare module "@starci/ai-router/langchain" { interface
|
|
352
|
+
RoutedChatKinds { gemini: ChatGoogle } }` and `registerChatKind("gemini", async (context) => new ChatGoogle(...))`.
|
|
353
|
+
|
|
354
|
+
### Call it from an Anthropic client
|
|
355
|
+
|
|
356
|
+
`POST /v1/messages` speaks the Anthropic Messages protocol under the same virtual key, sent as `x-api-key`. The request
|
|
357
|
+
reaches an Anthropic provider untranslated (cache breakpoints, thinking blocks, betas), the answer and the stream events
|
|
358
|
+
come back as the provider sent them, and errors are in Anthropic's shape. A name only an OpenAI-protocol provider serves
|
|
359
|
+
is refused 400 `model_not_supported`.
|
|
360
|
+
|
|
361
|
+
```ts
|
|
362
|
+
import Anthropic from "@anthropic-ai/sdk"
|
|
363
|
+
|
|
364
|
+
const client = new Anthropic({ apiKey: process.env.STARCI_AI_KEY, baseURL: "http://127.0.0.1:3002", maxRetries: 0 })
|
|
365
|
+
const message = await client.messages.create({ model: "sonnet", max_tokens: 16000, messages: [{ role: "user", content: "hi" }] })
|
|
300
366
|
```
|
|
301
367
|
|
|
302
368
|
### Deploy
|
|
@@ -356,7 +422,7 @@ fails with an `AiRouterError` saying the port is unknown and `listen.port` is th
|
|
|
356
422
|
(`127.0.0.1`) and is `0.0.0.0` only when you set it. With `serve`, `gateway.appPort` plus `gateway.portOffset`
|
|
357
423
|
(default 1) does the same; it cannot be combined with `gateway.port`.
|
|
358
424
|
|
|
359
|
-
A client of an API on 3001 uses `http://127.0.0.1:3002/v1` as its OpenAI base URL: LangChain (below)
|
|
425
|
+
A client of an API on 3001 uses `http://127.0.0.1:3002/v1` as its OpenAI base URL: LangChain (below) or any
|
|
360
426
|
OpenAI-compatible agent, with a virtual `sk-starci-...` key as its API key.
|
|
361
427
|
|
|
362
428
|
## Use it in a Nest app
|
|
@@ -515,10 +581,11 @@ A named registration provides no default instance; `@InjectAiRouterService()` wi
|
|
|
515
581
|
| `providers[].headers` | extra headers for every request (the auth header, `authorization` or `x-api-key`, is ignored here) | none |
|
|
516
582
|
| `providers[].weight` | multiplies the round-robin share of every lane of the provider | 1 |
|
|
517
583
|
| `providers[].maxConcurrency` | in-flight requests across the provider's keys, in this process | unbounded |
|
|
518
|
-
| `providers[].defaultMaxTokens` | the `max_tokens` sent to Anthropic, which requires one, when
|
|
584
|
+
| `providers[].defaultMaxTokens` | the `max_tokens` sent to Anthropic, which requires one, when neither the request nor the model's `params` sets one | 16000 |
|
|
585
|
+
| `providers[].thinking` | on an `anthropic` provider: `drop` never sends an answer's thinking blocks back; `replay` sends a tool-calling turn back as it was received | `drop` |
|
|
519
586
|
| `providers[].budget` | `{ costPerDay?, costPerMonth? }` in the models' cost unit; past it the provider is out until the next UTC day or month | none |
|
|
520
587
|
| `providers[].keys[]` | `{ alias, env }` or `{ alias, resolve }`, plus `weight`, `maxConcurrency` (`keysFromEnv` builds them) | at least one |
|
|
521
|
-
| `providers[].models[]` | `{ id, aliases?, contextWindow?, cost?, capabilities?, weight?, identity?, dimensions? }` | at least one |
|
|
588
|
+
| `providers[].models[]` | `{ id, aliases?, contextWindow?, cost?, capabilities?, weight?, identity?, dimensions?, params?, omit?, forcedToolUse?, cacheControl? }` | at least one |
|
|
522
589
|
| `defaults` | the policy every call starts from | none |
|
|
523
590
|
| `profiles` | named policies a call selects with `{ profile }` | none |
|
|
524
591
|
| `resilience.keyBreaker` | `{ failureThreshold, cooldownMs, maxCooldownMs }` per key | 3, 15000, 300000 |
|
|
@@ -536,12 +603,78 @@ A named registration provides no default instance; `@InjectAiRouterService()` wi
|
|
|
536
603
|
| `random` | the jitter source | `Math.random` |
|
|
537
604
|
|
|
538
605
|
A model's `id` is the name sent to the provider; a caller may name it by `id` or any alias, and an alias shared by
|
|
539
|
-
several providers makes them interchangeable for that call. `cost` is per one million tokens, `input` and `output
|
|
540
|
-
|
|
606
|
+
several providers makes them interchangeable for that call. `cost` is per one million tokens, `input` and `output`
|
|
607
|
+
plus the optional `cacheRead`, `cacheWrite` and `cacheWrite1h` (each `input` when left out), in whatever unit you
|
|
608
|
+
budget in. `params`, `omit`, `forcedToolUse` and `cacheControl` shape the request for that model
|
|
609
|
+
([One model's shape](#one-models-shape-never-the-apps)). `capabilities` defaults to `chat`, `streaming`, `tools` and `json`; an embeddings model
|
|
541
610
|
declares `["embeddings"]` (refused on a kind with no embeddings API: `anthropic`, `deepseek`, `groq`). `identity` names
|
|
542
611
|
one model served under different ids (`openai/text-embedding-3-small` at OpenRouter is `text-embedding-3-small`), and
|
|
543
612
|
`dimensions` the vector length an embeddings model answers: a response of another length fails the attempt.
|
|
544
613
|
|
|
614
|
+
## Prompt caching
|
|
615
|
+
|
|
616
|
+
Mark cache breakpoints once, in the OpenAI shape; each lane receives them as its provider takes them, and a provider
|
|
617
|
+
that caches a prefix on its own (DeepSeek, OpenAI, Gemini, most OpenAI-compatible endpoints) receives none:
|
|
618
|
+
|
|
619
|
+
```ts
|
|
620
|
+
await router.chat({
|
|
621
|
+
model: "auto-heavy",
|
|
622
|
+
messages: [
|
|
623
|
+
{ role: "system", content: [
|
|
624
|
+
{ type: "text", text: craft, cache_control: { type: "ephemeral", ttl: "1h" } }, // the tools render before it
|
|
625
|
+
{ type: "text", text: shopRules, cache_control: { type: "ephemeral" } },
|
|
626
|
+
{ type: "text", text: stepGoal, cache_control: { type: "ephemeral" } },
|
|
627
|
+
] },
|
|
628
|
+
...conversation,
|
|
629
|
+
],
|
|
630
|
+
tools,
|
|
631
|
+
cache_control: { type: "ephemeral" }, // automatic: one breakpoint on the last cacheable block
|
|
632
|
+
})
|
|
633
|
+
```
|
|
634
|
+
|
|
635
|
+
A breakpoint sits on a content part, on a message (its last part), on a tool, or at the top level. Anthropic takes at
|
|
636
|
+
most four, the top-level one included: a request with more is refused as `invalid_request` before it is sent. `ttl` is
|
|
637
|
+
`"5m"` (the default) or `"1h"`. An `anthropic` provider and `anthropic/*` models on `openrouter` take breakpoints;
|
|
638
|
+
`cacheControl: true` on a model says its provider does too. [docs/design.md](docs/design.md#prompt-caching) has the
|
|
639
|
+
table per provider.
|
|
640
|
+
|
|
641
|
+
Usage reports the cache in one shape for every provider, and `cost` prices each part:
|
|
642
|
+
|
|
643
|
+
```ts
|
|
644
|
+
reply.usage
|
|
645
|
+
// { prompt_tokens: 10000, completion_tokens: 500, total_tokens: 10500,
|
|
646
|
+
// prompt_tokens_details: { cached_tokens: 6000, cache_write_tokens: 3000, cache_write_1h_tokens: 1000 },
|
|
647
|
+
// completion_tokens_details: { reasoning_tokens: 200 } }
|
|
648
|
+
// with cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 } (Claude Sonnet 5.5's list prices,
|
|
649
|
+
// per million tokens, read 2026-11-10): 1000 at 2, 6000 at 0.1, 2000 at 2.5, 1000 at 4, 500 at 10.
|
|
650
|
+
```
|
|
651
|
+
|
|
652
|
+
`prompt_tokens` is the whole prompt; the cache reads and writes are parts of it. A cache price a model does not declare
|
|
653
|
+
is its `input` price.
|
|
654
|
+
|
|
655
|
+
## One model's shape, never the app's
|
|
656
|
+
|
|
657
|
+
A request that fails over reaches the next model in that model's own shape, from the model's config:
|
|
658
|
+
|
|
659
|
+
```ts
|
|
660
|
+
{
|
|
661
|
+
id: "claude-sonnet-5-5",
|
|
662
|
+
cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 },
|
|
663
|
+
params: { max_tokens: 32000, output_config: { effort: "medium" } }, // sent when the request sets none
|
|
664
|
+
omit: ["temperature", "top_p", "top_k"], // never sent: this model refuses non-default sampling
|
|
665
|
+
forcedToolUse: false, // a forced tool_choice becomes auto plus an instruction: this model refuses to be forced
|
|
666
|
+
}
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
With `forcedToolUse: false` a call is not guaranteed, so the caller checks that one was made and retries when none
|
|
670
|
+
was, as it would for any optional tool call.
|
|
671
|
+
|
|
672
|
+
On an `anthropic` provider, `thinking: "replay"` keeps the thinking blocks of an answer that calls tools in this
|
|
673
|
+
process and sends the turn back exactly as it was received on the next request of the loop, while the conversation
|
|
674
|
+
before it is unchanged; the default, `"drop"`, never sends them back, which the API accepts.
|
|
675
|
+
[docs/design.md](docs/design.md#thinking-with-tools) says why, and what a client that speaks the Anthropic protocol to
|
|
676
|
+
the gateway gets instead.
|
|
677
|
+
|
|
545
678
|
## Policies
|
|
546
679
|
|
|
547
680
|
A call's policy is `defaults`, then the selected profile, then `options.policy`, merged field by field (later wins).
|
|
@@ -634,7 +767,9 @@ reply; breaking out of the loop cancels the body and frees the key's slot.
|
|
|
634
767
|
`afterFirstToken: true`. The router does not retry it, since a second answer cannot be spliced onto the first; retry
|
|
635
768
|
the whole call if your use allows it.
|
|
636
769
|
- `timeoutMs` bounds the wait for the first token and then each gap between chunks; `deadlineMs` bounds the whole
|
|
637
|
-
stream.
|
|
770
|
+
stream. A provider that shows it is at work before its first token (Anthropic's pings and hidden thinking) restarts
|
|
771
|
+
that wait, so a model that thinks for a minute is not timed out; a unary call has no such sign, so give a thinking
|
|
772
|
+
model's profile a `timeoutMs` that covers its thinking, or stream.
|
|
638
773
|
- Stopping early counts as a success of the lane. Usage is read from a final usage chunk when the provider sends one
|
|
639
774
|
(OpenAI does when you pass `stream_options: { include_usage: true }`).
|
|
640
775
|
- Policy errors (an unknown model or profile) throw from `stream()` itself; everything else is thrown from iteration.
|
|
@@ -717,9 +852,14 @@ await app.close()
|
|
|
717
852
|
```
|
|
718
853
|
|
|
719
854
|
A reply may set `status`, `headers`, `latencyMs`, `error: "network"`, `text`, `toolCalls` (`{ id, name, arguments,
|
|
720
|
-
argumentChunks? }`), `
|
|
855
|
+
argumentChunks? }`), `thinking` (`{ text?, signature? }` blocks before the text), `usage`, `cache` (`{ read?, write?,
|
|
856
|
+
write1h? }`, parts of the prompt, reported in the protocol's own usage shape), `reasoningTokens`, `embedding`, a raw
|
|
857
|
+
`body`, and `stream: { chunks, delayMs, failAfter, failWith: "network" |
|
|
721
858
|
"error-event", stallAfter, stallMs, includeUsage }`. A request to `/messages` (an `anthropic` provider) is answered in
|
|
722
|
-
the Anthropic shape, unary and SSE, and its key is read from `x-api-key`.
|
|
859
|
+
the Anthropic shape, unary and SSE, and its key is read from `x-api-key`. Each recorded call lists where its request
|
|
860
|
+
marked cache breakpoints, `call.cacheBreakpoints` (`top`, `system.<i>`, `tools.<i>`, `messages.<i>`,
|
|
861
|
+
`messages.<i>.content.<j>`), the same for either protocol, so a spec asserts `["system.0", "system.1"]` reached Claude
|
|
862
|
+
and `[]` reached DeepSeek. `PROVIDER_ERRORS[kind][case]` holds each
|
|
723
863
|
provider's real error answers (`rate_limited`, `quota_exhausted`, `invalid_key`, `overloaded`, `bad_request`, and a
|
|
724
864
|
provider's second shapes such as `anthropic.spend_cap` or `openrouter.free_daily`), to script with
|
|
725
865
|
`fake.on({ key }, PROVIDER_ERRORS.anthropic.quota_exhausted)`. Everything is deterministic except the latencies you
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import type { OnApplicationShutdown } from "@nestjs/common";
|
|
2
|
-
import type { AiRouter, CallOptions, ChatCompletion, ChatCompletionRequest, ChatStream, EmbeddingsRequest, EmbeddingsResponse, HealthSnapshot, ResetTarget, RouterCatalog, RouterEvents, StatsSnapshot } from "./types.js";
|
|
2
|
+
import type { AiRouter, AnthropicMessageResult, AnthropicMessagesRequest, AnthropicStream, CallOptions, ChatCompletion, ChatCompletionRequest, ChatStream, EmbeddingsRequest, EmbeddingsResponse, HealthSnapshot, ResetTarget, RouterCatalog, RouterEvents, StatsSnapshot } from "./types.js";
|
|
3
3
|
import type { AiRouterHandlers } from "./ai-router.options.js";
|
|
4
4
|
/**
|
|
5
5
|
* The injectable facade over one router. Every call it makes also follows the application's shutdown, and every
|
|
@@ -16,6 +16,10 @@ export declare class AiRouterService<Profiles extends string = string> implement
|
|
|
16
16
|
stream(request: ChatCompletionRequest, options?: CallOptions<Profiles>): ChatStream;
|
|
17
17
|
/** Sends an embeddings request through the pool. */
|
|
18
18
|
embeddings(request: EmbeddingsRequest, options?: CallOptions<Profiles>): Promise<EmbeddingsResponse>;
|
|
19
|
+
/** Sends a request in the Anthropic Messages shape, untranslated, through the pool's Anthropic providers. */
|
|
20
|
+
messages(request: AnthropicMessagesRequest, options?: CallOptions<Profiles>): Promise<AnthropicMessageResult>;
|
|
21
|
+
/** Streams an Anthropic Messages request as the provider's own events; the stream ends `cancelled` when the application shuts down. */
|
|
22
|
+
messagesStream(request: AnthropicMessagesRequest, options?: CallOptions<Profiles>): AnthropicStream;
|
|
19
23
|
/** Reads every breaker, blacklisted key, headroom and budget from the store. */
|
|
20
24
|
health(): Promise<HealthSnapshot>;
|
|
21
25
|
/** Returns this process's counters. */
|
|
@@ -36,6 +36,14 @@ export class AiRouterService {
|
|
|
36
36
|
embeddings(request, options = {}) {
|
|
37
37
|
return this.router.embeddings(request, this.#options(options));
|
|
38
38
|
}
|
|
39
|
+
/** Sends a request in the Anthropic Messages shape, untranslated, through the pool's Anthropic providers. */
|
|
40
|
+
messages(request, options = {}) {
|
|
41
|
+
return this.router.messages(request, this.#options(options));
|
|
42
|
+
}
|
|
43
|
+
/** Streams an Anthropic Messages request as the provider's own events; the stream ends `cancelled` when the application shuts down. */
|
|
44
|
+
messagesStream(request, options = {}) {
|
|
45
|
+
return this.router.messagesStream(request, this.#options(options));
|
|
46
|
+
}
|
|
39
47
|
/** Reads every breaker, blacklisted key, headroom and budget from the store. */
|
|
40
48
|
health() {
|
|
41
49
|
return this.router.health();
|
package/dist/cache.d.ts
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
import type { ModelConfig, ProviderConfig, Usage } from "./types.js";
|
|
2
|
+
/**
|
|
3
|
+
* Prompt-cache breakpoints in the OpenAI shape (docs/design.md, "Prompt caching"). A request marks one with
|
|
4
|
+
* `cache_control: { type: "ephemeral", ttl? }` on a content part, on a message (it then sits on the message's last
|
|
5
|
+
* part), on a tool, or at the top level (automatic caching). A provider that caches a prefix on its own never sees a
|
|
6
|
+
* marker: it would be an unknown field there.
|
|
7
|
+
*/
|
|
8
|
+
/** The most breakpoints one request may carry, the top-level one included (Anthropic's limit, read 2026-11-10). */
|
|
9
|
+
export declare const MAX_CACHE_BREAKPOINTS = 4;
|
|
10
|
+
/** Whether a model's provider takes `cache_control` markers: the model's own `cacheControl`, else by provider kind. */
|
|
11
|
+
export declare function takesCacheControl(provider: ProviderConfig, model: ModelConfig): boolean;
|
|
12
|
+
/** A marker a request carries, or undefined: anything that is not an object with a `type` is no marker. */
|
|
13
|
+
export declare function markerOf(holder: Record<string, unknown>): Record<string, unknown> | undefined;
|
|
14
|
+
/** An OpenAI chat request without any `cache_control`: top level, messages, content parts and tools. */
|
|
15
|
+
export declare function withoutCacheControl(body: Record<string, unknown>): Record<string, unknown>;
|
|
16
|
+
/**
|
|
17
|
+
* An OpenAI chat request in the shape an OpenAI-compatible provider that takes breakpoints documents (OpenRouter,
|
|
18
|
+
* openrouter.ai/docs/guides/best-practices/prompt-caching, read 2026-11-10): markers on content parts and at the top
|
|
19
|
+
* level stay; a marker on a message moves to its last part (a string content becomes one text part); a marker on a
|
|
20
|
+
* tool is removed, since tools render before the system prompt and a system breakpoint covers them.
|
|
21
|
+
*/
|
|
22
|
+
export declare function withPartCacheControl(body: Record<string, unknown>): Record<string, unknown>;
|
|
23
|
+
/** The parts of a usage the cache and thinking account for; zero where the provider reported none. */
|
|
24
|
+
export interface TokenParts {
|
|
25
|
+
/** Prompt tokens read from the cache. */
|
|
26
|
+
readonly cached: number;
|
|
27
|
+
/** Prompt tokens written to the cache, the one-hour ones included. */
|
|
28
|
+
readonly cacheWrite: number;
|
|
29
|
+
/** Prompt tokens written to a one-hour cache. */
|
|
30
|
+
readonly cacheWrite1h: number;
|
|
31
|
+
/** Completion tokens spent thinking. */
|
|
32
|
+
readonly reasoning: number;
|
|
33
|
+
}
|
|
34
|
+
/** Reads the cache and thinking parts of a usage, bounded by the totals they are parts of. */
|
|
35
|
+
export declare function tokenParts(usage: Usage | undefined): TokenParts;
|
package/dist/cache.js
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
import { isPlainRecord } from "@starci/common";
|
|
2
|
+
import { presetOf } from "./providers.js";
|
|
3
|
+
/**
|
|
4
|
+
* Prompt-cache breakpoints in the OpenAI shape (docs/design.md, "Prompt caching"). A request marks one with
|
|
5
|
+
* `cache_control: { type: "ephemeral", ttl? }` on a content part, on a message (it then sits on the message's last
|
|
6
|
+
* part), on a tool, or at the top level (automatic caching). A provider that caches a prefix on its own never sees a
|
|
7
|
+
* marker: it would be an unknown field there.
|
|
8
|
+
*/
|
|
9
|
+
/** The most breakpoints one request may carry, the top-level one included (Anthropic's limit, read 2026-11-10). */
|
|
10
|
+
export const MAX_CACHE_BREAKPOINTS = 4;
|
|
11
|
+
const MARKER = "cache_control";
|
|
12
|
+
/** Whether a model's provider takes `cache_control` markers: the model's own `cacheControl`, else by provider kind. */
|
|
13
|
+
export function takesCacheControl(provider, model) {
|
|
14
|
+
if (model.cacheControl !== undefined)
|
|
15
|
+
return model.cacheControl;
|
|
16
|
+
const kind = presetOf(provider).kind;
|
|
17
|
+
return kind === "anthropic" || (kind === "openrouter" && model.id.startsWith("anthropic/"));
|
|
18
|
+
}
|
|
19
|
+
/** A marker a request carries, or undefined: anything that is not an object with a `type` is no marker. */
|
|
20
|
+
export function markerOf(holder) {
|
|
21
|
+
const marker = holder[MARKER];
|
|
22
|
+
return isPlainRecord(marker) && typeof marker.type === "string" ? marker : undefined;
|
|
23
|
+
}
|
|
24
|
+
function without(record) {
|
|
25
|
+
return MARKER in record ? Object.fromEntries(Object.entries(record).filter(([field]) => field !== MARKER)) : record;
|
|
26
|
+
}
|
|
27
|
+
function mapList(value, each) {
|
|
28
|
+
return Array.isArray(value) ? value.map((item) => (isPlainRecord(item) ? each(item) : item)) : value;
|
|
29
|
+
}
|
|
30
|
+
/** An OpenAI chat request without any `cache_control`: top level, messages, content parts and tools. */
|
|
31
|
+
export function withoutCacheControl(body) {
|
|
32
|
+
const out = without(body);
|
|
33
|
+
const messages = mapList(body.messages, (message) => ({ ...without(message), ...(Array.isArray(message.content) ? { content: mapList(message.content, without) } : {}) }));
|
|
34
|
+
const tools = mapList(body.tools, without);
|
|
35
|
+
return { ...out, ...(body.messages === undefined ? {} : { messages }), ...(body.tools === undefined ? {} : { tools }) };
|
|
36
|
+
}
|
|
37
|
+
/**
|
|
38
|
+
* An OpenAI chat request in the shape an OpenAI-compatible provider that takes breakpoints documents (OpenRouter,
|
|
39
|
+
* openrouter.ai/docs/guides/best-practices/prompt-caching, read 2026-11-10): markers on content parts and at the top
|
|
40
|
+
* level stay; a marker on a message moves to its last part (a string content becomes one text part); a marker on a
|
|
41
|
+
* tool is removed, since tools render before the system prompt and a system breakpoint covers them.
|
|
42
|
+
*/
|
|
43
|
+
export function withPartCacheControl(body) {
|
|
44
|
+
const messages = mapList(body.messages, (message) => {
|
|
45
|
+
const marker = markerOf(message);
|
|
46
|
+
if (!(MARKER in message))
|
|
47
|
+
return message;
|
|
48
|
+
const rest = without(message);
|
|
49
|
+
if (marker === undefined)
|
|
50
|
+
return rest;
|
|
51
|
+
const content = rest.content;
|
|
52
|
+
if (typeof content === "string")
|
|
53
|
+
return content === "" ? rest : { ...rest, content: [{ type: "text", text: content, [MARKER]: marker }] };
|
|
54
|
+
const parts = Array.isArray(content) ? content : [];
|
|
55
|
+
const last = parts.at(-1);
|
|
56
|
+
return isPlainRecord(last) ? { ...rest, content: [...parts.slice(0, -1), { ...last, [MARKER]: marker }] } : rest;
|
|
57
|
+
});
|
|
58
|
+
return { ...body, ...(body.messages === undefined ? {} : { messages }), ...(body.tools === undefined ? {} : { tools: mapList(body.tools, without) }) };
|
|
59
|
+
}
|
|
60
|
+
function count(value) {
|
|
61
|
+
return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : 0;
|
|
62
|
+
}
|
|
63
|
+
/** Reads the cache and thinking parts of a usage, bounded by the totals they are parts of. */
|
|
64
|
+
export function tokenParts(usage) {
|
|
65
|
+
const prompt = isPlainRecord(usage?.prompt_tokens_details) ? usage.prompt_tokens_details : {};
|
|
66
|
+
const completion = isPlainRecord(usage?.completion_tokens_details) ? usage.completion_tokens_details : {};
|
|
67
|
+
const total = count(usage?.prompt_tokens);
|
|
68
|
+
const cached = Math.min(count(prompt.cached_tokens), total);
|
|
69
|
+
const cacheWrite = Math.min(count(prompt.cache_write_tokens), total - cached);
|
|
70
|
+
return {
|
|
71
|
+
cached,
|
|
72
|
+
cacheWrite,
|
|
73
|
+
cacheWrite1h: Math.min(count(prompt.cache_write_1h_tokens), cacheWrite),
|
|
74
|
+
reasoning: Math.min(count(completion.reasoning_tokens), count(usage?.completion_tokens)),
|
|
75
|
+
};
|
|
76
|
+
}
|
package/dist/config.js
CHANGED
|
@@ -130,6 +130,8 @@ export function validateConfig(config) {
|
|
|
130
130
|
positive(provider.weight, `provider "${provider.id}" weight`);
|
|
131
131
|
positive(provider.maxConcurrency, `provider "${provider.id}" maxConcurrency`);
|
|
132
132
|
positive(provider.defaultMaxTokens, `provider "${provider.id}" defaultMaxTokens`);
|
|
133
|
+
if (provider.thinking !== undefined && preset.protocol !== "anthropic")
|
|
134
|
+
throw configError(`provider "${provider.id}" (${preset.kind}) sets thinking, which only an anthropic provider has`);
|
|
133
135
|
positive(provider.budget?.costPerDay, `provider "${provider.id}" budget.costPerDay`, true);
|
|
134
136
|
positive(provider.budget?.costPerMonth, `provider "${provider.id}" budget.costPerMonth`, true);
|
|
135
137
|
if (provider.keys.length === 0)
|
|
@@ -158,6 +160,13 @@ export function validateConfig(config) {
|
|
|
158
160
|
models.add(model.id);
|
|
159
161
|
positive(model.weight, `model "${model.id}" weight`);
|
|
160
162
|
positive(model.contextWindow, `model "${model.id}" contextWindow`);
|
|
163
|
+
for (const price of ["input", "output", "cacheRead", "cacheWrite", "cacheWrite1h"])
|
|
164
|
+
positive(model.cost?.[price], `model "${model.id}" cost.${price}`, true);
|
|
165
|
+
for (const field of model.omit ?? []) {
|
|
166
|
+
// Without its messages or its model a request is no request; every other field is the model's to refuse.
|
|
167
|
+
if (field === "" || field === "messages" || field === "model")
|
|
168
|
+
throw configError(`model "${model.id}" cannot omit "${field}"`);
|
|
169
|
+
}
|
|
161
170
|
if (model.dimensions !== undefined && (!Number.isInteger(model.dimensions) || model.dimensions < 1))
|
|
162
171
|
throw configError(`model "${model.id}" dimensions must be a positive integer`);
|
|
163
172
|
if (!preset.embeddings && (model.capabilities ?? []).includes("embeddings"))
|
|
@@ -310,12 +319,15 @@ export function buildCatalog(config) {
|
|
|
310
319
|
for (const model of provider.models) {
|
|
311
320
|
let entry = models.get(model.id);
|
|
312
321
|
if (entry === undefined) {
|
|
313
|
-
entry = { aliases: new Set(), providers: [], capabilities: new Set() };
|
|
322
|
+
entry = { aliases: new Set(), providers: [], protocols: new Set(), capabilities: new Set() };
|
|
314
323
|
models.set(model.id, entry);
|
|
315
324
|
}
|
|
316
325
|
for (const alias of model.aliases ?? [])
|
|
317
326
|
entry.aliases.add(alias);
|
|
318
327
|
entry.providers.push(provider.id);
|
|
328
|
+
entry.protocols.add(presetOf(provider).protocol);
|
|
329
|
+
if (entry.dimensions === undefined && model.dimensions !== undefined)
|
|
330
|
+
entry.dimensions = model.dimensions;
|
|
319
331
|
for (const capability of model.capabilities ?? DEFAULT_CAPABILITIES)
|
|
320
332
|
entry.capabilities.add(capability);
|
|
321
333
|
if (entry.contextWindow === undefined && model.contextWindow !== undefined)
|
|
@@ -331,19 +343,36 @@ export function buildCatalog(config) {
|
|
|
331
343
|
const capabilities = new Set(matchesOf(config, { model: name }).flatMap(({ model }) => model.capabilities ?? DEFAULT_CAPABILITIES));
|
|
332
344
|
return OPERATIONS.filter((operation) => capabilities.has(operation));
|
|
333
345
|
};
|
|
346
|
+
// The protocols a profile's candidates are served through, in the order the profile tries them.
|
|
347
|
+
const protocols = (policy, names) => {
|
|
348
|
+
const source = sourceOf(defaults, policy, {}, undefined);
|
|
349
|
+
const refs = "candidates" in source ? source.candidates.flat() : names.map((model) => ({ model }));
|
|
350
|
+
return [...new Set(refs.flatMap((ref) => matchesOf(config, ref).map(({ provider }) => presetOf(provider).protocol)))];
|
|
351
|
+
};
|
|
334
352
|
return {
|
|
335
353
|
defaults: namesOf({}, defaults),
|
|
354
|
+
providers: config.providers.map((provider) => ({ id: provider.id, kind: presetOf(provider).kind, protocol: presetOf(provider).protocol })),
|
|
336
355
|
profiles: Object.entries(config.profiles ?? {}).map(([name, policy]) => {
|
|
337
356
|
const named = namesOf(defaults, policy);
|
|
338
|
-
|
|
357
|
+
const dimensions = named.model === undefined ? undefined : matchesOf(config, { model: named.model })[0]?.model.dimensions;
|
|
358
|
+
return {
|
|
359
|
+
name,
|
|
360
|
+
...named,
|
|
361
|
+
pinned: policy.pinned === true,
|
|
362
|
+
serves: serves(named.model),
|
|
363
|
+
protocols: protocols(policy, [...(named.model === undefined ? [] : [named.model]), ...named.fallbacks]),
|
|
364
|
+
...(dimensions === undefined ? {} : { dimensions }),
|
|
365
|
+
};
|
|
339
366
|
}),
|
|
340
367
|
models: [...models].map(([id, entry]) => ({
|
|
341
368
|
id,
|
|
342
369
|
aliases: [...entry.aliases],
|
|
343
370
|
providers: entry.providers,
|
|
371
|
+
protocols: [...entry.protocols],
|
|
344
372
|
capabilities: [...entry.capabilities],
|
|
345
373
|
...(entry.contextWindow === undefined ? {} : { contextWindow: entry.contextWindow }),
|
|
346
374
|
...(entry.cost === undefined ? {} : { cost: entry.cost }),
|
|
375
|
+
...(entry.dimensions === undefined ? {} : { dimensions: entry.dimensions }),
|
|
347
376
|
})),
|
|
348
377
|
};
|
|
349
378
|
}
|