@starci/ai-router 0.6.1 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +43 -0
- package/README.md +176 -21
- package/dist/ai-router.service.d.ts +8 -1
- package/dist/ai-router.service.js +12 -0
- package/dist/cache.d.ts +35 -0
- package/dist/cache.js +76 -0
- package/dist/config.js +31 -2
- package/dist/fake.d.ts +27 -0
- package/dist/fake.js +84 -15
- package/dist/gateway/config-file.js +9 -1
- package/dist/gateway/gateway.js +189 -17
- package/dist/gateway/index.d.ts +2 -0
- package/dist/gateway/index.js +1 -0
- package/dist/known-models.d.ts +81 -0
- package/dist/known-models.js +142 -0
- package/dist/langchain/index.d.ts +137 -0
- package/dist/langchain/index.js +182 -0
- package/dist/model-request.d.ts +7 -0
- package/dist/model-request.js +54 -0
- package/dist/pool.d.ts +5 -1
- package/dist/pool.js +17 -4
- package/dist/router.js +86 -21
- package/dist/testing.d.ts +1 -1
- package/dist/types.d.ts +133 -3
- package/dist/wire/anthropic.d.ts +18 -8
- package/dist/wire/anthropic.js +238 -58
- package/dist/wire/native.d.ts +29 -0
- package/dist/wire/native.js +56 -0
- package/dist/wire/thinking-memory.d.ts +32 -0
- package/dist/wire/thinking-memory.js +61 -0
- package/dist/wire/wire.d.ts +12 -7
- package/dist/wire/wire.js +10 -1
- package/docs/design.md +189 -19
- package/package.json +26 -6
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,48 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.8.0 - 2026-11-16
|
|
4
|
+
|
|
5
|
+
The router knows the current Claude 5 and OpenAI models by id, offers OpenRouter's catalog at run time, lists what a caller may choose, and routes `auto-*` profiles by effort tier within the providers that have keys. Additive: nothing declared before changes.
|
|
6
|
+
|
|
7
|
+
- Added: `KNOWN_MODELS` and `knownModelConfigs("anthropic" | "openai")`: `claude-fable-5-2`, `claude-opus-5-5`, `claude-sonnet-5-5`, `claude-haiku-5-5`, `gpt-6-astra`, `gpt-6.1-sol`, `gpt-6.1-terra`, `gpt-6-luna`, each with chat, streaming, json, tools and vision and a 1M (Claude) or 1.05M (OpenAI) token window. Sources, read 2026-11-16: https://platform.claude.com/docs/en/models/overview (ids, window, image input, tool use, prices, cache-read share) and https://developers.openai.com/api/docs/models (ids, window, image input, function calling). Left out because the pages do not state it: Haiku 5.5's price (the page says "From $0.10 / $0.50"), every Claude cache-write price, and OpenAI prices. Opus 5.5 and Sonnet 5.5 carry `forcedToolUse: false`.
|
|
8
|
+
- Added: `fetchOpenRouterModels({ baseUrl?, apiKey?, fetch?, ttlMs?, now? })` reads OpenRouter's `GET /api/v1/models` (https://openrouter.ai/docs/api-reference/list-available-models), cached per base URL (one hour by default), into `ModelConfig`s: the id sent is `<vendor>/<model>`, the name a caller uses is `openrouter/<vendor>/<model>` (an alias); vision from `architecture.input_modalities`, tools from `supported_parameters`, the window from `context_length`, each left out when the entry omits it; prices are not read. `clearOpenRouterCatalogCache()`.
|
|
9
|
+
- Added: `defaultProviderSpecs({ openRouterModels? })` for `poolFromEnv`: Anthropic, OpenAI and OpenRouter, each present only when its key is in env (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`); without a key the provider and its models are absent and nothing fails at boot.
|
|
10
|
+
- Added: `availableModels()` on the router and `AiRouterService` (and `availableModels(config)`): `{ id, provider, label, vision, tools, contextTokens? }` per chat model per provider in the pool, so an app lists choices. `AvailableModel`.
|
|
11
|
+
- Added: `withAutoProfiles(config)` adds `auto-low`, `auto-medium`, `auto-high`, `auto-max` and `auto` (medium): each starts at its effort tier, then the tiers below, then those above, over the known models the pool serves, so it never reaches a provider without a key. `EFFORT_TIERS`, `EffortTier`. The tier of a model is this package's ranking from the vendors' descriptions and prices, not a vendor claim. A profile the app declares is kept.
|
|
12
|
+
|
|
13
|
+
### Migration from 0.7
|
|
14
|
+
|
|
15
|
+
- Code that implements `AiRouter` adds `availableModels`.
|
|
16
|
+
|
|
17
|
+
## 0.7.0 - 2026-11-13
|
|
18
|
+
|
|
19
|
+
One virtual key, two protocols: the gateway speaks Anthropic Messages beside OpenAI, a LangChain app asks for the client
|
|
20
|
+
of a name, and prompt caching and thinking cross both. [docs/design.md](docs/design.md) has the sections "Prompt
|
|
21
|
+
caching", "Thinking with tools", "A model's own request shape", "The native Anthropic route", "Who served, and what it
|
|
22
|
+
cost" and "The LangChain entry" (Anthropic's docs read 2026-11-10; LangChain's packages read at the versions below).
|
|
23
|
+
|
|
24
|
+
- Added: `@starci/ai-router/langchain`. `createRoutedModels({ gatewayUrl, virtualKey, fetch?, timeoutMs?, headers? })` gives `chat(name)`, a discriminated union by protocol (`{ kind: "openai", model: ChatOpenAI }`, `{ kind: "anthropic", model: ChatAnthropic }`, with `name` and `fallbacks`), `embeddings(name)` (`{ kind: "openai", embeddings: OpenAIEmbeddings, dimensions }`) and `refresh()`; `usageOf(reply)` reads one usage shape for both kinds (`inputTokens`, `cacheReadTokens`, `cacheWriteTokens`, `outputTokens`, `reasoningTokens`, `model`, `provider`, `cost`); `RoutedChatKinds` and `registerChatKind` add a kind without touching the core; `RoutedModelsError`. The clients never retry (`maxRetries: 0`). `@langchain/openai` (^1.8.2), `@langchain/anthropic` (^1.6.2) and `@langchain/core` (^1.3.5) are optional peer dependencies, imported when a kind is first built.
|
|
25
|
+
- Added: the gateway's `POST /v1/messages`, the Anthropic Messages protocol under the same virtual key (`x-api-key` or a bearer token): the request reaches an Anthropic provider untranslated, the answer and stream events come back as sent, errors are in Anthropic's shape, `anthropic-beta` is forwarded, and a name no Anthropic provider serves is refused 400 `model_not_supported`. In process: `router.messages()` and `router.messagesStream()` (`AnthropicMessagesRequest`, `AnthropicMessageResult`, `AnthropicStream`, `AnthropicStreamEvent`), also on `AiRouterService`.
|
|
26
|
+
- Added: `GET /v1/models` lists `kind`, `kinds`, `serves` and `dimensions` per name; the catalog has `providers` (`CatalogProvider`), `CatalogModel.protocols` and `dimensions`, `CatalogProfile.protocols` and `dimensions`. A request header `x-ai-router-kind: openai | anthropic` keeps a chat call on that protocol's providers.
|
|
27
|
+
- Added: prompt-cache breakpoints in the OpenAI shape: `cache_control` (`{ type: "ephemeral", ttl? }`, `CacheControl`) on a content part, a message, a tool or the top level. The Anthropic adapter sends them as blocks, tool and top-level breakpoints (a marked system prompt becomes system blocks) and refuses more than four; `anthropic/*` on OpenRouter keeps part and top-level breakpoints; every other lane receives none. `ModelConfig.cacheControl` overrides per model.
|
|
28
|
+
- Added: usage reports the cache and thinking apart, in one shape for every provider: `prompt_tokens_details.cached_tokens`, `cache_write_tokens`, `cache_write_1h_tokens` and `completion_tokens_details.reasoning_tokens` (`PromptTokensDetails`, `CompletionTokensDetails`); `ModelCost` takes `cacheRead`, `cacheWrite` and `cacheWrite1h` and a call's cost prices each part; `LaneStats` counts `cachedTokens`, `cacheWriteTokens` and `reasoningTokens`.
|
|
29
|
+
- Added: a model's own request shape, applied per attempt on both routes: `ModelConfig.params` (default request fields), `omit` (fields never sent) and `forcedToolUse: false` (a forced `tool_choice` becomes `auto` plus an instruction, for models that refuse one).
|
|
30
|
+
- Added: `ProviderConfig.thinking` on an `anthropic` provider: `drop` (the default) never sends thinking blocks back; `replay` keeps each tool-calling turn in process and sends it back exactly as received while the conversation before it is unchanged. Thinking text, when the request asks to see it, is `reasoning_content`.
|
|
31
|
+
- Added: every answer names its lane inside its usage, `usage.served = { provider, model, attempts, cost? }`, on both routes, unary and streamed; response headers `x-ai-router-provider` and `x-ai-router-cost`; the gateway takes a client's well-formed `x-request-id` as the call's id.
|
|
32
|
+
- Added: the fake provider scripts `thinking`, `cache` and `reasoningTokens`, gives each answer its own id (`FakeCall.answerId`), and records where each request marked cache breakpoints (`FakeCall.cacheBreakpoints`); `FakeThinking`, `FakeCacheUsage`.
|
|
33
|
+
- Changed: the `max_tokens` an Anthropic provider is sent when nothing sets one is 16000 (was 4096): thinking counts toward it on current models. `defaultMaxTokens` or a model's `params` sets another.
|
|
34
|
+
- Changed: a provider stream that shows it is at work before its first token (Anthropic's pings and hidden thinking) restarts the attempt's idle timeout instead of being timed out at `timeoutMs`.
|
|
35
|
+
- Changed: Anthropic usage adds `cache_write_tokens` to `prompt_tokens_details` (and `cache_write_1h_tokens` when the answer splits writes); a refusal sets `message.refusal`.
|
|
36
|
+
- Fixed: the README's and the design's example price of `claude-sonnet-5-5` (2 and 10 per million tokens, cache read 0.1, writes 2.5 and 4).
|
|
37
|
+
|
|
38
|
+
### Migration from 0.6
|
|
39
|
+
|
|
40
|
+
- A request that carried `cache_control` to a provider other than Anthropic or `anthropic/*` on OpenRouter no longer sends it; a model whose provider takes breakpoints (Gemini on OpenRouter, Qwen) sets `cacheControl: true`.
|
|
41
|
+
- A pool with an Anthropic model whose output limit is below 16000 sets `defaultMaxTokens` (or the model's `params.max_tokens`).
|
|
42
|
+
- A custom `Wire` takes the lane's `ModelConfig` in `chat` (was the model id) and may return and receive a `context`; `WireEvent` adds `alive`.
|
|
43
|
+
- Code that builds a `RouterCatalog`, a `CatalogModel` or a `CatalogProfile` adds `providers` and `protocols`; code that implements `AiRouter` adds `messages` and `messagesStream`; a `LaneStats` literal adds the three token counters.
|
|
44
|
+
- Fake answers carry `msg_fake_<n>` and `fake-<n>` ids, no longer `msg_fake` and `fake-stream`.
|
|
45
|
+
|
|
3
46
|
## 0.6.1 - 2026-11-13
|
|
4
47
|
|
|
5
48
|
- Changed: the README's examples use neutral profile, label and owner names. No code changes.
|
package/README.md
CHANGED
|
@@ -1,13 +1,15 @@
|
|
|
1
1
|
# @starci/ai-router
|
|
2
2
|
|
|
3
|
-
A router for
|
|
4
|
-
It load-balances below (Anthropic, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible
|
|
5
|
-
blacklists a key whose quota is spent or whose secret is refused, and exposes one
|
|
6
|
-
usable three ways over one engine:
|
|
7
|
-
|
|
8
|
-
- **as a gateway service** (`run.js serve`): an
|
|
9
|
-
included, `/v1/embeddings`, `/v1/models`)
|
|
10
|
-
app (LangChain, an OpenAI SDK, curl, a non-Node
|
|
3
|
+
A router for chat and embeddings calls across many providers, each with an array of keys, behind one virtual key per
|
|
4
|
+
app. It load-balances below (Anthropic, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible
|
|
5
|
+
endpoint), blacklists a key whose quota is spent or whose secret is refused, and exposes one gateway above that speaks
|
|
6
|
+
the OpenAI and the Anthropic protocols. It is usable three ways over one engine:
|
|
7
|
+
|
|
8
|
+
- **as a gateway service** (`run.js serve`): an HTTP endpoint in the OpenAI shape (`/v1/chat/completions`, streaming
|
|
9
|
+
included, `/v1/embeddings`, `/v1/models`) and in the Anthropic Messages shape (`/v1/messages`) with its own virtual
|
|
10
|
+
API keys, per-key quotas and usage accounting, so any app (LangChain, an OpenAI or Anthropic SDK, curl, a non-Node
|
|
11
|
+
service) uses the pool by pointing its base URL at it; `@starci/ai-router/langchain` hands a LangChain app the right
|
|
12
|
+
client for a name;
|
|
11
13
|
- **as a NestJS module**: register the pool once at the app root, inject `AiRouterService`, and call `chat`, `stream`
|
|
12
14
|
and `embeddings`; `AiRouterGatewayModule` hosts the gateway in the same process;
|
|
13
15
|
- **as a plain Node library**: `createRouter(config)` from `@starci/ai-router/gateway`, with no Nest at all.
|
|
@@ -16,11 +18,13 @@ The router picks a healthy key for each attempt, backs off what is failing, hono
|
|
|
16
18
|
quota or credit is spent until its window ends (or its secret changes), retries on a different key or provider, falls
|
|
17
19
|
back across providers and models by profile, keeps embeddings on one model, and tells you what it did without ever
|
|
18
20
|
exposing a key or a prompt. [docs/design.md](docs/design.md) is the design: the provider matrix, the error
|
|
19
|
-
classification per provider, the blacklist lifecycle, profiles and key scopes, the Anthropic adapter
|
|
21
|
+
classification per provider, the blacklist lifecycle, profiles and key scopes, the Anthropic adapter, prompt caching,
|
|
22
|
+
thinking with tools, the native Anthropic route, the LangChain entry and the env naming.
|
|
20
23
|
|
|
21
24
|
- ESM, typed, Node 20 or newer (the durable key store needs Node 22.13 or newer for `node:sqlite`). One runtime
|
|
22
25
|
dependency, `yaml`, for the config file. `@nestjs/common`, `@nestjs/core` (11 or 12) and `reflect-metadata` are
|
|
23
26
|
optional peer dependencies: the root entry needs them, `@starci/ai-router/gateway`, `@starci/ai-router/cli` and the `run.js` process entry do not.
|
|
27
|
+
`@langchain/openai`, `@langchain/anthropic` and `@langchain/core` are optional peers of `@starci/ai-router/langchain` alone.
|
|
24
28
|
- OpenAI chat-completions and embeddings shapes in and out. A provider's `kind` sets its protocol, auth header, default
|
|
25
29
|
base URL and error classification: `anthropic` (the Messages API, through a protocol adapter for chat, streaming,
|
|
26
30
|
tools and JSON), `openrouter`, `qwen` (DashScope compatible mode), `deepseek`, `openai`, `gemini` (its OpenAI
|
|
@@ -57,7 +61,7 @@ AiRouterModule.registerAsync({
|
|
|
57
61
|
useFactory: async () =>
|
|
58
62
|
poolFromEnv({
|
|
59
63
|
providers: [
|
|
60
|
-
{ kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input:
|
|
64
|
+
{ kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 } }] },
|
|
61
65
|
{ kind: "openrouter", models: [{ id: "anthropic/claude-sonnet-5-5" }, { id: "openai/text-embedding-3-small", identity: "text-embedding-3-small", dimensions: 1536, capabilities: ["embeddings"] }] },
|
|
62
66
|
{ kind: "deepseek", models: [{ id: "deepseek-v4-pro" }] },
|
|
63
67
|
{ kind: "qwen", models: [{ id: "qwen3.8-max" }] },
|
|
@@ -91,6 +95,21 @@ A virtual key carries the profiles it may call and, optionally, an allowlist of
|
|
|
91
95
|
default). A model it names that is neither is refused 403 `model_not_allowed`, never remapped; `/v1/models` lists only
|
|
92
96
|
what it may call; its quotas count every call whatever the model. See [key scopes](docs/design.md#key-scopes).
|
|
93
97
|
|
|
98
|
+
## Known models, OpenRouter and `auto`
|
|
99
|
+
|
|
100
|
+
The router knows the Claude 5 ids (`claude-fable-5-2`, `claude-opus-5-5`, `claude-sonnet-5-5`, `claude-haiku-5-5`) and the OpenAI GPT-6 ids (`gpt-6-astra`, `gpt-6.1-sol`, `gpt-6.1-terra`, `gpt-6-luna`) with the facts their vendors state (the sources and what is left out are in the 0.8.0 changelog entry). OpenRouter's catalog is read at run time. A provider is in the pool only when its key is in env, so a deployment with one key lists one provider's models and nothing fails at boot.
|
|
101
|
+
|
|
102
|
+
```ts
|
|
103
|
+
import { availableModels, createRouter, defaultProviderSpecs, fetchOpenRouterModels, poolFromEnv, withAutoProfiles } from "@starci/ai-router/gateway"
|
|
104
|
+
|
|
105
|
+
const openRouterModels = process.env.OPENROUTER_API_KEY ? await fetchOpenRouterModels({ apiKey: process.env.OPENROUTER_API_KEY }) : undefined
|
|
106
|
+
const router = createRouter(withAutoProfiles(poolFromEnv({ providers: defaultProviderSpecs({ openRouterModels }) })))
|
|
107
|
+
|
|
108
|
+
router.availableModels() // [{ id: "claude-opus-5-5", provider: "anthropic", label: "Claude Opus 5.5", vision: true, tools: true, contextTokens: 1000000 }, ...]
|
|
109
|
+
await router.chat({ messages }, { profile: "auto-high" }) // opus first, then the tiers below, then above; only providers with keys
|
|
110
|
+
await router.chat({ model: "openrouter/vendor/model", messages }) // sent to OpenRouter as vendor/model
|
|
111
|
+
```
|
|
112
|
+
|
|
94
113
|
## Run it as a gateway
|
|
95
114
|
|
|
96
115
|
One process (one container) holds the pool, the virtual keys and the admin API. The package declares no `bin` (the
|
|
@@ -148,15 +167,20 @@ profiles:
|
|
|
148
167
|
|---|---|---|
|
|
149
168
|
| `POST /v1/chat/completions` | virtual key | a chat completion; `stream: true` answers OpenAI `chat.completion.chunk` events over SSE ending with `data: [DONE]` |
|
|
150
169
|
| `POST /v1/embeddings` | virtual key | an embeddings response |
|
|
151
|
-
| `
|
|
170
|
+
| `POST /v1/messages` | virtual key (as `x-api-key`, or a bearer token) | an Anthropic message, untranslated, from an Anthropic provider; `stream: true` answers Anthropic's own events |
|
|
171
|
+
| `GET /v1/models`, `GET /v1/models/{id}` | virtual key | the profiles and pool models (ids and aliases) this key may call, each with `kind` (the protocol of its first candidate), `kinds`, `serves` and, for embeddings, `dimensions` |
|
|
152
172
|
| `GET /health` | none; the admin token adds the pool's breakers and keys | `{ "status": "ok" }`, or `degraded` when no provider has a usable key |
|
|
153
173
|
| `/admin/keys...` | admin token | the [admin API](#admin-api) |
|
|
154
174
|
|
|
155
175
|
A request's `model` names a profile (`chat`) or a pool model by id or alias (`gpt-4o-mini`, `fast`); a profile wins
|
|
156
176
|
when a name is both. A profile call routes by the profile's policy; a direct model call falls back only to the pool
|
|
157
|
-
defaults' fallbacks that the key may call itself. The answer is the provider's
|
|
158
|
-
|
|
159
|
-
|
|
177
|
+
defaults' fallbacks that the key may call itself. The answer is the provider's body without the router's `routing`
|
|
178
|
+
(which names a key); the lane that answered is inside its usage, `usage.served = { provider, model, attempts, cost? }`,
|
|
179
|
+
on every route, unary and streamed (the final usage chunk; the `message_start` event of an Anthropic stream). The
|
|
180
|
+
response carries `x-request-id` (the client's own when it sends a well-formed one no call in flight uses, so an app in
|
|
181
|
+
the router's process can match the call to the router's events; it must be unique per call), and a unary one
|
|
182
|
+
`x-ai-router-provider`, `x-ai-router-model`, `x-ai-router-attempts` and `x-ai-router-cost`. A request header
|
|
183
|
+
`x-ai-router-kind: openai` (or `anthropic`) keeps a chat call on the providers of that protocol. For a stream the gateway always asks the provider for usage (for accounting) and forwards the
|
|
160
184
|
usage chunk only when the client set `stream_options.include_usage`. Nothing is sent before the first chunk, so a pool
|
|
161
185
|
failure before the first token is an ordinary error response; after it, the stream ends with an OpenAI error event.
|
|
162
186
|
|
|
@@ -299,6 +323,63 @@ curl http://127.0.0.1:3002/v1/chat/completions -H "Authorization: Bearer $STARCI
|
|
|
299
323
|
-H "content-type: application/json" -d '{"model":"chat","stream":true,"messages":[{"role":"user","content":"hi"}]}'
|
|
300
324
|
```
|
|
301
325
|
|
|
326
|
+
### LangChain clients by name
|
|
327
|
+
|
|
328
|
+
An app that uses LangChain asks `@starci/ai-router/langchain` for the client of a name and never builds a provider
|
|
329
|
+
client itself. `kind` is the wire protocol the gateway serves that name through (`openai` for every OpenAI-compatible
|
|
330
|
+
provider, `anthropic` for an Anthropic one); the common path uses `model` and never reads it.
|
|
331
|
+
|
|
332
|
+
```ts
|
|
333
|
+
import { createRoutedModels, usageOf } from "@starci/ai-router/langchain"
|
|
334
|
+
import { createAgent } from "langchain"
|
|
335
|
+
|
|
336
|
+
const ai = createRoutedModels({ gatewayUrl: "http://127.0.0.1:3002", virtualKey: process.env.STARCI_AI_KEY ?? "", fetch: platformFetch, timeoutMs: 120_000 })
|
|
337
|
+
|
|
338
|
+
const picked = await ai.chat("auto-heavy") // a pool model id, an alias or a profile
|
|
339
|
+
const agent = createAgent({ model: picked.model, tools }) // a ChatAnthropic or a ChatOpenAI: the agent does not care
|
|
340
|
+
const reply = await picked.model.invoke(messages)
|
|
341
|
+
usageOf(reply) // { inputTokens, cacheReadTokens, cacheWriteTokens, outputTokens, reasoningTokens, model, provider, cost }
|
|
342
|
+
|
|
343
|
+
if (picked.kind === "anthropic") {
|
|
344
|
+
// a provider's own feature, on its own client: automatic prompt caching as a call option
|
|
345
|
+
await picked.model.invoke(messages, { cache_control: { type: "ephemeral" } })
|
|
346
|
+
}
|
|
347
|
+
|
|
348
|
+
const next = picked.fallbacks[0] // the same name through its other protocol, for the app's own retry
|
|
349
|
+
const { embeddings, dimensions } = await ai.embeddings("embed")
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
- `ai.chat(name, { timeoutMs?, maxTokens?, fields? })` reads the gateway's `GET /v1/models` once (again when a name is
|
|
353
|
+
unknown, or on `ai.refresh()`) and builds the client: `maxRetries: 0` (the router fails over between keys and
|
|
354
|
+
providers; the app owns its tries), usage on streams, the injected `fetch`, the timeout. `fields.openai` and
|
|
355
|
+
`fields.anthropic` are extra constructor fields of that kind's client.
|
|
356
|
+
- A profile whose candidates span two protocols is the client of its first candidate's protocol; `fallbacks` holds the
|
|
357
|
+
other protocols' clients in the order the pool tries them, each kept on its own protocol's providers. Moving to one
|
|
358
|
+
is the app's retry.
|
|
359
|
+
- `usageOf(reply)` has one shape for both kinds, unary and streamed (the chunks added up): `inputTokens` is the prompt
|
|
360
|
+
billed at the full price, beside the cache reads and writes; `model` and `provider` are the pool's ids of the lane
|
|
361
|
+
that answered.
|
|
362
|
+
- `@langchain/openai`, `@langchain/anthropic` and `@langchain/core` are optional peer dependencies, imported when a kind
|
|
363
|
+
is first built: install the ones the pool's protocols need. A refusal is a `RoutedModelsError` with a code
|
|
364
|
+
(`ROUTED_NAME_UNKNOWN`, `ROUTED_CATALOG_UNAVAILABLE`, `ROUTED_KIND_UNSUPPORTED`, `ROUTED_NOT_CHAT`,
|
|
365
|
+
`ROUTED_NOT_EMBEDDINGS`, `ROUTED_PACKAGE_MISSING`).
|
|
366
|
+
- A new protocol is one interface member and one factory: `declare module "@starci/ai-router/langchain" { interface
|
|
367
|
+
RoutedChatKinds { gemini: ChatGoogle } }` and `registerChatKind("gemini", async (context) => new ChatGoogle(...))`.
|
|
368
|
+
|
|
369
|
+
### Call it from an Anthropic client
|
|
370
|
+
|
|
371
|
+
`POST /v1/messages` speaks the Anthropic Messages protocol under the same virtual key, sent as `x-api-key`. The request
|
|
372
|
+
reaches an Anthropic provider untranslated (cache breakpoints, thinking blocks, betas), the answer and the stream events
|
|
373
|
+
come back as the provider sent them, and errors are in Anthropic's shape. A name only an OpenAI-protocol provider serves
|
|
374
|
+
is refused 400 `model_not_supported`.
|
|
375
|
+
|
|
376
|
+
```ts
|
|
377
|
+
import Anthropic from "@anthropic-ai/sdk"
|
|
378
|
+
|
|
379
|
+
const client = new Anthropic({ apiKey: process.env.STARCI_AI_KEY, baseURL: "http://127.0.0.1:3002", maxRetries: 0 })
|
|
380
|
+
const message = await client.messages.create({ model: "sonnet", max_tokens: 16000, messages: [{ role: "user", content: "hi" }] })
|
|
381
|
+
```
|
|
382
|
+
|
|
302
383
|
### Deploy
|
|
303
384
|
|
|
304
385
|
One container or process: `node node_modules/@starci/ai-router/dist/gateway/run.js serve --config
|
|
@@ -515,10 +596,11 @@ A named registration provides no default instance; `@InjectAiRouterService()` wi
|
|
|
515
596
|
| `providers[].headers` | extra headers for every request (the auth header, `authorization` or `x-api-key`, is ignored here) | none |
|
|
516
597
|
| `providers[].weight` | multiplies the round-robin share of every lane of the provider | 1 |
|
|
517
598
|
| `providers[].maxConcurrency` | in-flight requests across the provider's keys, in this process | unbounded |
|
|
518
|
-
| `providers[].defaultMaxTokens` | the `max_tokens` sent to Anthropic, which requires one, when
|
|
599
|
+
| `providers[].defaultMaxTokens` | the `max_tokens` sent to Anthropic, which requires one, when neither the request nor the model's `params` sets one | 16000 |
|
|
600
|
+
| `providers[].thinking` | on an `anthropic` provider: `drop` never sends an answer's thinking blocks back; `replay` sends a tool-calling turn back as it was received | `drop` |
|
|
519
601
|
| `providers[].budget` | `{ costPerDay?, costPerMonth? }` in the models' cost unit; past it the provider is out until the next UTC day or month | none |
|
|
520
602
|
| `providers[].keys[]` | `{ alias, env }` or `{ alias, resolve }`, plus `weight`, `maxConcurrency` (`keysFromEnv` builds them) | at least one |
|
|
521
|
-
| `providers[].models[]` | `{ id, aliases?, contextWindow?, cost?, capabilities?, weight?, identity?, dimensions? }` | at least one |
|
|
603
|
+
| `providers[].models[]` | `{ id, aliases?, contextWindow?, cost?, capabilities?, weight?, identity?, dimensions?, params?, omit?, forcedToolUse?, cacheControl? }` | at least one |
|
|
522
604
|
| `defaults` | the policy every call starts from | none |
|
|
523
605
|
| `profiles` | named policies a call selects with `{ profile }` | none |
|
|
524
606
|
| `resilience.keyBreaker` | `{ failureThreshold, cooldownMs, maxCooldownMs }` per key | 3, 15000, 300000 |
|
|
@@ -536,12 +618,78 @@ A named registration provides no default instance; `@InjectAiRouterService()` wi
|
|
|
536
618
|
| `random` | the jitter source | `Math.random` |
|
|
537
619
|
|
|
538
620
|
A model's `id` is the name sent to the provider; a caller may name it by `id` or any alias, and an alias shared by
|
|
539
|
-
several providers makes them interchangeable for that call. `cost` is per one million tokens, `input` and `output
|
|
540
|
-
|
|
621
|
+
several providers makes them interchangeable for that call. `cost` is per one million tokens, `input` and `output`
|
|
622
|
+
plus the optional `cacheRead`, `cacheWrite` and `cacheWrite1h` (each `input` when left out), in whatever unit you
|
|
623
|
+
budget in. `params`, `omit`, `forcedToolUse` and `cacheControl` shape the request for that model
|
|
624
|
+
([One model's shape](#one-models-shape-never-the-apps)). `capabilities` defaults to `chat`, `streaming`, `tools` and `json`; an embeddings model
|
|
541
625
|
declares `["embeddings"]` (refused on a kind with no embeddings API: `anthropic`, `deepseek`, `groq`). `identity` names
|
|
542
626
|
one model served under different ids (`openai/text-embedding-3-small` at OpenRouter is `text-embedding-3-small`), and
|
|
543
627
|
`dimensions` the vector length an embeddings model answers: a response of another length fails the attempt.
|
|
544
628
|
|
|
629
|
+
## Prompt caching
|
|
630
|
+
|
|
631
|
+
Mark cache breakpoints once, in the OpenAI shape; each lane receives them as its provider takes them, and a provider
|
|
632
|
+
that caches a prefix on its own (DeepSeek, OpenAI, Gemini, most OpenAI-compatible endpoints) receives none:
|
|
633
|
+
|
|
634
|
+
```ts
|
|
635
|
+
await router.chat({
|
|
636
|
+
model: "auto-heavy",
|
|
637
|
+
messages: [
|
|
638
|
+
{ role: "system", content: [
|
|
639
|
+
{ type: "text", text: craft, cache_control: { type: "ephemeral", ttl: "1h" } }, // the tools render before it
|
|
640
|
+
{ type: "text", text: shopRules, cache_control: { type: "ephemeral" } },
|
|
641
|
+
{ type: "text", text: stepGoal, cache_control: { type: "ephemeral" } },
|
|
642
|
+
] },
|
|
643
|
+
...conversation,
|
|
644
|
+
],
|
|
645
|
+
tools,
|
|
646
|
+
cache_control: { type: "ephemeral" }, // automatic: one breakpoint on the last cacheable block
|
|
647
|
+
})
|
|
648
|
+
```
|
|
649
|
+
|
|
650
|
+
A breakpoint sits on a content part, on a message (its last part), on a tool, or at the top level. Anthropic takes at
|
|
651
|
+
most four, the top-level one included: a request with more is refused as `invalid_request` before it is sent. `ttl` is
|
|
652
|
+
`"5m"` (the default) or `"1h"`. An `anthropic` provider and `anthropic/*` models on `openrouter` take breakpoints;
|
|
653
|
+
`cacheControl: true` on a model says its provider does too. [docs/design.md](docs/design.md#prompt-caching) has the
|
|
654
|
+
table per provider.
|
|
655
|
+
|
|
656
|
+
Usage reports the cache in one shape for every provider, and `cost` prices each part:
|
|
657
|
+
|
|
658
|
+
```ts
|
|
659
|
+
reply.usage
|
|
660
|
+
// { prompt_tokens: 10000, completion_tokens: 500, total_tokens: 10500,
|
|
661
|
+
// prompt_tokens_details: { cached_tokens: 6000, cache_write_tokens: 3000, cache_write_1h_tokens: 1000 },
|
|
662
|
+
// completion_tokens_details: { reasoning_tokens: 200 } }
|
|
663
|
+
// with cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 } (Claude Sonnet 5.5's list prices,
|
|
664
|
+
// per million tokens, read 2026-11-10): 1000 at 2, 6000 at 0.1, 2000 at 2.5, 1000 at 4, 500 at 10.
|
|
665
|
+
```
|
|
666
|
+
|
|
667
|
+
`prompt_tokens` is the whole prompt; the cache reads and writes are parts of it. A cache price a model does not declare
|
|
668
|
+
is its `input` price.
|
|
669
|
+
|
|
670
|
+
## One model's shape, never the app's
|
|
671
|
+
|
|
672
|
+
A request that fails over reaches the next model in that model's own shape, from the model's config:
|
|
673
|
+
|
|
674
|
+
```ts
|
|
675
|
+
{
|
|
676
|
+
id: "claude-sonnet-5-5",
|
|
677
|
+
cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 },
|
|
678
|
+
params: { max_tokens: 32000, output_config: { effort: "medium" } }, // sent when the request sets none
|
|
679
|
+
omit: ["temperature", "top_p", "top_k"], // never sent: this model refuses non-default sampling
|
|
680
|
+
forcedToolUse: false, // a forced tool_choice becomes auto plus an instruction: this model refuses to be forced
|
|
681
|
+
}
|
|
682
|
+
```
|
|
683
|
+
|
|
684
|
+
With `forcedToolUse: false` a call is not guaranteed, so the caller checks that one was made and retries when none
|
|
685
|
+
was, as it would for any optional tool call.
|
|
686
|
+
|
|
687
|
+
On an `anthropic` provider, `thinking: "replay"` keeps the thinking blocks of an answer that calls tools in this
|
|
688
|
+
process and sends the turn back exactly as it was received on the next request of the loop, while the conversation
|
|
689
|
+
before it is unchanged; the default, `"drop"`, never sends them back, which the API accepts.
|
|
690
|
+
[docs/design.md](docs/design.md#thinking-with-tools) says why, and what a client that speaks the Anthropic protocol to
|
|
691
|
+
the gateway gets instead.
|
|
692
|
+
|
|
545
693
|
## Policies
|
|
546
694
|
|
|
547
695
|
A call's policy is `defaults`, then the selected profile, then `options.policy`, merged field by field (later wins).
|
|
@@ -634,7 +782,9 @@ reply; breaking out of the loop cancels the body and frees the key's slot.
|
|
|
634
782
|
`afterFirstToken: true`. The router does not retry it, since a second answer cannot be spliced onto the first; retry
|
|
635
783
|
the whole call if your use allows it.
|
|
636
784
|
- `timeoutMs` bounds the wait for the first token and then each gap between chunks; `deadlineMs` bounds the whole
|
|
637
|
-
stream.
|
|
785
|
+
stream. A provider that shows it is at work before its first token (Anthropic's pings and hidden thinking) restarts
|
|
786
|
+
that wait, so a model that thinks for a minute is not timed out; a unary call has no such sign, so give a thinking
|
|
787
|
+
model's profile a `timeoutMs` that covers its thinking, or stream.
|
|
638
788
|
- Stopping early counts as a success of the lane. Usage is read from a final usage chunk when the provider sends one
|
|
639
789
|
(OpenAI does when you pass `stream_options: { include_usage: true }`).
|
|
640
790
|
- Policy errors (an unknown model or profile) throw from `stream()` itself; everything else is thrown from iteration.
|
|
@@ -717,9 +867,14 @@ await app.close()
|
|
|
717
867
|
```
|
|
718
868
|
|
|
719
869
|
A reply may set `status`, `headers`, `latencyMs`, `error: "network"`, `text`, `toolCalls` (`{ id, name, arguments,
|
|
720
|
-
argumentChunks? }`), `
|
|
870
|
+
argumentChunks? }`), `thinking` (`{ text?, signature? }` blocks before the text), `usage`, `cache` (`{ read?, write?,
|
|
871
|
+
write1h? }`, parts of the prompt, reported in the protocol's own usage shape), `reasoningTokens`, `embedding`, a raw
|
|
872
|
+
`body`, and `stream: { chunks, delayMs, failAfter, failWith: "network" |
|
|
721
873
|
"error-event", stallAfter, stallMs, includeUsage }`. A request to `/messages` (an `anthropic` provider) is answered in
|
|
722
|
-
the Anthropic shape, unary and SSE, and its key is read from `x-api-key`.
|
|
874
|
+
the Anthropic shape, unary and SSE, and its key is read from `x-api-key`. Each recorded call lists where its request
|
|
875
|
+
marked cache breakpoints, `call.cacheBreakpoints` (`top`, `system.<i>`, `tools.<i>`, `messages.<i>`,
|
|
876
|
+
`messages.<i>.content.<j>`), the same for either protocol, so a spec asserts `["system.0", "system.1"]` reached Claude
|
|
877
|
+
and `[]` reached DeepSeek. `PROVIDER_ERRORS[kind][case]` holds each
|
|
723
878
|
provider's real error answers (`rate_limited`, `quota_exhausted`, `invalid_key`, `overloaded`, `bad_request`, and a
|
|
724
879
|
provider's second shapes such as `anthropic.spend_cap` or `openrouter.free_daily`), to script with
|
|
725
880
|
`fake.on({ key }, PROVIDER_ERRORS.anthropic.quota_exhausted)`. Everything is deterministic except the latencies you
|
|
@@ -1,5 +1,6 @@
|
|
|
1
|
+
import type { AvailableModel } from "./known-models.js";
|
|
1
2
|
import type { OnApplicationShutdown } from "@nestjs/common";
|
|
2
|
-
import type { AiRouter, CallOptions, ChatCompletion, ChatCompletionRequest, ChatStream, EmbeddingsRequest, EmbeddingsResponse, HealthSnapshot, ResetTarget, RouterCatalog, RouterEvents, StatsSnapshot } from "./types.js";
|
|
3
|
+
import type { AiRouter, AnthropicMessageResult, AnthropicMessagesRequest, AnthropicStream, CallOptions, ChatCompletion, ChatCompletionRequest, ChatStream, EmbeddingsRequest, EmbeddingsResponse, HealthSnapshot, ResetTarget, RouterCatalog, RouterEvents, StatsSnapshot } from "./types.js";
|
|
3
4
|
import type { AiRouterHandlers } from "./ai-router.options.js";
|
|
4
5
|
/**
|
|
5
6
|
* The injectable facade over one router. Every call it makes also follows the application's shutdown, and every
|
|
@@ -16,6 +17,10 @@ export declare class AiRouterService<Profiles extends string = string> implement
|
|
|
16
17
|
stream(request: ChatCompletionRequest, options?: CallOptions<Profiles>): ChatStream;
|
|
17
18
|
/** Sends an embeddings request through the pool. */
|
|
18
19
|
embeddings(request: EmbeddingsRequest, options?: CallOptions<Profiles>): Promise<EmbeddingsResponse>;
|
|
20
|
+
/** Sends a request in the Anthropic Messages shape, untranslated, through the pool's Anthropic providers. */
|
|
21
|
+
messages(request: AnthropicMessagesRequest, options?: CallOptions<Profiles>): Promise<AnthropicMessageResult>;
|
|
22
|
+
/** Streams an Anthropic Messages request as the provider's own events; the stream ends `cancelled` when the application shuts down. */
|
|
23
|
+
messagesStream(request: AnthropicMessagesRequest, options?: CallOptions<Profiles>): AnthropicStream;
|
|
19
24
|
/** Reads every breaker, blacklisted key, headroom and budget from the store. */
|
|
20
25
|
health(): Promise<HealthSnapshot>;
|
|
21
26
|
/** Returns this process's counters. */
|
|
@@ -26,6 +31,8 @@ export declare class AiRouterService<Profiles extends string = string> implement
|
|
|
26
31
|
reset(target?: ResetTarget): Promise<void>;
|
|
27
32
|
/** Lists the pool's profiles and models by name. */
|
|
28
33
|
catalog(): RouterCatalog;
|
|
34
|
+
/** The chat models a caller may choose, from the providers whose keys are configured. */
|
|
35
|
+
availableModels(): AvailableModel[];
|
|
29
36
|
/** Aborts every call still in flight through this service and releases its handlers. */
|
|
30
37
|
onApplicationShutdown(): void;
|
|
31
38
|
}
|
|
@@ -36,6 +36,14 @@ export class AiRouterService {
|
|
|
36
36
|
embeddings(request, options = {}) {
|
|
37
37
|
return this.router.embeddings(request, this.#options(options));
|
|
38
38
|
}
|
|
39
|
+
/** Sends a request in the Anthropic Messages shape, untranslated, through the pool's Anthropic providers. */
|
|
40
|
+
messages(request, options = {}) {
|
|
41
|
+
return this.router.messages(request, this.#options(options));
|
|
42
|
+
}
|
|
43
|
+
/** Streams an Anthropic Messages request as the provider's own events; the stream ends `cancelled` when the application shuts down. */
|
|
44
|
+
messagesStream(request, options = {}) {
|
|
45
|
+
return this.router.messagesStream(request, this.#options(options));
|
|
46
|
+
}
|
|
39
47
|
/** Reads every breaker, blacklisted key, headroom and budget from the store. */
|
|
40
48
|
health() {
|
|
41
49
|
return this.router.health();
|
|
@@ -62,6 +70,10 @@ export class AiRouterService {
|
|
|
62
70
|
catalog() {
|
|
63
71
|
return this.router.catalog();
|
|
64
72
|
}
|
|
73
|
+
/** The chat models a caller may choose, from the providers whose keys are configured. */
|
|
74
|
+
availableModels() {
|
|
75
|
+
return this.router.availableModels();
|
|
76
|
+
}
|
|
65
77
|
/** Aborts every call still in flight through this service and releases its handlers. */
|
|
66
78
|
onApplicationShutdown() {
|
|
67
79
|
this.#shutdown.abort();
|
package/dist/cache.d.ts
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
import type { ModelConfig, ProviderConfig, Usage } from "./types.js";
|
|
2
|
+
/**
|
|
3
|
+
* Prompt-cache breakpoints in the OpenAI shape (docs/design.md, "Prompt caching"). A request marks one with
|
|
4
|
+
* `cache_control: { type: "ephemeral", ttl? }` on a content part, on a message (it then sits on the message's last
|
|
5
|
+
* part), on a tool, or at the top level (automatic caching). A provider that caches a prefix on its own never sees a
|
|
6
|
+
* marker: it would be an unknown field there.
|
|
7
|
+
*/
|
|
8
|
+
/** The most breakpoints one request may carry, the top-level one included (Anthropic's limit, read 2026-11-10). */
|
|
9
|
+
export declare const MAX_CACHE_BREAKPOINTS = 4;
|
|
10
|
+
/** Whether a model's provider takes `cache_control` markers: the model's own `cacheControl`, else by provider kind. */
|
|
11
|
+
export declare function takesCacheControl(provider: ProviderConfig, model: ModelConfig): boolean;
|
|
12
|
+
/** A marker a request carries, or undefined: anything that is not an object with a `type` is no marker. */
|
|
13
|
+
export declare function markerOf(holder: Record<string, unknown>): Record<string, unknown> | undefined;
|
|
14
|
+
/** An OpenAI chat request without any `cache_control`: top level, messages, content parts and tools. */
|
|
15
|
+
export declare function withoutCacheControl(body: Record<string, unknown>): Record<string, unknown>;
|
|
16
|
+
/**
|
|
17
|
+
* An OpenAI chat request in the shape an OpenAI-compatible provider that takes breakpoints documents (OpenRouter,
|
|
18
|
+
* openrouter.ai/docs/guides/best-practices/prompt-caching, read 2026-11-10): markers on content parts and at the top
|
|
19
|
+
* level stay; a marker on a message moves to its last part (a string content becomes one text part); a marker on a
|
|
20
|
+
* tool is removed, since tools render before the system prompt and a system breakpoint covers them.
|
|
21
|
+
*/
|
|
22
|
+
export declare function withPartCacheControl(body: Record<string, unknown>): Record<string, unknown>;
|
|
23
|
+
/** The parts of a usage the cache and thinking account for; zero where the provider reported none. */
|
|
24
|
+
export interface TokenParts {
|
|
25
|
+
/** Prompt tokens read from the cache. */
|
|
26
|
+
readonly cached: number;
|
|
27
|
+
/** Prompt tokens written to the cache, the one-hour ones included. */
|
|
28
|
+
readonly cacheWrite: number;
|
|
29
|
+
/** Prompt tokens written to a one-hour cache. */
|
|
30
|
+
readonly cacheWrite1h: number;
|
|
31
|
+
/** Completion tokens spent thinking. */
|
|
32
|
+
readonly reasoning: number;
|
|
33
|
+
}
|
|
34
|
+
/** Reads the cache and thinking parts of a usage, bounded by the totals they are parts of. */
|
|
35
|
+
export declare function tokenParts(usage: Usage | undefined): TokenParts;
|
package/dist/cache.js
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
import { isPlainRecord } from "@starci/common";
|
|
2
|
+
import { presetOf } from "./providers.js";
|
|
3
|
+
/**
|
|
4
|
+
* Prompt-cache breakpoints in the OpenAI shape (docs/design.md, "Prompt caching"). A request marks one with
|
|
5
|
+
* `cache_control: { type: "ephemeral", ttl? }` on a content part, on a message (it then sits on the message's last
|
|
6
|
+
* part), on a tool, or at the top level (automatic caching). A provider that caches a prefix on its own never sees a
|
|
7
|
+
* marker: it would be an unknown field there.
|
|
8
|
+
*/
|
|
9
|
+
/** The most breakpoints one request may carry, the top-level one included (Anthropic's limit, read 2026-11-10). */
|
|
10
|
+
export const MAX_CACHE_BREAKPOINTS = 4;
|
|
11
|
+
const MARKER = "cache_control";
|
|
12
|
+
/** Whether a model's provider takes `cache_control` markers: the model's own `cacheControl`, else by provider kind. */
|
|
13
|
+
export function takesCacheControl(provider, model) {
|
|
14
|
+
if (model.cacheControl !== undefined)
|
|
15
|
+
return model.cacheControl;
|
|
16
|
+
const kind = presetOf(provider).kind;
|
|
17
|
+
return kind === "anthropic" || (kind === "openrouter" && model.id.startsWith("anthropic/"));
|
|
18
|
+
}
|
|
19
|
+
/** A marker a request carries, or undefined: anything that is not an object with a `type` is no marker. */
|
|
20
|
+
export function markerOf(holder) {
|
|
21
|
+
const marker = holder[MARKER];
|
|
22
|
+
return isPlainRecord(marker) && typeof marker.type === "string" ? marker : undefined;
|
|
23
|
+
}
|
|
24
|
+
function without(record) {
|
|
25
|
+
return MARKER in record ? Object.fromEntries(Object.entries(record).filter(([field]) => field !== MARKER)) : record;
|
|
26
|
+
}
|
|
27
|
+
function mapList(value, each) {
|
|
28
|
+
return Array.isArray(value) ? value.map((item) => (isPlainRecord(item) ? each(item) : item)) : value;
|
|
29
|
+
}
|
|
30
|
+
/** An OpenAI chat request without any `cache_control`: top level, messages, content parts and tools. */
|
|
31
|
+
export function withoutCacheControl(body) {
|
|
32
|
+
const out = without(body);
|
|
33
|
+
const messages = mapList(body.messages, (message) => ({ ...without(message), ...(Array.isArray(message.content) ? { content: mapList(message.content, without) } : {}) }));
|
|
34
|
+
const tools = mapList(body.tools, without);
|
|
35
|
+
return { ...out, ...(body.messages === undefined ? {} : { messages }), ...(body.tools === undefined ? {} : { tools }) };
|
|
36
|
+
}
|
|
37
|
+
/**
|
|
38
|
+
* An OpenAI chat request in the shape an OpenAI-compatible provider that takes breakpoints documents (OpenRouter,
|
|
39
|
+
* openrouter.ai/docs/guides/best-practices/prompt-caching, read 2026-11-10): markers on content parts and at the top
|
|
40
|
+
* level stay; a marker on a message moves to its last part (a string content becomes one text part); a marker on a
|
|
41
|
+
* tool is removed, since tools render before the system prompt and a system breakpoint covers them.
|
|
42
|
+
*/
|
|
43
|
+
export function withPartCacheControl(body) {
|
|
44
|
+
const messages = mapList(body.messages, (message) => {
|
|
45
|
+
const marker = markerOf(message);
|
|
46
|
+
if (!(MARKER in message))
|
|
47
|
+
return message;
|
|
48
|
+
const rest = without(message);
|
|
49
|
+
if (marker === undefined)
|
|
50
|
+
return rest;
|
|
51
|
+
const content = rest.content;
|
|
52
|
+
if (typeof content === "string")
|
|
53
|
+
return content === "" ? rest : { ...rest, content: [{ type: "text", text: content, [MARKER]: marker }] };
|
|
54
|
+
const parts = Array.isArray(content) ? content : [];
|
|
55
|
+
const last = parts.at(-1);
|
|
56
|
+
return isPlainRecord(last) ? { ...rest, content: [...parts.slice(0, -1), { ...last, [MARKER]: marker }] } : rest;
|
|
57
|
+
});
|
|
58
|
+
return { ...body, ...(body.messages === undefined ? {} : { messages }), ...(body.tools === undefined ? {} : { tools: mapList(body.tools, without) }) };
|
|
59
|
+
}
|
|
60
|
+
function count(value) {
|
|
61
|
+
return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : 0;
|
|
62
|
+
}
|
|
63
|
+
/** Reads the cache and thinking parts of a usage, bounded by the totals they are parts of. */
|
|
64
|
+
export function tokenParts(usage) {
|
|
65
|
+
const prompt = isPlainRecord(usage?.prompt_tokens_details) ? usage.prompt_tokens_details : {};
|
|
66
|
+
const completion = isPlainRecord(usage?.completion_tokens_details) ? usage.completion_tokens_details : {};
|
|
67
|
+
const total = count(usage?.prompt_tokens);
|
|
68
|
+
const cached = Math.min(count(prompt.cached_tokens), total);
|
|
69
|
+
const cacheWrite = Math.min(count(prompt.cache_write_tokens), total - cached);
|
|
70
|
+
return {
|
|
71
|
+
cached,
|
|
72
|
+
cacheWrite,
|
|
73
|
+
cacheWrite1h: Math.min(count(prompt.cache_write_1h_tokens), cacheWrite),
|
|
74
|
+
reasoning: Math.min(count(completion.reasoning_tokens), count(usage?.completion_tokens)),
|
|
75
|
+
};
|
|
76
|
+
}
|
package/dist/config.js
CHANGED
|
@@ -130,6 +130,8 @@ export function validateConfig(config) {
|
|
|
130
130
|
positive(provider.weight, `provider "${provider.id}" weight`);
|
|
131
131
|
positive(provider.maxConcurrency, `provider "${provider.id}" maxConcurrency`);
|
|
132
132
|
positive(provider.defaultMaxTokens, `provider "${provider.id}" defaultMaxTokens`);
|
|
133
|
+
if (provider.thinking !== undefined && preset.protocol !== "anthropic")
|
|
134
|
+
throw configError(`provider "${provider.id}" (${preset.kind}) sets thinking, which only an anthropic provider has`);
|
|
133
135
|
positive(provider.budget?.costPerDay, `provider "${provider.id}" budget.costPerDay`, true);
|
|
134
136
|
positive(provider.budget?.costPerMonth, `provider "${provider.id}" budget.costPerMonth`, true);
|
|
135
137
|
if (provider.keys.length === 0)
|
|
@@ -158,6 +160,13 @@ export function validateConfig(config) {
|
|
|
158
160
|
models.add(model.id);
|
|
159
161
|
positive(model.weight, `model "${model.id}" weight`);
|
|
160
162
|
positive(model.contextWindow, `model "${model.id}" contextWindow`);
|
|
163
|
+
for (const price of ["input", "output", "cacheRead", "cacheWrite", "cacheWrite1h"])
|
|
164
|
+
positive(model.cost?.[price], `model "${model.id}" cost.${price}`, true);
|
|
165
|
+
for (const field of model.omit ?? []) {
|
|
166
|
+
// Without its messages or its model a request is no request; every other field is the model's to refuse.
|
|
167
|
+
if (field === "" || field === "messages" || field === "model")
|
|
168
|
+
throw configError(`model "${model.id}" cannot omit "${field}"`);
|
|
169
|
+
}
|
|
161
170
|
if (model.dimensions !== undefined && (!Number.isInteger(model.dimensions) || model.dimensions < 1))
|
|
162
171
|
throw configError(`model "${model.id}" dimensions must be a positive integer`);
|
|
163
172
|
if (!preset.embeddings && (model.capabilities ?? []).includes("embeddings"))
|
|
@@ -310,12 +319,15 @@ export function buildCatalog(config) {
|
|
|
310
319
|
for (const model of provider.models) {
|
|
311
320
|
let entry = models.get(model.id);
|
|
312
321
|
if (entry === undefined) {
|
|
313
|
-
entry = { aliases: new Set(), providers: [], capabilities: new Set() };
|
|
322
|
+
entry = { aliases: new Set(), providers: [], protocols: new Set(), capabilities: new Set() };
|
|
314
323
|
models.set(model.id, entry);
|
|
315
324
|
}
|
|
316
325
|
for (const alias of model.aliases ?? [])
|
|
317
326
|
entry.aliases.add(alias);
|
|
318
327
|
entry.providers.push(provider.id);
|
|
328
|
+
entry.protocols.add(presetOf(provider).protocol);
|
|
329
|
+
if (entry.dimensions === undefined && model.dimensions !== undefined)
|
|
330
|
+
entry.dimensions = model.dimensions;
|
|
319
331
|
for (const capability of model.capabilities ?? DEFAULT_CAPABILITIES)
|
|
320
332
|
entry.capabilities.add(capability);
|
|
321
333
|
if (entry.contextWindow === undefined && model.contextWindow !== undefined)
|
|
@@ -331,19 +343,36 @@ export function buildCatalog(config) {
|
|
|
331
343
|
const capabilities = new Set(matchesOf(config, { model: name }).flatMap(({ model }) => model.capabilities ?? DEFAULT_CAPABILITIES));
|
|
332
344
|
return OPERATIONS.filter((operation) => capabilities.has(operation));
|
|
333
345
|
};
|
|
346
|
+
// The protocols a profile's candidates are served through, in the order the profile tries them.
|
|
347
|
+
const protocols = (policy, names) => {
|
|
348
|
+
const source = sourceOf(defaults, policy, {}, undefined);
|
|
349
|
+
const refs = "candidates" in source ? source.candidates.flat() : names.map((model) => ({ model }));
|
|
350
|
+
return [...new Set(refs.flatMap((ref) => matchesOf(config, ref).map(({ provider }) => presetOf(provider).protocol)))];
|
|
351
|
+
};
|
|
334
352
|
return {
|
|
335
353
|
defaults: namesOf({}, defaults),
|
|
354
|
+
providers: config.providers.map((provider) => ({ id: provider.id, kind: presetOf(provider).kind, protocol: presetOf(provider).protocol })),
|
|
336
355
|
profiles: Object.entries(config.profiles ?? {}).map(([name, policy]) => {
|
|
337
356
|
const named = namesOf(defaults, policy);
|
|
338
|
-
|
|
357
|
+
const dimensions = named.model === undefined ? undefined : matchesOf(config, { model: named.model })[0]?.model.dimensions;
|
|
358
|
+
return {
|
|
359
|
+
name,
|
|
360
|
+
...named,
|
|
361
|
+
pinned: policy.pinned === true,
|
|
362
|
+
serves: serves(named.model),
|
|
363
|
+
protocols: protocols(policy, [...(named.model === undefined ? [] : [named.model]), ...named.fallbacks]),
|
|
364
|
+
...(dimensions === undefined ? {} : { dimensions }),
|
|
365
|
+
};
|
|
339
366
|
}),
|
|
340
367
|
models: [...models].map(([id, entry]) => ({
|
|
341
368
|
id,
|
|
342
369
|
aliases: [...entry.aliases],
|
|
343
370
|
providers: entry.providers,
|
|
371
|
+
protocols: [...entry.protocols],
|
|
344
372
|
capabilities: [...entry.capabilities],
|
|
345
373
|
...(entry.contextWindow === undefined ? {} : { contextWindow: entry.contextWindow }),
|
|
346
374
|
...(entry.cost === undefined ? {} : { cost: entry.cost }),
|
|
375
|
+
...(entry.dimensions === undefined ? {} : { dimensions: entry.dimensions }),
|
|
347
376
|
})),
|
|
348
377
|
};
|
|
349
378
|
}
|