@starci/ai-router 0.6.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,38 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.7.0 - 2026-11-13
4
+
5
+ One virtual key, two protocols: the gateway speaks Anthropic Messages beside OpenAI, a LangChain app asks for the client
6
+ of a name, and prompt caching and thinking cross both. [docs/design.md](docs/design.md) has the sections "Prompt
7
+ caching", "Thinking with tools", "A model's own request shape", "The native Anthropic route", "Who served, and what it
8
+ cost" and "The LangChain entry" (Anthropic's docs read 2026-11-10; LangChain's packages read at the versions below).
9
+
10
+ - Added: `@starci/ai-router/langchain`. `createRoutedModels({ gatewayUrl, virtualKey, fetch?, timeoutMs?, headers? })` gives `chat(name)`, a discriminated union by protocol (`{ kind: "openai", model: ChatOpenAI }`, `{ kind: "anthropic", model: ChatAnthropic }`, with `name` and `fallbacks`), `embeddings(name)` (`{ kind: "openai", embeddings: OpenAIEmbeddings, dimensions }`) and `refresh()`; `usageOf(reply)` reads one usage shape for both kinds (`inputTokens`, `cacheReadTokens`, `cacheWriteTokens`, `outputTokens`, `reasoningTokens`, `model`, `provider`, `cost`); `RoutedChatKinds` and `registerChatKind` add a kind without touching the core; `RoutedModelsError`. The clients never retry (`maxRetries: 0`). `@langchain/openai` (^1.8.2), `@langchain/anthropic` (^1.6.2) and `@langchain/core` (^1.3.5) are optional peer dependencies, imported when a kind is first built.
11
+ - Added: the gateway's `POST /v1/messages`, the Anthropic Messages protocol under the same virtual key (`x-api-key` or a bearer token): the request reaches an Anthropic provider untranslated, the answer and stream events come back as sent, errors are in Anthropic's shape, `anthropic-beta` is forwarded, and a name no Anthropic provider serves is refused 400 `model_not_supported`. In process: `router.messages()` and `router.messagesStream()` (`AnthropicMessagesRequest`, `AnthropicMessageResult`, `AnthropicStream`, `AnthropicStreamEvent`), also on `AiRouterService`.
12
+ - Added: `GET /v1/models` lists `kind`, `kinds`, `serves` and `dimensions` per name; the catalog has `providers` (`CatalogProvider`), `CatalogModel.protocols` and `dimensions`, `CatalogProfile.protocols` and `dimensions`. A request header `x-ai-router-kind: openai | anthropic` keeps a chat call on that protocol's providers.
13
+ - Added: prompt-cache breakpoints in the OpenAI shape: `cache_control` (`{ type: "ephemeral", ttl? }`, `CacheControl`) on a content part, a message, a tool or the top level. The Anthropic adapter sends them as blocks, tool and top-level breakpoints (a marked system prompt becomes system blocks) and refuses more than four; `anthropic/*` on OpenRouter keeps part and top-level breakpoints; every other lane receives none. `ModelConfig.cacheControl` overrides per model.
14
+ - Added: usage reports the cache and thinking apart, in one shape for every provider: `prompt_tokens_details.cached_tokens`, `cache_write_tokens`, `cache_write_1h_tokens` and `completion_tokens_details.reasoning_tokens` (`PromptTokensDetails`, `CompletionTokensDetails`); `ModelCost` takes `cacheRead`, `cacheWrite` and `cacheWrite1h` and a call's cost prices each part; `LaneStats` counts `cachedTokens`, `cacheWriteTokens` and `reasoningTokens`.
15
+ - Added: a model's own request shape, applied per attempt on both routes: `ModelConfig.params` (default request fields), `omit` (fields never sent) and `forcedToolUse: false` (a forced `tool_choice` becomes `auto` plus an instruction, for models that refuse one).
16
+ - Added: `ProviderConfig.thinking` on an `anthropic` provider: `drop` (the default) never sends thinking blocks back; `replay` keeps each tool-calling turn in process and sends it back exactly as received while the conversation before it is unchanged. Thinking text, when the request asks to see it, is `reasoning_content`.
17
+ - Added: every answer names its lane inside its usage, `usage.served = { provider, model, attempts, cost? }`, on both routes, unary and streamed; response headers `x-ai-router-provider` and `x-ai-router-cost`; the gateway takes a client's well-formed `x-request-id` as the call's id.
18
+ - Added: the fake provider scripts `thinking`, `cache` and `reasoningTokens`, gives each answer its own id (`FakeCall.answerId`), and records where each request marked cache breakpoints (`FakeCall.cacheBreakpoints`); `FakeThinking`, `FakeCacheUsage`.
19
+ - Changed: the `max_tokens` an Anthropic provider is sent when nothing sets one is 16000 (was 4096): thinking counts toward it on current models. `defaultMaxTokens` or a model's `params` sets another.
20
+ - Changed: a provider stream that shows it is at work before its first token (Anthropic's pings and hidden thinking) restarts the attempt's idle timeout instead of being timed out at `timeoutMs`.
21
+ - Changed: Anthropic usage adds `cache_write_tokens` to `prompt_tokens_details` (and `cache_write_1h_tokens` when the answer splits writes); a refusal sets `message.refusal`.
22
+ - Fixed: the README's and the design's example price of `claude-sonnet-5-5` (2 and 10 per million tokens, cache read 0.1, writes 2.5 and 4).
23
+
24
+ ### Migration from 0.6
25
+
26
+ - A request that carried `cache_control` to a provider other than Anthropic or `anthropic/*` on OpenRouter no longer sends it; a model whose provider takes breakpoints (Gemini on OpenRouter, Qwen) sets `cacheControl: true`.
27
+ - A pool with an Anthropic model whose output limit is below 16000 sets `defaultMaxTokens` (or the model's `params.max_tokens`).
28
+ - A custom `Wire` takes the lane's `ModelConfig` in `chat` (was the model id) and may return and receive a `context`; `WireEvent` adds `alive`.
29
+ - Code that builds a `RouterCatalog`, a `CatalogModel` or a `CatalogProfile` adds `providers` and `protocols`; code that implements `AiRouter` adds `messages` and `messagesStream`; a `LaneStats` literal adds the three token counters.
30
+ - Fake answers carry `msg_fake_<n>` and `fake-<n>` ids, no longer `msg_fake` and `fake-stream`.
31
+
32
+ ## 0.6.1 - 2026-11-13
33
+
34
+ - Changed: the README's examples use neutral profile, label and owner names. No code changes.
35
+
3
36
  ## 0.6.0 - 2026-11-08
4
37
 
5
38
  - Added: `AiRouterError`, `KeyAdminError` and `MountedKeyError` extend `DomainError` of `@starci/common` (a runtime dependency), so an app and its canon recognise them as coded errors. `AiRouterError.code` is `AI_ROUTER_<CLASS>` (`AI_ROUTER_ERROR_CODES`, `AiRouterErrorCode`, exported); the two key errors keep their code values. Class names, fields and messages are unchanged.
package/README.md CHANGED
@@ -1,13 +1,15 @@
1
1
  # @starci/ai-router
2
2
 
3
- A router for OpenAI-compatible calls across many providers, each with an array of keys, behind one virtual key per app.
4
- It load-balances below (Anthropic, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible endpoint),
5
- blacklists a key whose quota is spent or whose secret is refused, and exposes one OpenAI-compatible API above. It is
6
- usable three ways over one engine:
7
-
8
- - **as a gateway service** (`run.js serve`): an OpenAI-compatible HTTP endpoint (`/v1/chat/completions`, streaming
9
- included, `/v1/embeddings`, `/v1/models`) with its own virtual API keys, per-key quotas and usage accounting, so any
10
- app (LangChain, an OpenAI SDK, curl, a non-Node service) uses the pool by pointing its base URL at it;
3
+ A router for chat and embeddings calls across many providers, each with an array of keys, behind one virtual key per
4
+ app. It load-balances below (Anthropic, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible
5
+ endpoint), blacklists a key whose quota is spent or whose secret is refused, and exposes one gateway above that speaks
6
+ the OpenAI and the Anthropic protocols. It is usable three ways over one engine:
7
+
8
+ - **as a gateway service** (`run.js serve`): an HTTP endpoint in the OpenAI shape (`/v1/chat/completions`, streaming
9
+ included, `/v1/embeddings`, `/v1/models`) and in the Anthropic Messages shape (`/v1/messages`) with its own virtual
10
+ API keys, per-key quotas and usage accounting, so any app (LangChain, an OpenAI or Anthropic SDK, curl, a non-Node
11
+ service) uses the pool by pointing its base URL at it; `@starci/ai-router/langchain` hands a LangChain app the right
12
+ client for a name;
11
13
  - **as a NestJS module**: register the pool once at the app root, inject `AiRouterService`, and call `chat`, `stream`
12
14
  and `embeddings`; `AiRouterGatewayModule` hosts the gateway in the same process;
13
15
  - **as a plain Node library**: `createRouter(config)` from `@starci/ai-router/gateway`, with no Nest at all.
@@ -16,11 +18,13 @@ The router picks a healthy key for each attempt, backs off what is failing, hono
16
18
  quota or credit is spent until its window ends (or its secret changes), retries on a different key or provider, falls
17
19
  back across providers and models by profile, keeps embeddings on one model, and tells you what it did without ever
18
20
  exposing a key or a prompt. [docs/design.md](docs/design.md) is the design: the provider matrix, the error
19
- classification per provider, the blacklist lifecycle, profiles and key scopes, the Anthropic adapter and the env naming.
21
+ classification per provider, the blacklist lifecycle, profiles and key scopes, the Anthropic adapter, prompt caching,
22
+ thinking with tools, the native Anthropic route, the LangChain entry and the env naming.
20
23
 
21
24
  - ESM, typed, Node 20 or newer (the durable key store needs Node 22.13 or newer for `node:sqlite`). One runtime
22
25
  dependency, `yaml`, for the config file. `@nestjs/common`, `@nestjs/core` (11 or 12) and `reflect-metadata` are
23
26
  optional peer dependencies: the root entry needs them, `@starci/ai-router/gateway`, `@starci/ai-router/cli` and the `run.js` process entry do not.
27
+ `@langchain/openai`, `@langchain/anthropic` and `@langchain/core` are optional peers of `@starci/ai-router/langchain` alone.
24
28
  - OpenAI chat-completions and embeddings shapes in and out. A provider's `kind` sets its protocol, auth header, default
25
29
  base URL and error classification: `anthropic` (the Messages API, through a protocol adapter for chat, streaming,
26
30
  tools and JSON), `openrouter`, `qwen` (DashScope compatible mode), `deepseek`, `openai`, `gemini` (its OpenAI
@@ -57,7 +61,7 @@ AiRouterModule.registerAsync({
57
61
  useFactory: async () =>
58
62
  poolFromEnv({
59
63
  providers: [
60
- { kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input: 3, output: 15 } }] },
64
+ { kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 } }] },
61
65
  { kind: "openrouter", models: [{ id: "anthropic/claude-sonnet-5-5" }, { id: "openai/text-embedding-3-small", identity: "text-embedding-3-small", dimensions: 1536, capabilities: ["embeddings"] }] },
62
66
  { kind: "deepseek", models: [{ id: "deepseek-v4-pro" }] },
63
67
  { kind: "qwen", models: [{ id: "qwen3.8-max" }] },
@@ -138,7 +142,7 @@ providers:
138
142
  models: [{ id: openai/gpt-4o-mini, aliases: [fast] }]
139
143
  defaults: { fallbacks: [smart], deadlineMs: 60000 }
140
144
  profiles:
141
- lesson: { model: fast, fallbacks: [smart] }
145
+ chat: { model: fast, fallbacks: [smart] }
142
146
  grading: { model: smart, allowFallback: false, maxAttempts: 3 }
143
147
  ```
144
148
 
@@ -148,15 +152,20 @@ profiles:
148
152
  |---|---|---|
149
153
  | `POST /v1/chat/completions` | virtual key | a chat completion; `stream: true` answers OpenAI `chat.completion.chunk` events over SSE ending with `data: [DONE]` |
150
154
  | `POST /v1/embeddings` | virtual key | an embeddings response |
151
- | `GET /v1/models`, `GET /v1/models/{id}` | virtual key | the profiles and pool models (ids and aliases) this key may call |
155
+ | `POST /v1/messages` | virtual key (as `x-api-key`, or a bearer token) | an Anthropic message, untranslated, from an Anthropic provider; `stream: true` answers Anthropic's own events |
156
+ | `GET /v1/models`, `GET /v1/models/{id}` | virtual key | the profiles and pool models (ids and aliases) this key may call, each with `kind` (the protocol of its first candidate), `kinds`, `serves` and, for embeddings, `dimensions` |
152
157
  | `GET /health` | none; the admin token adds the pool's breakers and keys | `{ "status": "ok" }`, or `degraded` when no provider has a usable key |
153
158
  | `/admin/keys...` | admin token | the [admin API](#admin-api) |
154
159
 
155
- A request's `model` names a profile (`lesson`) or a pool model by id or alias (`gpt-4o-mini`, `fast`); a profile wins
160
+ A request's `model` names a profile (`chat`) or a pool model by id or alias (`gpt-4o-mini`, `fast`); a profile wins
156
161
  when a name is both. A profile call routes by the profile's policy; a direct model call falls back only to the pool
157
- defaults' fallbacks that the key may call itself. The answer is the provider's OpenAI body without the router's
158
- `routing`; the response carries `x-request-id`, and a unary one `x-ai-router-model` (the model that answered) and
159
- `x-ai-router-attempts`. For a stream the gateway always asks the provider for usage (for accounting) and forwards the
162
+ defaults' fallbacks that the key may call itself. The answer is the provider's body without the router's `routing`
163
+ (which names a key); the lane that answered is inside its usage, `usage.served = { provider, model, attempts, cost? }`,
164
+ on every route, unary and streamed (the final usage chunk; the `message_start` event of an Anthropic stream). The
165
+ response carries `x-request-id` (the client's own when it sends a well-formed one no call in flight uses, so an app in
166
+ the router's process can match the call to the router's events; it must be unique per call), and a unary one
167
+ `x-ai-router-provider`, `x-ai-router-model`, `x-ai-router-attempts` and `x-ai-router-cost`. A request header
168
+ `x-ai-router-kind: openai` (or `anthropic`) keeps a chat call on the providers of that protocol. For a stream the gateway always asks the provider for usage (for accounting) and forwards the
160
169
  usage chunk only when the client set `stream_options.include_usage`. Nothing is sent before the first chunk, so a pool
161
170
  failure before the first token is an ordinary error response; after it, the stream ends with an OpenAI error event.
162
171
 
@@ -216,7 +225,7 @@ AiRouterGatewayModule.registerAsync({
216
225
  keys: await openSqliteKeyStore("/data/keys.sqlite"),
217
226
  mountedKeys: [
218
227
  // The secret is sk-starci-<label>- and at least 32 letters, digits, "_" or "-": openssl rand -hex 24
219
- { label: "nivo_lite", secret: () => readFileSync("/run/secrets/nivo_lite_ai_key", "utf8").trim(), profiles: ["lesson"], owner: "nivo-lite", quota: { costPerMonth: 20 } },
228
+ { label: "app_one", secret: () => readFileSync("/run/secrets/app_one_ai_key", "utf8").trim(), profiles: ["chat"], owner: "app-one", quota: { costPerMonth: 20 } },
220
229
  ],
221
230
  listen: true,
222
231
  }),
@@ -254,7 +263,7 @@ Guarded by `Authorization: Bearer <admin token>` (403 `admin_disabled` when no t
254
263
 
255
264
  ```sh
256
265
  RUN="node node_modules/@starci/ai-router/dist/gateway/run.js"
257
- $RUN keys create --label lesson_app --profiles lesson --rpm 60 --cost-per-month 20 --config pool.yaml
266
+ $RUN keys create --label chat_app --profiles chat --rpm 60 --cost-per-month 20 --config pool.yaml
258
267
  $RUN keys list --config pool.yaml
259
268
  $RUN keys usage key_1a2b3c4d5e6f7a8b --from 2026-10-01 --config pool.yaml
260
269
  $RUN keys rotate key_1a2b3c4d5e6f7a8b --url http://gateway:8080 # admin token from AI_ROUTER_ADMIN_TOKEN
@@ -273,7 +282,7 @@ LangChain (JS):
273
282
  import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai"
274
283
 
275
284
  const llm = new ChatOpenAI({
276
- model: "lesson", // a profile, or a pool model such as "gpt-4o-mini"
285
+ model: "chat", // a profile, or a pool model such as "gpt-4o-mini"
277
286
  apiKey: process.env.STARCI_AI_KEY, // sk-starci-...
278
287
  configuration: { baseURL: "http://127.0.0.1:3002/v1" },
279
288
  })
@@ -283,7 +292,7 @@ for await (const chunk of await llm.stream("Count to five.")) process.stdout.wri
283
292
  const embeddings = new OpenAIEmbeddings({ model: "text-embedding-3-small", apiKey: process.env.STARCI_AI_KEY, configuration: { baseURL: "http://127.0.0.1:3002/v1" } })
284
293
  ```
285
294
 
286
- LangChain (Python): `ChatOpenAI(model="lesson", api_key=..., base_url="http://127.0.0.1:3002/v1")`.
295
+ LangChain (Python): `ChatOpenAI(model="chat", api_key=..., base_url="http://127.0.0.1:3002/v1")`.
287
296
 
288
297
  The OpenAI SDK, or any compatible client:
289
298
 
@@ -291,12 +300,69 @@ The OpenAI SDK, or any compatible client:
291
300
  import OpenAI from "openai"
292
301
 
293
302
  const client = new OpenAI({ apiKey: process.env.STARCI_AI_KEY, baseURL: "http://127.0.0.1:3002/v1" })
294
- const completion = await client.chat.completions.create({ model: "lesson", messages: [{ role: "user", content: "hi" }] })
303
+ const completion = await client.chat.completions.create({ model: "chat", messages: [{ role: "user", content: "hi" }] })
295
304
  ```
296
305
 
297
306
  ```sh
298
307
  curl http://127.0.0.1:3002/v1/chat/completions -H "Authorization: Bearer $STARCI_AI_KEY" \
299
- -H "content-type: application/json" -d '{"model":"lesson","stream":true,"messages":[{"role":"user","content":"hi"}]}'
308
+ -H "content-type: application/json" -d '{"model":"chat","stream":true,"messages":[{"role":"user","content":"hi"}]}'
309
+ ```
310
+
311
+ ### LangChain clients by name
312
+
313
+ An app that uses LangChain asks `@starci/ai-router/langchain` for the client of a name and never builds a provider
314
+ client itself. `kind` is the wire protocol the gateway serves that name through (`openai` for every OpenAI-compatible
315
+ provider, `anthropic` for an Anthropic one); the common path uses `model` and never reads it.
316
+
317
+ ```ts
318
+ import { createRoutedModels, usageOf } from "@starci/ai-router/langchain"
319
+ import { createAgent } from "langchain"
320
+
321
+ const ai = createRoutedModels({ gatewayUrl: "http://127.0.0.1:3002", virtualKey: process.env.STARCI_AI_KEY ?? "", fetch: platformFetch, timeoutMs: 120_000 })
322
+
323
+ const picked = await ai.chat("auto-heavy") // a pool model id, an alias or a profile
324
+ const agent = createAgent({ model: picked.model, tools }) // a ChatAnthropic or a ChatOpenAI: the agent does not care
325
+ const reply = await picked.model.invoke(messages)
326
+ usageOf(reply) // { inputTokens, cacheReadTokens, cacheWriteTokens, outputTokens, reasoningTokens, model, provider, cost }
327
+
328
+ if (picked.kind === "anthropic") {
329
+ // a provider's own feature, on its own client: automatic prompt caching as a call option
330
+ await picked.model.invoke(messages, { cache_control: { type: "ephemeral" } })
331
+ }
332
+
333
+ const next = picked.fallbacks[0] // the same name through its other protocol, for the app's own retry
334
+ const { embeddings, dimensions } = await ai.embeddings("embed")
335
+ ```
336
+
337
+ - `ai.chat(name, { timeoutMs?, maxTokens?, fields? })` reads the gateway's `GET /v1/models` once (again when a name is
338
+ unknown, or on `ai.refresh()`) and builds the client: `maxRetries: 0` (the router fails over between keys and
339
+ providers; the app owns its tries), usage on streams, the injected `fetch`, the timeout. `fields.openai` and
340
+ `fields.anthropic` are extra constructor fields of that kind's client.
341
+ - A profile whose candidates span two protocols is the client of its first candidate's protocol; `fallbacks` holds the
342
+ other protocols' clients in the order the pool tries them, each kept on its own protocol's providers. Moving to one
343
+ is the app's retry.
344
+ - `usageOf(reply)` has one shape for both kinds, unary and streamed (the chunks added up): `inputTokens` is the prompt
345
+ billed at the full price, beside the cache reads and writes; `model` and `provider` are the pool's ids of the lane
346
+ that answered.
347
+ - `@langchain/openai`, `@langchain/anthropic` and `@langchain/core` are optional peer dependencies, imported when a kind
348
+ is first built: install the ones the pool's protocols need. A refusal is a `RoutedModelsError` with a code
349
+ (`ROUTED_NAME_UNKNOWN`, `ROUTED_CATALOG_UNAVAILABLE`, `ROUTED_KIND_UNSUPPORTED`, `ROUTED_NOT_CHAT`,
350
+ `ROUTED_NOT_EMBEDDINGS`, `ROUTED_PACKAGE_MISSING`).
351
+ - A new protocol is one interface member and one factory: `declare module "@starci/ai-router/langchain" { interface
352
+ RoutedChatKinds { gemini: ChatGoogle } }` and `registerChatKind("gemini", async (context) => new ChatGoogle(...))`.
353
+
354
+ ### Call it from an Anthropic client
355
+
356
+ `POST /v1/messages` speaks the Anthropic Messages protocol under the same virtual key, sent as `x-api-key`. The request
357
+ reaches an Anthropic provider untranslated (cache breakpoints, thinking blocks, betas), the answer and the stream events
358
+ come back as the provider sent them, and errors are in Anthropic's shape. A name only an OpenAI-protocol provider serves
359
+ is refused 400 `model_not_supported`.
360
+
361
+ ```ts
362
+ import Anthropic from "@anthropic-ai/sdk"
363
+
364
+ const client = new Anthropic({ apiKey: process.env.STARCI_AI_KEY, baseURL: "http://127.0.0.1:3002", maxRetries: 0 })
365
+ const message = await client.messages.create({ model: "sonnet", max_tokens: 16000, messages: [{ role: "user", content: "hi" }] })
300
366
  ```
301
367
 
302
368
  ### Deploy
@@ -356,7 +422,7 @@ fails with an `AiRouterError` saying the port is unknown and `listen.port` is th
356
422
  (`127.0.0.1`) and is `0.0.0.0` only when you set it. With `serve`, `gateway.appPort` plus `gateway.portOffset`
357
423
  (default 1) does the same; it cannot be combined with `gateway.port`.
358
424
 
359
- A client of an API on 3001 uses `http://127.0.0.1:3002/v1` as its OpenAI base URL: LangChain (below), OpenClaw, or any
425
+ A client of an API on 3001 uses `http://127.0.0.1:3002/v1` as its OpenAI base URL: LangChain (below) or any
360
426
  OpenAI-compatible agent, with a virtual `sk-starci-...` key as its API key.
361
427
 
362
428
  ## Use it in a Nest app
@@ -515,10 +581,11 @@ A named registration provides no default instance; `@InjectAiRouterService()` wi
515
581
  | `providers[].headers` | extra headers for every request (the auth header, `authorization` or `x-api-key`, is ignored here) | none |
516
582
  | `providers[].weight` | multiplies the round-robin share of every lane of the provider | 1 |
517
583
  | `providers[].maxConcurrency` | in-flight requests across the provider's keys, in this process | unbounded |
518
- | `providers[].defaultMaxTokens` | the `max_tokens` sent to Anthropic, which requires one, when a request sets none | 4096 |
584
+ | `providers[].defaultMaxTokens` | the `max_tokens` sent to Anthropic, which requires one, when neither the request nor the model's `params` sets one | 16000 |
585
+ | `providers[].thinking` | on an `anthropic` provider: `drop` never sends an answer's thinking blocks back; `replay` sends a tool-calling turn back as it was received | `drop` |
519
586
  | `providers[].budget` | `{ costPerDay?, costPerMonth? }` in the models' cost unit; past it the provider is out until the next UTC day or month | none |
520
587
  | `providers[].keys[]` | `{ alias, env }` or `{ alias, resolve }`, plus `weight`, `maxConcurrency` (`keysFromEnv` builds them) | at least one |
521
- | `providers[].models[]` | `{ id, aliases?, contextWindow?, cost?, capabilities?, weight?, identity?, dimensions? }` | at least one |
588
+ | `providers[].models[]` | `{ id, aliases?, contextWindow?, cost?, capabilities?, weight?, identity?, dimensions?, params?, omit?, forcedToolUse?, cacheControl? }` | at least one |
522
589
  | `defaults` | the policy every call starts from | none |
523
590
  | `profiles` | named policies a call selects with `{ profile }` | none |
524
591
  | `resilience.keyBreaker` | `{ failureThreshold, cooldownMs, maxCooldownMs }` per key | 3, 15000, 300000 |
@@ -536,12 +603,78 @@ A named registration provides no default instance; `@InjectAiRouterService()` wi
536
603
  | `random` | the jitter source | `Math.random` |
537
604
 
538
605
  A model's `id` is the name sent to the provider; a caller may name it by `id` or any alias, and an alias shared by
539
- several providers makes them interchangeable for that call. `cost` is per one million tokens, `input` and `output`, in
540
- whatever unit you budget in. `capabilities` defaults to `chat`, `streaming`, `tools` and `json`; an embeddings model
606
+ several providers makes them interchangeable for that call. `cost` is per one million tokens, `input` and `output`
607
+ plus the optional `cacheRead`, `cacheWrite` and `cacheWrite1h` (each `input` when left out), in whatever unit you
608
+ budget in. `params`, `omit`, `forcedToolUse` and `cacheControl` shape the request for that model
609
+ ([One model's shape](#one-models-shape-never-the-apps)). `capabilities` defaults to `chat`, `streaming`, `tools` and `json`; an embeddings model
541
610
  declares `["embeddings"]` (refused on a kind with no embeddings API: `anthropic`, `deepseek`, `groq`). `identity` names
542
611
  one model served under different ids (`openai/text-embedding-3-small` at OpenRouter is `text-embedding-3-small`), and
543
612
  `dimensions` the vector length an embeddings model answers: a response of another length fails the attempt.
544
613
 
614
+ ## Prompt caching
615
+
616
+ Mark cache breakpoints once, in the OpenAI shape; each lane receives them as its provider takes them, and a provider
617
+ that caches a prefix on its own (DeepSeek, OpenAI, Gemini, most OpenAI-compatible endpoints) receives none:
618
+
619
+ ```ts
620
+ await router.chat({
621
+ model: "auto-heavy",
622
+ messages: [
623
+ { role: "system", content: [
624
+ { type: "text", text: craft, cache_control: { type: "ephemeral", ttl: "1h" } }, // the tools render before it
625
+ { type: "text", text: shopRules, cache_control: { type: "ephemeral" } },
626
+ { type: "text", text: stepGoal, cache_control: { type: "ephemeral" } },
627
+ ] },
628
+ ...conversation,
629
+ ],
630
+ tools,
631
+ cache_control: { type: "ephemeral" }, // automatic: one breakpoint on the last cacheable block
632
+ })
633
+ ```
634
+
635
+ A breakpoint sits on a content part, on a message (its last part), on a tool, or at the top level. Anthropic takes at
636
+ most four, the top-level one included: a request with more is refused as `invalid_request` before it is sent. `ttl` is
637
+ `"5m"` (the default) or `"1h"`. An `anthropic` provider and `anthropic/*` models on `openrouter` take breakpoints;
638
+ `cacheControl: true` on a model says its provider does too. [docs/design.md](docs/design.md#prompt-caching) has the
639
+ table per provider.
640
+
641
+ Usage reports the cache in one shape for every provider, and `cost` prices each part:
642
+
643
+ ```ts
644
+ reply.usage
645
+ // { prompt_tokens: 10000, completion_tokens: 500, total_tokens: 10500,
646
+ // prompt_tokens_details: { cached_tokens: 6000, cache_write_tokens: 3000, cache_write_1h_tokens: 1000 },
647
+ // completion_tokens_details: { reasoning_tokens: 200 } }
648
+ // with cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 } (Claude Sonnet 5.5's list prices,
649
+ // per million tokens, read 2026-11-10): 1000 at 2, 6000 at 0.1, 2000 at 2.5, 1000 at 4, 500 at 10.
650
+ ```
651
+
652
+ `prompt_tokens` is the whole prompt; the cache reads and writes are parts of it. A cache price a model does not declare
653
+ is its `input` price.
654
+
655
+ ## One model's shape, never the app's
656
+
657
+ A request that fails over reaches the next model in that model's own shape, from the model's config:
658
+
659
+ ```ts
660
+ {
661
+ id: "claude-sonnet-5-5",
662
+ cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 },
663
+ params: { max_tokens: 32000, output_config: { effort: "medium" } }, // sent when the request sets none
664
+ omit: ["temperature", "top_p", "top_k"], // never sent: this model refuses non-default sampling
665
+ forcedToolUse: false, // a forced tool_choice becomes auto plus an instruction: this model refuses to be forced
666
+ }
667
+ ```
668
+
669
+ With `forcedToolUse: false` a call is not guaranteed, so the caller checks that one was made and retries when none
670
+ was, as it would for any optional tool call.
671
+
672
+ On an `anthropic` provider, `thinking: "replay"` keeps the thinking blocks of an answer that calls tools in this
673
+ process and sends the turn back exactly as it was received on the next request of the loop, while the conversation
674
+ before it is unchanged; the default, `"drop"`, never sends them back, which the API accepts.
675
+ [docs/design.md](docs/design.md#thinking-with-tools) says why, and what a client that speaks the Anthropic protocol to
676
+ the gateway gets instead.
677
+
545
678
  ## Policies
546
679
 
547
680
  A call's policy is `defaults`, then the selected profile, then `options.policy`, merged field by field (later wins).
@@ -634,7 +767,9 @@ reply; breaking out of the loop cancels the body and frees the key's slot.
634
767
  `afterFirstToken: true`. The router does not retry it, since a second answer cannot be spliced onto the first; retry
635
768
  the whole call if your use allows it.
636
769
  - `timeoutMs` bounds the wait for the first token and then each gap between chunks; `deadlineMs` bounds the whole
637
- stream.
770
+ stream. A provider that shows it is at work before its first token (Anthropic's pings and hidden thinking) restarts
771
+ that wait, so a model that thinks for a minute is not timed out; a unary call has no such sign, so give a thinking
772
+ model's profile a `timeoutMs` that covers its thinking, or stream.
638
773
  - Stopping early counts as a success of the lane. Usage is read from a final usage chunk when the provider sends one
639
774
  (OpenAI does when you pass `stream_options: { include_usage: true }`).
640
775
  - Policy errors (an unknown model or profile) throw from `stream()` itself; everything else is thrown from iteration.
@@ -717,9 +852,14 @@ await app.close()
717
852
  ```
718
853
 
719
854
  A reply may set `status`, `headers`, `latencyMs`, `error: "network"`, `text`, `toolCalls` (`{ id, name, arguments,
720
- argumentChunks? }`), `usage`, `embedding`, a raw `body`, and `stream: { chunks, delayMs, failAfter, failWith: "network" |
855
+ argumentChunks? }`), `thinking` (`{ text?, signature? }` blocks before the text), `usage`, `cache` (`{ read?, write?,
856
+ write1h? }`, parts of the prompt, reported in the protocol's own usage shape), `reasoningTokens`, `embedding`, a raw
857
+ `body`, and `stream: { chunks, delayMs, failAfter, failWith: "network" |
721
858
  "error-event", stallAfter, stallMs, includeUsage }`. A request to `/messages` (an `anthropic` provider) is answered in
722
- the Anthropic shape, unary and SSE, and its key is read from `x-api-key`. `PROVIDER_ERRORS[kind][case]` holds each
859
+ the Anthropic shape, unary and SSE, and its key is read from `x-api-key`. Each recorded call lists where its request
860
+ marked cache breakpoints, `call.cacheBreakpoints` (`top`, `system.<i>`, `tools.<i>`, `messages.<i>`,
861
+ `messages.<i>.content.<j>`), the same for either protocol, so a spec asserts `["system.0", "system.1"]` reached Claude
862
+ and `[]` reached DeepSeek. `PROVIDER_ERRORS[kind][case]` holds each
723
863
  provider's real error answers (`rate_limited`, `quota_exhausted`, `invalid_key`, `overloaded`, `bad_request`, and a
724
864
  provider's second shapes such as `anthropic.spend_cap` or `openrouter.free_daily`), to script with
725
865
  `fake.on({ key }, PROVIDER_ERRORS.anthropic.quota_exhausted)`. Everything is deterministic except the latencies you
@@ -1,5 +1,5 @@
1
1
  import type { OnApplicationShutdown } from "@nestjs/common";
2
- import type { AiRouter, CallOptions, ChatCompletion, ChatCompletionRequest, ChatStream, EmbeddingsRequest, EmbeddingsResponse, HealthSnapshot, ResetTarget, RouterCatalog, RouterEvents, StatsSnapshot } from "./types.js";
2
+ import type { AiRouter, AnthropicMessageResult, AnthropicMessagesRequest, AnthropicStream, CallOptions, ChatCompletion, ChatCompletionRequest, ChatStream, EmbeddingsRequest, EmbeddingsResponse, HealthSnapshot, ResetTarget, RouterCatalog, RouterEvents, StatsSnapshot } from "./types.js";
3
3
  import type { AiRouterHandlers } from "./ai-router.options.js";
4
4
  /**
5
5
  * The injectable facade over one router. Every call it makes also follows the application's shutdown, and every
@@ -16,6 +16,10 @@ export declare class AiRouterService<Profiles extends string = string> implement
16
16
  stream(request: ChatCompletionRequest, options?: CallOptions<Profiles>): ChatStream;
17
17
  /** Sends an embeddings request through the pool. */
18
18
  embeddings(request: EmbeddingsRequest, options?: CallOptions<Profiles>): Promise<EmbeddingsResponse>;
19
+ /** Sends a request in the Anthropic Messages shape, untranslated, through the pool's Anthropic providers. */
20
+ messages(request: AnthropicMessagesRequest, options?: CallOptions<Profiles>): Promise<AnthropicMessageResult>;
21
+ /** Streams an Anthropic Messages request as the provider's own events; the stream ends `cancelled` when the application shuts down. */
22
+ messagesStream(request: AnthropicMessagesRequest, options?: CallOptions<Profiles>): AnthropicStream;
19
23
  /** Reads every breaker, blacklisted key, headroom and budget from the store. */
20
24
  health(): Promise<HealthSnapshot>;
21
25
  /** Returns this process's counters. */
@@ -36,6 +36,14 @@ export class AiRouterService {
36
36
  embeddings(request, options = {}) {
37
37
  return this.router.embeddings(request, this.#options(options));
38
38
  }
39
+ /** Sends a request in the Anthropic Messages shape, untranslated, through the pool's Anthropic providers. */
40
+ messages(request, options = {}) {
41
+ return this.router.messages(request, this.#options(options));
42
+ }
43
+ /** Streams an Anthropic Messages request as the provider's own events; the stream ends `cancelled` when the application shuts down. */
44
+ messagesStream(request, options = {}) {
45
+ return this.router.messagesStream(request, this.#options(options));
46
+ }
39
47
  /** Reads every breaker, blacklisted key, headroom and budget from the store. */
40
48
  health() {
41
49
  return this.router.health();
@@ -0,0 +1,35 @@
1
+ import type { ModelConfig, ProviderConfig, Usage } from "./types.js";
2
+ /**
3
+ * Prompt-cache breakpoints in the OpenAI shape (docs/design.md, "Prompt caching"). A request marks one with
4
+ * `cache_control: { type: "ephemeral", ttl? }` on a content part, on a message (it then sits on the message's last
5
+ * part), on a tool, or at the top level (automatic caching). A provider that caches a prefix on its own never sees a
6
+ * marker: it would be an unknown field there.
7
+ */
8
+ /** The most breakpoints one request may carry, the top-level one included (Anthropic's limit, read 2026-11-10). */
9
+ export declare const MAX_CACHE_BREAKPOINTS = 4;
10
+ /** Whether a model's provider takes `cache_control` markers: the model's own `cacheControl`, else by provider kind. */
11
+ export declare function takesCacheControl(provider: ProviderConfig, model: ModelConfig): boolean;
12
+ /** A marker a request carries, or undefined: anything that is not an object with a `type` is no marker. */
13
+ export declare function markerOf(holder: Record<string, unknown>): Record<string, unknown> | undefined;
14
+ /** An OpenAI chat request without any `cache_control`: top level, messages, content parts and tools. */
15
+ export declare function withoutCacheControl(body: Record<string, unknown>): Record<string, unknown>;
16
+ /**
17
+ * An OpenAI chat request in the shape an OpenAI-compatible provider that takes breakpoints documents (OpenRouter,
18
+ * openrouter.ai/docs/guides/best-practices/prompt-caching, read 2026-11-10): markers on content parts and at the top
19
+ * level stay; a marker on a message moves to its last part (a string content becomes one text part); a marker on a
20
+ * tool is removed, since tools render before the system prompt and a system breakpoint covers them.
21
+ */
22
+ export declare function withPartCacheControl(body: Record<string, unknown>): Record<string, unknown>;
23
+ /** The parts of a usage the cache and thinking account for; zero where the provider reported none. */
24
+ export interface TokenParts {
25
+ /** Prompt tokens read from the cache. */
26
+ readonly cached: number;
27
+ /** Prompt tokens written to the cache, the one-hour ones included. */
28
+ readonly cacheWrite: number;
29
+ /** Prompt tokens written to a one-hour cache. */
30
+ readonly cacheWrite1h: number;
31
+ /** Completion tokens spent thinking. */
32
+ readonly reasoning: number;
33
+ }
34
+ /** Reads the cache and thinking parts of a usage, bounded by the totals they are parts of. */
35
+ export declare function tokenParts(usage: Usage | undefined): TokenParts;
package/dist/cache.js ADDED
@@ -0,0 +1,76 @@
1
+ import { isPlainRecord } from "@starci/common";
2
+ import { presetOf } from "./providers.js";
3
+ /**
4
+ * Prompt-cache breakpoints in the OpenAI shape (docs/design.md, "Prompt caching"). A request marks one with
5
+ * `cache_control: { type: "ephemeral", ttl? }` on a content part, on a message (it then sits on the message's last
6
+ * part), on a tool, or at the top level (automatic caching). A provider that caches a prefix on its own never sees a
7
+ * marker: it would be an unknown field there.
8
+ */
9
+ /** The most breakpoints one request may carry, the top-level one included (Anthropic's limit, read 2026-11-10). */
10
+ export const MAX_CACHE_BREAKPOINTS = 4;
11
+ const MARKER = "cache_control";
12
+ /** Whether a model's provider takes `cache_control` markers: the model's own `cacheControl`, else by provider kind. */
13
+ export function takesCacheControl(provider, model) {
14
+ if (model.cacheControl !== undefined)
15
+ return model.cacheControl;
16
+ const kind = presetOf(provider).kind;
17
+ return kind === "anthropic" || (kind === "openrouter" && model.id.startsWith("anthropic/"));
18
+ }
19
+ /** A marker a request carries, or undefined: anything that is not an object with a `type` is no marker. */
20
+ export function markerOf(holder) {
21
+ const marker = holder[MARKER];
22
+ return isPlainRecord(marker) && typeof marker.type === "string" ? marker : undefined;
23
+ }
24
+ function without(record) {
25
+ return MARKER in record ? Object.fromEntries(Object.entries(record).filter(([field]) => field !== MARKER)) : record;
26
+ }
27
+ function mapList(value, each) {
28
+ return Array.isArray(value) ? value.map((item) => (isPlainRecord(item) ? each(item) : item)) : value;
29
+ }
30
+ /** An OpenAI chat request without any `cache_control`: top level, messages, content parts and tools. */
31
+ export function withoutCacheControl(body) {
32
+ const out = without(body);
33
+ const messages = mapList(body.messages, (message) => ({ ...without(message), ...(Array.isArray(message.content) ? { content: mapList(message.content, without) } : {}) }));
34
+ const tools = mapList(body.tools, without);
35
+ return { ...out, ...(body.messages === undefined ? {} : { messages }), ...(body.tools === undefined ? {} : { tools }) };
36
+ }
37
+ /**
38
+ * An OpenAI chat request in the shape an OpenAI-compatible provider that takes breakpoints documents (OpenRouter,
39
+ * openrouter.ai/docs/guides/best-practices/prompt-caching, read 2026-11-10): markers on content parts and at the top
40
+ * level stay; a marker on a message moves to its last part (a string content becomes one text part); a marker on a
41
+ * tool is removed, since tools render before the system prompt and a system breakpoint covers them.
42
+ */
43
+ export function withPartCacheControl(body) {
44
+ const messages = mapList(body.messages, (message) => {
45
+ const marker = markerOf(message);
46
+ if (!(MARKER in message))
47
+ return message;
48
+ const rest = without(message);
49
+ if (marker === undefined)
50
+ return rest;
51
+ const content = rest.content;
52
+ if (typeof content === "string")
53
+ return content === "" ? rest : { ...rest, content: [{ type: "text", text: content, [MARKER]: marker }] };
54
+ const parts = Array.isArray(content) ? content : [];
55
+ const last = parts.at(-1);
56
+ return isPlainRecord(last) ? { ...rest, content: [...parts.slice(0, -1), { ...last, [MARKER]: marker }] } : rest;
57
+ });
58
+ return { ...body, ...(body.messages === undefined ? {} : { messages }), ...(body.tools === undefined ? {} : { tools: mapList(body.tools, without) }) };
59
+ }
60
+ function count(value) {
61
+ return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : 0;
62
+ }
63
+ /** Reads the cache and thinking parts of a usage, bounded by the totals they are parts of. */
64
+ export function tokenParts(usage) {
65
+ const prompt = isPlainRecord(usage?.prompt_tokens_details) ? usage.prompt_tokens_details : {};
66
+ const completion = isPlainRecord(usage?.completion_tokens_details) ? usage.completion_tokens_details : {};
67
+ const total = count(usage?.prompt_tokens);
68
+ const cached = Math.min(count(prompt.cached_tokens), total);
69
+ const cacheWrite = Math.min(count(prompt.cache_write_tokens), total - cached);
70
+ return {
71
+ cached,
72
+ cacheWrite,
73
+ cacheWrite1h: Math.min(count(prompt.cache_write_1h_tokens), cacheWrite),
74
+ reasoning: Math.min(count(completion.reasoning_tokens), count(usage?.completion_tokens)),
75
+ };
76
+ }
package/dist/config.js CHANGED
@@ -130,6 +130,8 @@ export function validateConfig(config) {
130
130
  positive(provider.weight, `provider "${provider.id}" weight`);
131
131
  positive(provider.maxConcurrency, `provider "${provider.id}" maxConcurrency`);
132
132
  positive(provider.defaultMaxTokens, `provider "${provider.id}" defaultMaxTokens`);
133
+ if (provider.thinking !== undefined && preset.protocol !== "anthropic")
134
+ throw configError(`provider "${provider.id}" (${preset.kind}) sets thinking, which only an anthropic provider has`);
133
135
  positive(provider.budget?.costPerDay, `provider "${provider.id}" budget.costPerDay`, true);
134
136
  positive(provider.budget?.costPerMonth, `provider "${provider.id}" budget.costPerMonth`, true);
135
137
  if (provider.keys.length === 0)
@@ -158,6 +160,13 @@ export function validateConfig(config) {
158
160
  models.add(model.id);
159
161
  positive(model.weight, `model "${model.id}" weight`);
160
162
  positive(model.contextWindow, `model "${model.id}" contextWindow`);
163
+ for (const price of ["input", "output", "cacheRead", "cacheWrite", "cacheWrite1h"])
164
+ positive(model.cost?.[price], `model "${model.id}" cost.${price}`, true);
165
+ for (const field of model.omit ?? []) {
166
+ // Without its messages or its model a request is no request; every other field is the model's to refuse.
167
+ if (field === "" || field === "messages" || field === "model")
168
+ throw configError(`model "${model.id}" cannot omit "${field}"`);
169
+ }
161
170
  if (model.dimensions !== undefined && (!Number.isInteger(model.dimensions) || model.dimensions < 1))
162
171
  throw configError(`model "${model.id}" dimensions must be a positive integer`);
163
172
  if (!preset.embeddings && (model.capabilities ?? []).includes("embeddings"))
@@ -310,12 +319,15 @@ export function buildCatalog(config) {
310
319
  for (const model of provider.models) {
311
320
  let entry = models.get(model.id);
312
321
  if (entry === undefined) {
313
- entry = { aliases: new Set(), providers: [], capabilities: new Set() };
322
+ entry = { aliases: new Set(), providers: [], protocols: new Set(), capabilities: new Set() };
314
323
  models.set(model.id, entry);
315
324
  }
316
325
  for (const alias of model.aliases ?? [])
317
326
  entry.aliases.add(alias);
318
327
  entry.providers.push(provider.id);
328
+ entry.protocols.add(presetOf(provider).protocol);
329
+ if (entry.dimensions === undefined && model.dimensions !== undefined)
330
+ entry.dimensions = model.dimensions;
319
331
  for (const capability of model.capabilities ?? DEFAULT_CAPABILITIES)
320
332
  entry.capabilities.add(capability);
321
333
  if (entry.contextWindow === undefined && model.contextWindow !== undefined)
@@ -331,19 +343,36 @@ export function buildCatalog(config) {
331
343
  const capabilities = new Set(matchesOf(config, { model: name }).flatMap(({ model }) => model.capabilities ?? DEFAULT_CAPABILITIES));
332
344
  return OPERATIONS.filter((operation) => capabilities.has(operation));
333
345
  };
346
+ // The protocols a profile's candidates are served through, in the order the profile tries them.
347
+ const protocols = (policy, names) => {
348
+ const source = sourceOf(defaults, policy, {}, undefined);
349
+ const refs = "candidates" in source ? source.candidates.flat() : names.map((model) => ({ model }));
350
+ return [...new Set(refs.flatMap((ref) => matchesOf(config, ref).map(({ provider }) => presetOf(provider).protocol)))];
351
+ };
334
352
  return {
335
353
  defaults: namesOf({}, defaults),
354
+ providers: config.providers.map((provider) => ({ id: provider.id, kind: presetOf(provider).kind, protocol: presetOf(provider).protocol })),
336
355
  profiles: Object.entries(config.profiles ?? {}).map(([name, policy]) => {
337
356
  const named = namesOf(defaults, policy);
338
- return { name, ...named, pinned: policy.pinned === true, serves: serves(named.model) };
357
+ const dimensions = named.model === undefined ? undefined : matchesOf(config, { model: named.model })[0]?.model.dimensions;
358
+ return {
359
+ name,
360
+ ...named,
361
+ pinned: policy.pinned === true,
362
+ serves: serves(named.model),
363
+ protocols: protocols(policy, [...(named.model === undefined ? [] : [named.model]), ...named.fallbacks]),
364
+ ...(dimensions === undefined ? {} : { dimensions }),
365
+ };
339
366
  }),
340
367
  models: [...models].map(([id, entry]) => ({
341
368
  id,
342
369
  aliases: [...entry.aliases],
343
370
  providers: entry.providers,
371
+ protocols: [...entry.protocols],
344
372
  capabilities: [...entry.capabilities],
345
373
  ...(entry.contextWindow === undefined ? {} : { contextWindow: entry.contextWindow }),
346
374
  ...(entry.cost === undefined ? {} : { cost: entry.cost }),
375
+ ...(entry.dimensions === undefined ? {} : { dimensions: entry.dimensions }),
347
376
  })),
348
377
  };
349
378
  }