@starci/ai-router 0.6.1 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +29 -0
- package/README.md +161 -21
- package/dist/ai-router.service.d.ts +5 -1
- package/dist/ai-router.service.js +8 -0
- package/dist/cache.d.ts +35 -0
- package/dist/cache.js +76 -0
- package/dist/config.js +31 -2
- package/dist/fake.d.ts +27 -0
- package/dist/fake.js +84 -15
- package/dist/gateway/config-file.js +9 -1
- package/dist/gateway/gateway.js +189 -17
- package/dist/langchain/index.d.ts +137 -0
- package/dist/langchain/index.js +182 -0
- package/dist/model-request.d.ts +7 -0
- package/dist/model-request.js +54 -0
- package/dist/pool.d.ts +5 -1
- package/dist/pool.js +17 -4
- package/dist/router.js +84 -21
- package/dist/testing.d.ts +1 -1
- package/dist/types.d.ts +130 -3
- package/dist/wire/anthropic.d.ts +18 -8
- package/dist/wire/anthropic.js +238 -58
- package/dist/wire/native.d.ts +29 -0
- package/dist/wire/native.js +56 -0
- package/dist/wire/thinking-memory.d.ts +32 -0
- package/dist/wire/thinking-memory.js +61 -0
- package/dist/wire/wire.d.ts +12 -7
- package/dist/wire/wire.js +10 -1
- package/docs/design.md +189 -19
- package/package.json +26 -6
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,34 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.7.0 - 2026-11-13
|
|
4
|
+
|
|
5
|
+
One virtual key, two protocols: the gateway speaks Anthropic Messages beside OpenAI, a LangChain app asks for the client
|
|
6
|
+
of a name, and prompt caching and thinking cross both. [docs/design.md](docs/design.md) has the sections "Prompt
|
|
7
|
+
caching", "Thinking with tools", "A model's own request shape", "The native Anthropic route", "Who served, and what it
|
|
8
|
+
cost" and "The LangChain entry" (Anthropic's docs read 2026-11-10; LangChain's packages read at the versions below).
|
|
9
|
+
|
|
10
|
+
- Added: `@starci/ai-router/langchain`. `createRoutedModels({ gatewayUrl, virtualKey, fetch?, timeoutMs?, headers? })` gives `chat(name)`, a discriminated union by protocol (`{ kind: "openai", model: ChatOpenAI }`, `{ kind: "anthropic", model: ChatAnthropic }`, with `name` and `fallbacks`), `embeddings(name)` (`{ kind: "openai", embeddings: OpenAIEmbeddings, dimensions }`) and `refresh()`; `usageOf(reply)` reads one usage shape for both kinds (`inputTokens`, `cacheReadTokens`, `cacheWriteTokens`, `outputTokens`, `reasoningTokens`, `model`, `provider`, `cost`); `RoutedChatKinds` and `registerChatKind` add a kind without touching the core; `RoutedModelsError`. The clients never retry (`maxRetries: 0`). `@langchain/openai` (^1.8.2), `@langchain/anthropic` (^1.6.2) and `@langchain/core` (^1.3.5) are optional peer dependencies, imported when a kind is first built.
|
|
11
|
+
- Added: the gateway's `POST /v1/messages`, the Anthropic Messages protocol under the same virtual key (`x-api-key` or a bearer token): the request reaches an Anthropic provider untranslated, the answer and stream events come back as sent, errors are in Anthropic's shape, `anthropic-beta` is forwarded, and a name no Anthropic provider serves is refused 400 `model_not_supported`. In process: `router.messages()` and `router.messagesStream()` (`AnthropicMessagesRequest`, `AnthropicMessageResult`, `AnthropicStream`, `AnthropicStreamEvent`), also on `AiRouterService`.
|
|
12
|
+
- Added: `GET /v1/models` lists `kind`, `kinds`, `serves` and `dimensions` per name; the catalog has `providers` (`CatalogProvider`), `CatalogModel.protocols` and `dimensions`, `CatalogProfile.protocols` and `dimensions`. A request header `x-ai-router-kind: openai | anthropic` keeps a chat call on that protocol's providers.
|
|
13
|
+
- Added: prompt-cache breakpoints in the OpenAI shape: `cache_control` (`{ type: "ephemeral", ttl? }`, `CacheControl`) on a content part, a message, a tool or the top level. The Anthropic adapter sends them as blocks, tool and top-level breakpoints (a marked system prompt becomes system blocks) and refuses more than four; `anthropic/*` on OpenRouter keeps part and top-level breakpoints; every other lane receives none. `ModelConfig.cacheControl` overrides per model.
|
|
14
|
+
- Added: usage reports the cache and thinking apart, in one shape for every provider: `prompt_tokens_details.cached_tokens`, `cache_write_tokens`, `cache_write_1h_tokens` and `completion_tokens_details.reasoning_tokens` (`PromptTokensDetails`, `CompletionTokensDetails`); `ModelCost` takes `cacheRead`, `cacheWrite` and `cacheWrite1h` and a call's cost prices each part; `LaneStats` counts `cachedTokens`, `cacheWriteTokens` and `reasoningTokens`.
|
|
15
|
+
- Added: a model's own request shape, applied per attempt on both routes: `ModelConfig.params` (default request fields), `omit` (fields never sent) and `forcedToolUse: false` (a forced `tool_choice` becomes `auto` plus an instruction, for models that refuse one).
|
|
16
|
+
- Added: `ProviderConfig.thinking` on an `anthropic` provider: `drop` (the default) never sends thinking blocks back; `replay` keeps each tool-calling turn in process and sends it back exactly as received while the conversation before it is unchanged. Thinking text, when the request asks to see it, is `reasoning_content`.
|
|
17
|
+
- Added: every answer names its lane inside its usage, `usage.served = { provider, model, attempts, cost? }`, on both routes, unary and streamed; response headers `x-ai-router-provider` and `x-ai-router-cost`; the gateway takes a client's well-formed `x-request-id` as the call's id.
|
|
18
|
+
- Added: the fake provider scripts `thinking`, `cache` and `reasoningTokens`, gives each answer its own id (`FakeCall.answerId`), and records where each request marked cache breakpoints (`FakeCall.cacheBreakpoints`); `FakeThinking`, `FakeCacheUsage`.
|
|
19
|
+
- Changed: the `max_tokens` an Anthropic provider is sent when nothing sets one is 16000 (was 4096): thinking counts toward it on current models. `defaultMaxTokens` or a model's `params` sets another.
|
|
20
|
+
- Changed: a provider stream that shows it is at work before its first token (Anthropic's pings and hidden thinking) restarts the attempt's idle timeout instead of being timed out at `timeoutMs`.
|
|
21
|
+
- Changed: Anthropic usage adds `cache_write_tokens` to `prompt_tokens_details` (and `cache_write_1h_tokens` when the answer splits writes); a refusal sets `message.refusal`.
|
|
22
|
+
- Fixed: the README's and the design's example price of `claude-sonnet-5-5` (2 and 10 per million tokens, cache read 0.1, writes 2.5 and 4).
|
|
23
|
+
|
|
24
|
+
### Migration from 0.6
|
|
25
|
+
|
|
26
|
+
- A request that carried `cache_control` to a provider other than Anthropic or `anthropic/*` on OpenRouter no longer sends it; a model whose provider takes breakpoints (Gemini on OpenRouter, Qwen) sets `cacheControl: true`.
|
|
27
|
+
- A pool with an Anthropic model whose output limit is below 16000 sets `defaultMaxTokens` (or the model's `params.max_tokens`).
|
|
28
|
+
- A custom `Wire` takes the lane's `ModelConfig` in `chat` (was the model id) and may return and receive a `context`; `WireEvent` adds `alive`.
|
|
29
|
+
- Code that builds a `RouterCatalog`, a `CatalogModel` or a `CatalogProfile` adds `providers` and `protocols`; code that implements `AiRouter` adds `messages` and `messagesStream`; a `LaneStats` literal adds the three token counters.
|
|
30
|
+
- Fake answers carry `msg_fake_<n>` and `fake-<n>` ids, no longer `msg_fake` and `fake-stream`.
|
|
31
|
+
|
|
3
32
|
## 0.6.1 - 2026-11-13
|
|
4
33
|
|
|
5
34
|
- Changed: the README's examples use neutral profile, label and owner names. No code changes.
|
package/README.md
CHANGED
|
@@ -1,13 +1,15 @@
|
|
|
1
1
|
# @starci/ai-router
|
|
2
2
|
|
|
3
|
-
A router for
|
|
4
|
-
It load-balances below (Anthropic, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible
|
|
5
|
-
blacklists a key whose quota is spent or whose secret is refused, and exposes one
|
|
6
|
-
usable three ways over one engine:
|
|
7
|
-
|
|
8
|
-
- **as a gateway service** (`run.js serve`): an
|
|
9
|
-
included, `/v1/embeddings`, `/v1/models`)
|
|
10
|
-
app (LangChain, an OpenAI SDK, curl, a non-Node
|
|
3
|
+
A router for chat and embeddings calls across many providers, each with an array of keys, behind one virtual key per
|
|
4
|
+
app. It load-balances below (Anthropic, OpenRouter, Qwen, DeepSeek, OpenAI, Gemini, Groq and any OpenAI-compatible
|
|
5
|
+
endpoint), blacklists a key whose quota is spent or whose secret is refused, and exposes one gateway above that speaks
|
|
6
|
+
the OpenAI and the Anthropic protocols. It is usable three ways over one engine:
|
|
7
|
+
|
|
8
|
+
- **as a gateway service** (`run.js serve`): an HTTP endpoint in the OpenAI shape (`/v1/chat/completions`, streaming
|
|
9
|
+
included, `/v1/embeddings`, `/v1/models`) and in the Anthropic Messages shape (`/v1/messages`) with its own virtual
|
|
10
|
+
API keys, per-key quotas and usage accounting, so any app (LangChain, an OpenAI or Anthropic SDK, curl, a non-Node
|
|
11
|
+
service) uses the pool by pointing its base URL at it; `@starci/ai-router/langchain` hands a LangChain app the right
|
|
12
|
+
client for a name;
|
|
11
13
|
- **as a NestJS module**: register the pool once at the app root, inject `AiRouterService`, and call `chat`, `stream`
|
|
12
14
|
and `embeddings`; `AiRouterGatewayModule` hosts the gateway in the same process;
|
|
13
15
|
- **as a plain Node library**: `createRouter(config)` from `@starci/ai-router/gateway`, with no Nest at all.
|
|
@@ -16,11 +18,13 @@ The router picks a healthy key for each attempt, backs off what is failing, hono
|
|
|
16
18
|
quota or credit is spent until its window ends (or its secret changes), retries on a different key or provider, falls
|
|
17
19
|
back across providers and models by profile, keeps embeddings on one model, and tells you what it did without ever
|
|
18
20
|
exposing a key or a prompt. [docs/design.md](docs/design.md) is the design: the provider matrix, the error
|
|
19
|
-
classification per provider, the blacklist lifecycle, profiles and key scopes, the Anthropic adapter
|
|
21
|
+
classification per provider, the blacklist lifecycle, profiles and key scopes, the Anthropic adapter, prompt caching,
|
|
22
|
+
thinking with tools, the native Anthropic route, the LangChain entry and the env naming.
|
|
20
23
|
|
|
21
24
|
- ESM, typed, Node 20 or newer (the durable key store needs Node 22.13 or newer for `node:sqlite`). One runtime
|
|
22
25
|
dependency, `yaml`, for the config file. `@nestjs/common`, `@nestjs/core` (11 or 12) and `reflect-metadata` are
|
|
23
26
|
optional peer dependencies: the root entry needs them, `@starci/ai-router/gateway`, `@starci/ai-router/cli` and the `run.js` process entry do not.
|
|
27
|
+
`@langchain/openai`, `@langchain/anthropic` and `@langchain/core` are optional peers of `@starci/ai-router/langchain` alone.
|
|
24
28
|
- OpenAI chat-completions and embeddings shapes in and out. A provider's `kind` sets its protocol, auth header, default
|
|
25
29
|
base URL and error classification: `anthropic` (the Messages API, through a protocol adapter for chat, streaming,
|
|
26
30
|
tools and JSON), `openrouter`, `qwen` (DashScope compatible mode), `deepseek`, `openai`, `gemini` (its OpenAI
|
|
@@ -57,7 +61,7 @@ AiRouterModule.registerAsync({
|
|
|
57
61
|
useFactory: async () =>
|
|
58
62
|
poolFromEnv({
|
|
59
63
|
providers: [
|
|
60
|
-
{ kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input:
|
|
64
|
+
{ kind: "anthropic", models: [{ id: "claude-sonnet-5-5", cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 } }] },
|
|
61
65
|
{ kind: "openrouter", models: [{ id: "anthropic/claude-sonnet-5-5" }, { id: "openai/text-embedding-3-small", identity: "text-embedding-3-small", dimensions: 1536, capabilities: ["embeddings"] }] },
|
|
62
66
|
{ kind: "deepseek", models: [{ id: "deepseek-v4-pro" }] },
|
|
63
67
|
{ kind: "qwen", models: [{ id: "qwen3.8-max" }] },
|
|
@@ -148,15 +152,20 @@ profiles:
|
|
|
148
152
|
|---|---|---|
|
|
149
153
|
| `POST /v1/chat/completions` | virtual key | a chat completion; `stream: true` answers OpenAI `chat.completion.chunk` events over SSE ending with `data: [DONE]` |
|
|
150
154
|
| `POST /v1/embeddings` | virtual key | an embeddings response |
|
|
151
|
-
| `
|
|
155
|
+
| `POST /v1/messages` | virtual key (as `x-api-key`, or a bearer token) | an Anthropic message, untranslated, from an Anthropic provider; `stream: true` answers Anthropic's own events |
|
|
156
|
+
| `GET /v1/models`, `GET /v1/models/{id}` | virtual key | the profiles and pool models (ids and aliases) this key may call, each with `kind` (the protocol of its first candidate), `kinds`, `serves` and, for embeddings, `dimensions` |
|
|
152
157
|
| `GET /health` | none; the admin token adds the pool's breakers and keys | `{ "status": "ok" }`, or `degraded` when no provider has a usable key |
|
|
153
158
|
| `/admin/keys...` | admin token | the [admin API](#admin-api) |
|
|
154
159
|
|
|
155
160
|
A request's `model` names a profile (`chat`) or a pool model by id or alias (`gpt-4o-mini`, `fast`); a profile wins
|
|
156
161
|
when a name is both. A profile call routes by the profile's policy; a direct model call falls back only to the pool
|
|
157
|
-
defaults' fallbacks that the key may call itself. The answer is the provider's
|
|
158
|
-
|
|
159
|
-
|
|
162
|
+
defaults' fallbacks that the key may call itself. The answer is the provider's body without the router's `routing`
|
|
163
|
+
(which names a key); the lane that answered is inside its usage, `usage.served = { provider, model, attempts, cost? }`,
|
|
164
|
+
on every route, unary and streamed (the final usage chunk; the `message_start` event of an Anthropic stream). The
|
|
165
|
+
response carries `x-request-id` (the client's own when it sends a well-formed one no call in flight uses, so an app in
|
|
166
|
+
the router's process can match the call to the router's events; it must be unique per call), and a unary one
|
|
167
|
+
`x-ai-router-provider`, `x-ai-router-model`, `x-ai-router-attempts` and `x-ai-router-cost`. A request header
|
|
168
|
+
`x-ai-router-kind: openai` (or `anthropic`) keeps a chat call on the providers of that protocol. For a stream the gateway always asks the provider for usage (for accounting) and forwards the
|
|
160
169
|
usage chunk only when the client set `stream_options.include_usage`. Nothing is sent before the first chunk, so a pool
|
|
161
170
|
failure before the first token is an ordinary error response; after it, the stream ends with an OpenAI error event.
|
|
162
171
|
|
|
@@ -299,6 +308,63 @@ curl http://127.0.0.1:3002/v1/chat/completions -H "Authorization: Bearer $STARCI
|
|
|
299
308
|
-H "content-type: application/json" -d '{"model":"chat","stream":true,"messages":[{"role":"user","content":"hi"}]}'
|
|
300
309
|
```
|
|
301
310
|
|
|
311
|
+
### LangChain clients by name
|
|
312
|
+
|
|
313
|
+
An app that uses LangChain asks `@starci/ai-router/langchain` for the client of a name and never builds a provider
|
|
314
|
+
client itself. `kind` is the wire protocol the gateway serves that name through (`openai` for every OpenAI-compatible
|
|
315
|
+
provider, `anthropic` for an Anthropic one); the common path uses `model` and never reads it.
|
|
316
|
+
|
|
317
|
+
```ts
|
|
318
|
+
import { createRoutedModels, usageOf } from "@starci/ai-router/langchain"
|
|
319
|
+
import { createAgent } from "langchain"
|
|
320
|
+
|
|
321
|
+
const ai = createRoutedModels({ gatewayUrl: "http://127.0.0.1:3002", virtualKey: process.env.STARCI_AI_KEY ?? "", fetch: platformFetch, timeoutMs: 120_000 })
|
|
322
|
+
|
|
323
|
+
const picked = await ai.chat("auto-heavy") // a pool model id, an alias or a profile
|
|
324
|
+
const agent = createAgent({ model: picked.model, tools }) // a ChatAnthropic or a ChatOpenAI: the agent does not care
|
|
325
|
+
const reply = await picked.model.invoke(messages)
|
|
326
|
+
usageOf(reply) // { inputTokens, cacheReadTokens, cacheWriteTokens, outputTokens, reasoningTokens, model, provider, cost }
|
|
327
|
+
|
|
328
|
+
if (picked.kind === "anthropic") {
|
|
329
|
+
// a provider's own feature, on its own client: automatic prompt caching as a call option
|
|
330
|
+
await picked.model.invoke(messages, { cache_control: { type: "ephemeral" } })
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
const next = picked.fallbacks[0] // the same name through its other protocol, for the app's own retry
|
|
334
|
+
const { embeddings, dimensions } = await ai.embeddings("embed")
|
|
335
|
+
```
|
|
336
|
+
|
|
337
|
+
- `ai.chat(name, { timeoutMs?, maxTokens?, fields? })` reads the gateway's `GET /v1/models` once (again when a name is
|
|
338
|
+
unknown, or on `ai.refresh()`) and builds the client: `maxRetries: 0` (the router fails over between keys and
|
|
339
|
+
providers; the app owns its tries), usage on streams, the injected `fetch`, the timeout. `fields.openai` and
|
|
340
|
+
`fields.anthropic` are extra constructor fields of that kind's client.
|
|
341
|
+
- A profile whose candidates span two protocols is the client of its first candidate's protocol; `fallbacks` holds the
|
|
342
|
+
other protocols' clients in the order the pool tries them, each kept on its own protocol's providers. Moving to one
|
|
343
|
+
is the app's retry.
|
|
344
|
+
- `usageOf(reply)` has one shape for both kinds, unary and streamed (the chunks added up): `inputTokens` is the prompt
|
|
345
|
+
billed at the full price, beside the cache reads and writes; `model` and `provider` are the pool's ids of the lane
|
|
346
|
+
that answered.
|
|
347
|
+
- `@langchain/openai`, `@langchain/anthropic` and `@langchain/core` are optional peer dependencies, imported when a kind
|
|
348
|
+
is first built: install the ones the pool's protocols need. A refusal is a `RoutedModelsError` with a code
|
|
349
|
+
(`ROUTED_NAME_UNKNOWN`, `ROUTED_CATALOG_UNAVAILABLE`, `ROUTED_KIND_UNSUPPORTED`, `ROUTED_NOT_CHAT`,
|
|
350
|
+
`ROUTED_NOT_EMBEDDINGS`, `ROUTED_PACKAGE_MISSING`).
|
|
351
|
+
- A new protocol is one interface member and one factory: `declare module "@starci/ai-router/langchain" { interface
|
|
352
|
+
RoutedChatKinds { gemini: ChatGoogle } }` and `registerChatKind("gemini", async (context) => new ChatGoogle(...))`.
|
|
353
|
+
|
|
354
|
+
### Call it from an Anthropic client
|
|
355
|
+
|
|
356
|
+
`POST /v1/messages` speaks the Anthropic Messages protocol under the same virtual key, sent as `x-api-key`. The request
|
|
357
|
+
reaches an Anthropic provider untranslated (cache breakpoints, thinking blocks, betas), the answer and the stream events
|
|
358
|
+
come back as the provider sent them, and errors are in Anthropic's shape. A name only an OpenAI-protocol provider serves
|
|
359
|
+
is refused 400 `model_not_supported`.
|
|
360
|
+
|
|
361
|
+
```ts
|
|
362
|
+
import Anthropic from "@anthropic-ai/sdk"
|
|
363
|
+
|
|
364
|
+
const client = new Anthropic({ apiKey: process.env.STARCI_AI_KEY, baseURL: "http://127.0.0.1:3002", maxRetries: 0 })
|
|
365
|
+
const message = await client.messages.create({ model: "sonnet", max_tokens: 16000, messages: [{ role: "user", content: "hi" }] })
|
|
366
|
+
```
|
|
367
|
+
|
|
302
368
|
### Deploy
|
|
303
369
|
|
|
304
370
|
One container or process: `node node_modules/@starci/ai-router/dist/gateway/run.js serve --config
|
|
@@ -515,10 +581,11 @@ A named registration provides no default instance; `@InjectAiRouterService()` wi
|
|
|
515
581
|
| `providers[].headers` | extra headers for every request (the auth header, `authorization` or `x-api-key`, is ignored here) | none |
|
|
516
582
|
| `providers[].weight` | multiplies the round-robin share of every lane of the provider | 1 |
|
|
517
583
|
| `providers[].maxConcurrency` | in-flight requests across the provider's keys, in this process | unbounded |
|
|
518
|
-
| `providers[].defaultMaxTokens` | the `max_tokens` sent to Anthropic, which requires one, when
|
|
584
|
+
| `providers[].defaultMaxTokens` | the `max_tokens` sent to Anthropic, which requires one, when neither the request nor the model's `params` sets one | 16000 |
|
|
585
|
+
| `providers[].thinking` | on an `anthropic` provider: `drop` never sends an answer's thinking blocks back; `replay` sends a tool-calling turn back as it was received | `drop` |
|
|
519
586
|
| `providers[].budget` | `{ costPerDay?, costPerMonth? }` in the models' cost unit; past it the provider is out until the next UTC day or month | none |
|
|
520
587
|
| `providers[].keys[]` | `{ alias, env }` or `{ alias, resolve }`, plus `weight`, `maxConcurrency` (`keysFromEnv` builds them) | at least one |
|
|
521
|
-
| `providers[].models[]` | `{ id, aliases?, contextWindow?, cost?, capabilities?, weight?, identity?, dimensions? }` | at least one |
|
|
588
|
+
| `providers[].models[]` | `{ id, aliases?, contextWindow?, cost?, capabilities?, weight?, identity?, dimensions?, params?, omit?, forcedToolUse?, cacheControl? }` | at least one |
|
|
522
589
|
| `defaults` | the policy every call starts from | none |
|
|
523
590
|
| `profiles` | named policies a call selects with `{ profile }` | none |
|
|
524
591
|
| `resilience.keyBreaker` | `{ failureThreshold, cooldownMs, maxCooldownMs }` per key | 3, 15000, 300000 |
|
|
@@ -536,12 +603,78 @@ A named registration provides no default instance; `@InjectAiRouterService()` wi
|
|
|
536
603
|
| `random` | the jitter source | `Math.random` |
|
|
537
604
|
|
|
538
605
|
A model's `id` is the name sent to the provider; a caller may name it by `id` or any alias, and an alias shared by
|
|
539
|
-
several providers makes them interchangeable for that call. `cost` is per one million tokens, `input` and `output
|
|
540
|
-
|
|
606
|
+
several providers makes them interchangeable for that call. `cost` is per one million tokens, `input` and `output`
|
|
607
|
+
plus the optional `cacheRead`, `cacheWrite` and `cacheWrite1h` (each `input` when left out), in whatever unit you
|
|
608
|
+
budget in. `params`, `omit`, `forcedToolUse` and `cacheControl` shape the request for that model
|
|
609
|
+
([One model's shape](#one-models-shape-never-the-apps)). `capabilities` defaults to `chat`, `streaming`, `tools` and `json`; an embeddings model
|
|
541
610
|
declares `["embeddings"]` (refused on a kind with no embeddings API: `anthropic`, `deepseek`, `groq`). `identity` names
|
|
542
611
|
one model served under different ids (`openai/text-embedding-3-small` at OpenRouter is `text-embedding-3-small`), and
|
|
543
612
|
`dimensions` the vector length an embeddings model answers: a response of another length fails the attempt.
|
|
544
613
|
|
|
614
|
+
## Prompt caching
|
|
615
|
+
|
|
616
|
+
Mark cache breakpoints once, in the OpenAI shape; each lane receives them as its provider takes them, and a provider
|
|
617
|
+
that caches a prefix on its own (DeepSeek, OpenAI, Gemini, most OpenAI-compatible endpoints) receives none:
|
|
618
|
+
|
|
619
|
+
```ts
|
|
620
|
+
await router.chat({
|
|
621
|
+
model: "auto-heavy",
|
|
622
|
+
messages: [
|
|
623
|
+
{ role: "system", content: [
|
|
624
|
+
{ type: "text", text: craft, cache_control: { type: "ephemeral", ttl: "1h" } }, // the tools render before it
|
|
625
|
+
{ type: "text", text: shopRules, cache_control: { type: "ephemeral" } },
|
|
626
|
+
{ type: "text", text: stepGoal, cache_control: { type: "ephemeral" } },
|
|
627
|
+
] },
|
|
628
|
+
...conversation,
|
|
629
|
+
],
|
|
630
|
+
tools,
|
|
631
|
+
cache_control: { type: "ephemeral" }, // automatic: one breakpoint on the last cacheable block
|
|
632
|
+
})
|
|
633
|
+
```
|
|
634
|
+
|
|
635
|
+
A breakpoint sits on a content part, on a message (its last part), on a tool, or at the top level. Anthropic takes at
|
|
636
|
+
most four, the top-level one included: a request with more is refused as `invalid_request` before it is sent. `ttl` is
|
|
637
|
+
`"5m"` (the default) or `"1h"`. An `anthropic` provider and `anthropic/*` models on `openrouter` take breakpoints;
|
|
638
|
+
`cacheControl: true` on a model says its provider does too. [docs/design.md](docs/design.md#prompt-caching) has the
|
|
639
|
+
table per provider.
|
|
640
|
+
|
|
641
|
+
Usage reports the cache in one shape for every provider, and `cost` prices each part:
|
|
642
|
+
|
|
643
|
+
```ts
|
|
644
|
+
reply.usage
|
|
645
|
+
// { prompt_tokens: 10000, completion_tokens: 500, total_tokens: 10500,
|
|
646
|
+
// prompt_tokens_details: { cached_tokens: 6000, cache_write_tokens: 3000, cache_write_1h_tokens: 1000 },
|
|
647
|
+
// completion_tokens_details: { reasoning_tokens: 200 } }
|
|
648
|
+
// with cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 } (Claude Sonnet 5.5's list prices,
|
|
649
|
+
// per million tokens, read 2026-11-10): 1000 at 2, 6000 at 0.1, 2000 at 2.5, 1000 at 4, 500 at 10.
|
|
650
|
+
```
|
|
651
|
+
|
|
652
|
+
`prompt_tokens` is the whole prompt; the cache reads and writes are parts of it. A cache price a model does not declare
|
|
653
|
+
is its `input` price.
|
|
654
|
+
|
|
655
|
+
## One model's shape, never the app's
|
|
656
|
+
|
|
657
|
+
A request that fails over reaches the next model in that model's own shape, from the model's config:
|
|
658
|
+
|
|
659
|
+
```ts
|
|
660
|
+
{
|
|
661
|
+
id: "claude-sonnet-5-5",
|
|
662
|
+
cost: { input: 2, output: 10, cacheRead: 0.1, cacheWrite: 2.5, cacheWrite1h: 4 },
|
|
663
|
+
params: { max_tokens: 32000, output_config: { effort: "medium" } }, // sent when the request sets none
|
|
664
|
+
omit: ["temperature", "top_p", "top_k"], // never sent: this model refuses non-default sampling
|
|
665
|
+
forcedToolUse: false, // a forced tool_choice becomes auto plus an instruction: this model refuses to be forced
|
|
666
|
+
}
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
With `forcedToolUse: false` a call is not guaranteed, so the caller checks that one was made and retries when none
|
|
670
|
+
was, as it would for any optional tool call.
|
|
671
|
+
|
|
672
|
+
On an `anthropic` provider, `thinking: "replay"` keeps the thinking blocks of an answer that calls tools in this
|
|
673
|
+
process and sends the turn back exactly as it was received on the next request of the loop, while the conversation
|
|
674
|
+
before it is unchanged; the default, `"drop"`, never sends them back, which the API accepts.
|
|
675
|
+
[docs/design.md](docs/design.md#thinking-with-tools) says why, and what a client that speaks the Anthropic protocol to
|
|
676
|
+
the gateway gets instead.
|
|
677
|
+
|
|
545
678
|
## Policies
|
|
546
679
|
|
|
547
680
|
A call's policy is `defaults`, then the selected profile, then `options.policy`, merged field by field (later wins).
|
|
@@ -634,7 +767,9 @@ reply; breaking out of the loop cancels the body and frees the key's slot.
|
|
|
634
767
|
`afterFirstToken: true`. The router does not retry it, since a second answer cannot be spliced onto the first; retry
|
|
635
768
|
the whole call if your use allows it.
|
|
636
769
|
- `timeoutMs` bounds the wait for the first token and then each gap between chunks; `deadlineMs` bounds the whole
|
|
637
|
-
stream.
|
|
770
|
+
stream. A provider that shows it is at work before its first token (Anthropic's pings and hidden thinking) restarts
|
|
771
|
+
that wait, so a model that thinks for a minute is not timed out; a unary call has no such sign, so give a thinking
|
|
772
|
+
model's profile a `timeoutMs` that covers its thinking, or stream.
|
|
638
773
|
- Stopping early counts as a success of the lane. Usage is read from a final usage chunk when the provider sends one
|
|
639
774
|
(OpenAI does when you pass `stream_options: { include_usage: true }`).
|
|
640
775
|
- Policy errors (an unknown model or profile) throw from `stream()` itself; everything else is thrown from iteration.
|
|
@@ -717,9 +852,14 @@ await app.close()
|
|
|
717
852
|
```
|
|
718
853
|
|
|
719
854
|
A reply may set `status`, `headers`, `latencyMs`, `error: "network"`, `text`, `toolCalls` (`{ id, name, arguments,
|
|
720
|
-
argumentChunks? }`), `
|
|
855
|
+
argumentChunks? }`), `thinking` (`{ text?, signature? }` blocks before the text), `usage`, `cache` (`{ read?, write?,
|
|
856
|
+
write1h? }`, parts of the prompt, reported in the protocol's own usage shape), `reasoningTokens`, `embedding`, a raw
|
|
857
|
+
`body`, and `stream: { chunks, delayMs, failAfter, failWith: "network" |
|
|
721
858
|
"error-event", stallAfter, stallMs, includeUsage }`. A request to `/messages` (an `anthropic` provider) is answered in
|
|
722
|
-
the Anthropic shape, unary and SSE, and its key is read from `x-api-key`.
|
|
859
|
+
the Anthropic shape, unary and SSE, and its key is read from `x-api-key`. Each recorded call lists where its request
|
|
860
|
+
marked cache breakpoints, `call.cacheBreakpoints` (`top`, `system.<i>`, `tools.<i>`, `messages.<i>`,
|
|
861
|
+
`messages.<i>.content.<j>`), the same for either protocol, so a spec asserts `["system.0", "system.1"]` reached Claude
|
|
862
|
+
and `[]` reached DeepSeek. `PROVIDER_ERRORS[kind][case]` holds each
|
|
723
863
|
provider's real error answers (`rate_limited`, `quota_exhausted`, `invalid_key`, `overloaded`, `bad_request`, and a
|
|
724
864
|
provider's second shapes such as `anthropic.spend_cap` or `openrouter.free_daily`), to script with
|
|
725
865
|
`fake.on({ key }, PROVIDER_ERRORS.anthropic.quota_exhausted)`. Everything is deterministic except the latencies you
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import type { OnApplicationShutdown } from "@nestjs/common";
|
|
2
|
-
import type { AiRouter, CallOptions, ChatCompletion, ChatCompletionRequest, ChatStream, EmbeddingsRequest, EmbeddingsResponse, HealthSnapshot, ResetTarget, RouterCatalog, RouterEvents, StatsSnapshot } from "./types.js";
|
|
2
|
+
import type { AiRouter, AnthropicMessageResult, AnthropicMessagesRequest, AnthropicStream, CallOptions, ChatCompletion, ChatCompletionRequest, ChatStream, EmbeddingsRequest, EmbeddingsResponse, HealthSnapshot, ResetTarget, RouterCatalog, RouterEvents, StatsSnapshot } from "./types.js";
|
|
3
3
|
import type { AiRouterHandlers } from "./ai-router.options.js";
|
|
4
4
|
/**
|
|
5
5
|
* The injectable facade over one router. Every call it makes also follows the application's shutdown, and every
|
|
@@ -16,6 +16,10 @@ export declare class AiRouterService<Profiles extends string = string> implement
|
|
|
16
16
|
stream(request: ChatCompletionRequest, options?: CallOptions<Profiles>): ChatStream;
|
|
17
17
|
/** Sends an embeddings request through the pool. */
|
|
18
18
|
embeddings(request: EmbeddingsRequest, options?: CallOptions<Profiles>): Promise<EmbeddingsResponse>;
|
|
19
|
+
/** Sends a request in the Anthropic Messages shape, untranslated, through the pool's Anthropic providers. */
|
|
20
|
+
messages(request: AnthropicMessagesRequest, options?: CallOptions<Profiles>): Promise<AnthropicMessageResult>;
|
|
21
|
+
/** Streams an Anthropic Messages request as the provider's own events; the stream ends `cancelled` when the application shuts down. */
|
|
22
|
+
messagesStream(request: AnthropicMessagesRequest, options?: CallOptions<Profiles>): AnthropicStream;
|
|
19
23
|
/** Reads every breaker, blacklisted key, headroom and budget from the store. */
|
|
20
24
|
health(): Promise<HealthSnapshot>;
|
|
21
25
|
/** Returns this process's counters. */
|
|
@@ -36,6 +36,14 @@ export class AiRouterService {
|
|
|
36
36
|
embeddings(request, options = {}) {
|
|
37
37
|
return this.router.embeddings(request, this.#options(options));
|
|
38
38
|
}
|
|
39
|
+
/** Sends a request in the Anthropic Messages shape, untranslated, through the pool's Anthropic providers. */
|
|
40
|
+
messages(request, options = {}) {
|
|
41
|
+
return this.router.messages(request, this.#options(options));
|
|
42
|
+
}
|
|
43
|
+
/** Streams an Anthropic Messages request as the provider's own events; the stream ends `cancelled` when the application shuts down. */
|
|
44
|
+
messagesStream(request, options = {}) {
|
|
45
|
+
return this.router.messagesStream(request, this.#options(options));
|
|
46
|
+
}
|
|
39
47
|
/** Reads every breaker, blacklisted key, headroom and budget from the store. */
|
|
40
48
|
health() {
|
|
41
49
|
return this.router.health();
|
package/dist/cache.d.ts
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
import type { ModelConfig, ProviderConfig, Usage } from "./types.js";
|
|
2
|
+
/**
|
|
3
|
+
* Prompt-cache breakpoints in the OpenAI shape (docs/design.md, "Prompt caching"). A request marks one with
|
|
4
|
+
* `cache_control: { type: "ephemeral", ttl? }` on a content part, on a message (it then sits on the message's last
|
|
5
|
+
* part), on a tool, or at the top level (automatic caching). A provider that caches a prefix on its own never sees a
|
|
6
|
+
* marker: it would be an unknown field there.
|
|
7
|
+
*/
|
|
8
|
+
/** The most breakpoints one request may carry, the top-level one included (Anthropic's limit, read 2026-11-10). */
|
|
9
|
+
export declare const MAX_CACHE_BREAKPOINTS = 4;
|
|
10
|
+
/** Whether a model's provider takes `cache_control` markers: the model's own `cacheControl`, else by provider kind. */
|
|
11
|
+
export declare function takesCacheControl(provider: ProviderConfig, model: ModelConfig): boolean;
|
|
12
|
+
/** A marker a request carries, or undefined: anything that is not an object with a `type` is no marker. */
|
|
13
|
+
export declare function markerOf(holder: Record<string, unknown>): Record<string, unknown> | undefined;
|
|
14
|
+
/** An OpenAI chat request without any `cache_control`: top level, messages, content parts and tools. */
|
|
15
|
+
export declare function withoutCacheControl(body: Record<string, unknown>): Record<string, unknown>;
|
|
16
|
+
/**
|
|
17
|
+
* An OpenAI chat request in the shape an OpenAI-compatible provider that takes breakpoints documents (OpenRouter,
|
|
18
|
+
* openrouter.ai/docs/guides/best-practices/prompt-caching, read 2026-11-10): markers on content parts and at the top
|
|
19
|
+
* level stay; a marker on a message moves to its last part (a string content becomes one text part); a marker on a
|
|
20
|
+
* tool is removed, since tools render before the system prompt and a system breakpoint covers them.
|
|
21
|
+
*/
|
|
22
|
+
export declare function withPartCacheControl(body: Record<string, unknown>): Record<string, unknown>;
|
|
23
|
+
/** The parts of a usage the cache and thinking account for; zero where the provider reported none. */
|
|
24
|
+
export interface TokenParts {
|
|
25
|
+
/** Prompt tokens read from the cache. */
|
|
26
|
+
readonly cached: number;
|
|
27
|
+
/** Prompt tokens written to the cache, the one-hour ones included. */
|
|
28
|
+
readonly cacheWrite: number;
|
|
29
|
+
/** Prompt tokens written to a one-hour cache. */
|
|
30
|
+
readonly cacheWrite1h: number;
|
|
31
|
+
/** Completion tokens spent thinking. */
|
|
32
|
+
readonly reasoning: number;
|
|
33
|
+
}
|
|
34
|
+
/** Reads the cache and thinking parts of a usage, bounded by the totals they are parts of. */
|
|
35
|
+
export declare function tokenParts(usage: Usage | undefined): TokenParts;
|
package/dist/cache.js
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
import { isPlainRecord } from "@starci/common";
|
|
2
|
+
import { presetOf } from "./providers.js";
|
|
3
|
+
/**
|
|
4
|
+
* Prompt-cache breakpoints in the OpenAI shape (docs/design.md, "Prompt caching"). A request marks one with
|
|
5
|
+
* `cache_control: { type: "ephemeral", ttl? }` on a content part, on a message (it then sits on the message's last
|
|
6
|
+
* part), on a tool, or at the top level (automatic caching). A provider that caches a prefix on its own never sees a
|
|
7
|
+
* marker: it would be an unknown field there.
|
|
8
|
+
*/
|
|
9
|
+
/** The most breakpoints one request may carry, the top-level one included (Anthropic's limit, read 2026-11-10). */
|
|
10
|
+
export const MAX_CACHE_BREAKPOINTS = 4;
|
|
11
|
+
const MARKER = "cache_control";
|
|
12
|
+
/** Whether a model's provider takes `cache_control` markers: the model's own `cacheControl`, else by provider kind. */
|
|
13
|
+
export function takesCacheControl(provider, model) {
|
|
14
|
+
if (model.cacheControl !== undefined)
|
|
15
|
+
return model.cacheControl;
|
|
16
|
+
const kind = presetOf(provider).kind;
|
|
17
|
+
return kind === "anthropic" || (kind === "openrouter" && model.id.startsWith("anthropic/"));
|
|
18
|
+
}
|
|
19
|
+
/** A marker a request carries, or undefined: anything that is not an object with a `type` is no marker. */
|
|
20
|
+
export function markerOf(holder) {
|
|
21
|
+
const marker = holder[MARKER];
|
|
22
|
+
return isPlainRecord(marker) && typeof marker.type === "string" ? marker : undefined;
|
|
23
|
+
}
|
|
24
|
+
function without(record) {
|
|
25
|
+
return MARKER in record ? Object.fromEntries(Object.entries(record).filter(([field]) => field !== MARKER)) : record;
|
|
26
|
+
}
|
|
27
|
+
function mapList(value, each) {
|
|
28
|
+
return Array.isArray(value) ? value.map((item) => (isPlainRecord(item) ? each(item) : item)) : value;
|
|
29
|
+
}
|
|
30
|
+
/** An OpenAI chat request without any `cache_control`: top level, messages, content parts and tools. */
|
|
31
|
+
export function withoutCacheControl(body) {
|
|
32
|
+
const out = without(body);
|
|
33
|
+
const messages = mapList(body.messages, (message) => ({ ...without(message), ...(Array.isArray(message.content) ? { content: mapList(message.content, without) } : {}) }));
|
|
34
|
+
const tools = mapList(body.tools, without);
|
|
35
|
+
return { ...out, ...(body.messages === undefined ? {} : { messages }), ...(body.tools === undefined ? {} : { tools }) };
|
|
36
|
+
}
|
|
37
|
+
/**
|
|
38
|
+
* An OpenAI chat request in the shape an OpenAI-compatible provider that takes breakpoints documents (OpenRouter,
|
|
39
|
+
* openrouter.ai/docs/guides/best-practices/prompt-caching, read 2026-11-10): markers on content parts and at the top
|
|
40
|
+
* level stay; a marker on a message moves to its last part (a string content becomes one text part); a marker on a
|
|
41
|
+
* tool is removed, since tools render before the system prompt and a system breakpoint covers them.
|
|
42
|
+
*/
|
|
43
|
+
export function withPartCacheControl(body) {
|
|
44
|
+
const messages = mapList(body.messages, (message) => {
|
|
45
|
+
const marker = markerOf(message);
|
|
46
|
+
if (!(MARKER in message))
|
|
47
|
+
return message;
|
|
48
|
+
const rest = without(message);
|
|
49
|
+
if (marker === undefined)
|
|
50
|
+
return rest;
|
|
51
|
+
const content = rest.content;
|
|
52
|
+
if (typeof content === "string")
|
|
53
|
+
return content === "" ? rest : { ...rest, content: [{ type: "text", text: content, [MARKER]: marker }] };
|
|
54
|
+
const parts = Array.isArray(content) ? content : [];
|
|
55
|
+
const last = parts.at(-1);
|
|
56
|
+
return isPlainRecord(last) ? { ...rest, content: [...parts.slice(0, -1), { ...last, [MARKER]: marker }] } : rest;
|
|
57
|
+
});
|
|
58
|
+
return { ...body, ...(body.messages === undefined ? {} : { messages }), ...(body.tools === undefined ? {} : { tools: mapList(body.tools, without) }) };
|
|
59
|
+
}
|
|
60
|
+
function count(value) {
|
|
61
|
+
return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : 0;
|
|
62
|
+
}
|
|
63
|
+
/** Reads the cache and thinking parts of a usage, bounded by the totals they are parts of. */
|
|
64
|
+
export function tokenParts(usage) {
|
|
65
|
+
const prompt = isPlainRecord(usage?.prompt_tokens_details) ? usage.prompt_tokens_details : {};
|
|
66
|
+
const completion = isPlainRecord(usage?.completion_tokens_details) ? usage.completion_tokens_details : {};
|
|
67
|
+
const total = count(usage?.prompt_tokens);
|
|
68
|
+
const cached = Math.min(count(prompt.cached_tokens), total);
|
|
69
|
+
const cacheWrite = Math.min(count(prompt.cache_write_tokens), total - cached);
|
|
70
|
+
return {
|
|
71
|
+
cached,
|
|
72
|
+
cacheWrite,
|
|
73
|
+
cacheWrite1h: Math.min(count(prompt.cache_write_1h_tokens), cacheWrite),
|
|
74
|
+
reasoning: Math.min(count(completion.reasoning_tokens), count(usage?.completion_tokens)),
|
|
75
|
+
};
|
|
76
|
+
}
|
package/dist/config.js
CHANGED
|
@@ -130,6 +130,8 @@ export function validateConfig(config) {
|
|
|
130
130
|
positive(provider.weight, `provider "${provider.id}" weight`);
|
|
131
131
|
positive(provider.maxConcurrency, `provider "${provider.id}" maxConcurrency`);
|
|
132
132
|
positive(provider.defaultMaxTokens, `provider "${provider.id}" defaultMaxTokens`);
|
|
133
|
+
if (provider.thinking !== undefined && preset.protocol !== "anthropic")
|
|
134
|
+
throw configError(`provider "${provider.id}" (${preset.kind}) sets thinking, which only an anthropic provider has`);
|
|
133
135
|
positive(provider.budget?.costPerDay, `provider "${provider.id}" budget.costPerDay`, true);
|
|
134
136
|
positive(provider.budget?.costPerMonth, `provider "${provider.id}" budget.costPerMonth`, true);
|
|
135
137
|
if (provider.keys.length === 0)
|
|
@@ -158,6 +160,13 @@ export function validateConfig(config) {
|
|
|
158
160
|
models.add(model.id);
|
|
159
161
|
positive(model.weight, `model "${model.id}" weight`);
|
|
160
162
|
positive(model.contextWindow, `model "${model.id}" contextWindow`);
|
|
163
|
+
for (const price of ["input", "output", "cacheRead", "cacheWrite", "cacheWrite1h"])
|
|
164
|
+
positive(model.cost?.[price], `model "${model.id}" cost.${price}`, true);
|
|
165
|
+
for (const field of model.omit ?? []) {
|
|
166
|
+
// Without its messages or its model a request is no request; every other field is the model's to refuse.
|
|
167
|
+
if (field === "" || field === "messages" || field === "model")
|
|
168
|
+
throw configError(`model "${model.id}" cannot omit "${field}"`);
|
|
169
|
+
}
|
|
161
170
|
if (model.dimensions !== undefined && (!Number.isInteger(model.dimensions) || model.dimensions < 1))
|
|
162
171
|
throw configError(`model "${model.id}" dimensions must be a positive integer`);
|
|
163
172
|
if (!preset.embeddings && (model.capabilities ?? []).includes("embeddings"))
|
|
@@ -310,12 +319,15 @@ export function buildCatalog(config) {
|
|
|
310
319
|
for (const model of provider.models) {
|
|
311
320
|
let entry = models.get(model.id);
|
|
312
321
|
if (entry === undefined) {
|
|
313
|
-
entry = { aliases: new Set(), providers: [], capabilities: new Set() };
|
|
322
|
+
entry = { aliases: new Set(), providers: [], protocols: new Set(), capabilities: new Set() };
|
|
314
323
|
models.set(model.id, entry);
|
|
315
324
|
}
|
|
316
325
|
for (const alias of model.aliases ?? [])
|
|
317
326
|
entry.aliases.add(alias);
|
|
318
327
|
entry.providers.push(provider.id);
|
|
328
|
+
entry.protocols.add(presetOf(provider).protocol);
|
|
329
|
+
if (entry.dimensions === undefined && model.dimensions !== undefined)
|
|
330
|
+
entry.dimensions = model.dimensions;
|
|
319
331
|
for (const capability of model.capabilities ?? DEFAULT_CAPABILITIES)
|
|
320
332
|
entry.capabilities.add(capability);
|
|
321
333
|
if (entry.contextWindow === undefined && model.contextWindow !== undefined)
|
|
@@ -331,19 +343,36 @@ export function buildCatalog(config) {
|
|
|
331
343
|
const capabilities = new Set(matchesOf(config, { model: name }).flatMap(({ model }) => model.capabilities ?? DEFAULT_CAPABILITIES));
|
|
332
344
|
return OPERATIONS.filter((operation) => capabilities.has(operation));
|
|
333
345
|
};
|
|
346
|
+
// The protocols a profile's candidates are served through, in the order the profile tries them.
|
|
347
|
+
const protocols = (policy, names) => {
|
|
348
|
+
const source = sourceOf(defaults, policy, {}, undefined);
|
|
349
|
+
const refs = "candidates" in source ? source.candidates.flat() : names.map((model) => ({ model }));
|
|
350
|
+
return [...new Set(refs.flatMap((ref) => matchesOf(config, ref).map(({ provider }) => presetOf(provider).protocol)))];
|
|
351
|
+
};
|
|
334
352
|
return {
|
|
335
353
|
defaults: namesOf({}, defaults),
|
|
354
|
+
providers: config.providers.map((provider) => ({ id: provider.id, kind: presetOf(provider).kind, protocol: presetOf(provider).protocol })),
|
|
336
355
|
profiles: Object.entries(config.profiles ?? {}).map(([name, policy]) => {
|
|
337
356
|
const named = namesOf(defaults, policy);
|
|
338
|
-
|
|
357
|
+
const dimensions = named.model === undefined ? undefined : matchesOf(config, { model: named.model })[0]?.model.dimensions;
|
|
358
|
+
return {
|
|
359
|
+
name,
|
|
360
|
+
...named,
|
|
361
|
+
pinned: policy.pinned === true,
|
|
362
|
+
serves: serves(named.model),
|
|
363
|
+
protocols: protocols(policy, [...(named.model === undefined ? [] : [named.model]), ...named.fallbacks]),
|
|
364
|
+
...(dimensions === undefined ? {} : { dimensions }),
|
|
365
|
+
};
|
|
339
366
|
}),
|
|
340
367
|
models: [...models].map(([id, entry]) => ({
|
|
341
368
|
id,
|
|
342
369
|
aliases: [...entry.aliases],
|
|
343
370
|
providers: entry.providers,
|
|
371
|
+
protocols: [...entry.protocols],
|
|
344
372
|
capabilities: [...entry.capabilities],
|
|
345
373
|
...(entry.contextWindow === undefined ? {} : { contextWindow: entry.contextWindow }),
|
|
346
374
|
...(entry.cost === undefined ? {} : { cost: entry.cost }),
|
|
375
|
+
...(entry.dimensions === undefined ? {} : { dimensions: entry.dimensions }),
|
|
347
376
|
})),
|
|
348
377
|
};
|
|
349
378
|
}
|
package/dist/fake.d.ts
CHANGED
|
@@ -22,6 +22,20 @@ export interface FakeToolCall {
|
|
|
22
22
|
readonly arguments: string;
|
|
23
23
|
readonly argumentChunks?: readonly string[];
|
|
24
24
|
}
|
|
25
|
+
/** One thinking block a scripted reply opens with; its text is empty unless given, as when the provider hides it. */
|
|
26
|
+
export interface FakeThinking {
|
|
27
|
+
readonly text?: string;
|
|
28
|
+
/** The opaque signature an Anthropic thinking block carries. Default `sig_fake`. */
|
|
29
|
+
readonly signature?: string;
|
|
30
|
+
}
|
|
31
|
+
/** How much of a scripted reply's prompt the provider's cache served or stored; each is a part of `usage.prompt_tokens`. */
|
|
32
|
+
export interface FakeCacheUsage {
|
|
33
|
+
readonly read?: number;
|
|
34
|
+
/** Tokens written, every lifetime together. */
|
|
35
|
+
readonly write?: number;
|
|
36
|
+
/** The part of `write` stored for an hour. */
|
|
37
|
+
readonly write1h?: number;
|
|
38
|
+
}
|
|
25
39
|
/** One scripted answer. Everything is optional: the default is a 200 chat reply `ok`. */
|
|
26
40
|
export interface FakeReply {
|
|
27
41
|
readonly status?: number;
|
|
@@ -34,7 +48,13 @@ export interface FakeReply {
|
|
|
34
48
|
readonly text?: string;
|
|
35
49
|
/** Tool calls the reply makes (OpenAI `tool_calls`, Anthropic `tool_use` blocks). */
|
|
36
50
|
readonly toolCalls?: readonly FakeToolCall[];
|
|
51
|
+
/** Thinking before the text: Anthropic `thinking` blocks with a signature, OpenAI `reasoning_content` for their text. */
|
|
52
|
+
readonly thinking?: readonly FakeThinking[];
|
|
37
53
|
readonly usage?: Usage;
|
|
54
|
+
/** Cache reads and writes, reported in the protocol's own usage shape. */
|
|
55
|
+
readonly cache?: FakeCacheUsage;
|
|
56
|
+
/** Thinking tokens, reported in the protocol's own usage shape. */
|
|
57
|
+
readonly reasoningTokens?: number;
|
|
38
58
|
/** The vector of every embedding. Default `[0.1, 0.2, 0.3]`. */
|
|
39
59
|
readonly embedding?: readonly number[];
|
|
40
60
|
/** Replaces the whole JSON body. */
|
|
@@ -62,6 +82,13 @@ export interface FakeCall {
|
|
|
62
82
|
readonly protocol: "openai" | "anthropic";
|
|
63
83
|
readonly headers: Readonly<Record<string, string>>;
|
|
64
84
|
readonly body: Readonly<Record<string, unknown>>;
|
|
85
|
+
/**
|
|
86
|
+
* Where the request marks cache breakpoints, in request order: `top` (automatic caching), `system.<i>`, `tools.<i>`,
|
|
87
|
+
* `messages.<i>` (on a message) and `messages.<i>.content.<j>` (on a part or block). The same in either protocol.
|
|
88
|
+
*/
|
|
89
|
+
readonly cacheBreakpoints: readonly string[];
|
|
90
|
+
/** The id its answer carries, one per request (`msg_fake_<n>`, `fake-<n>`): a client that keys messages by id sees each once. */
|
|
91
|
+
readonly answerId: string;
|
|
65
92
|
}
|
|
66
93
|
/** A scriptable provider that answers in process, OpenAI-compatible and Anthropic Messages alike, for consumers' tests. */
|
|
67
94
|
export interface FakeProvider {
|