ai-lcr 0.6.1 → 0.6.3

package/CHANGELOG.md

@@ -4,11 +4,77 @@ All notable changes to `ai-lcr` are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/), and the project adheres to
 [Semantic Versioning](https://semver.org/).
 
+## [0.6.3] — 2026-06-11
+
+Caching — both kinds, each off by default and each a pure config flag with no
+service to run. The response cache is the layer Vercel AI Gateway notably
+doesn't offer; ai-lcr does it in-process and folds it into its cost truth.
+
+### Added
+
+- **`createLCR({ cache })`** — exact-match **response cache**. An identical
+  request replays the stored response and calls **no provider at all**: zero
+  latency, `costUsd: 0`. Storage is pluggable with **zero added dependencies**:
+  `cache: true` uses a bundled in-memory store, `cache: myStore` brings your own
+  (Redis / Vercel KV — required for cross-request hits on serverless, where
+  memory isn't shared), `cache: { store?, ttlMs? }` sets a TTL. A hit settles a
+  `CallRecord` with `cacheHit: true` and the avoided cost on its own
+  `cacheHitSavingUsd` line (a caching saving, never folded into routing savings).
+  Empty completions and usage-less results are never cached. New exports:
+  `createMemoryCacheStore`, types `CacheStore` / `CacheOptions` / `CachedCall` /
+  `CachedMeta` / `MemoryCacheOptions`.
+- **`createLCR({ promptCache })`** — automatic provider-side **prompt-cache**
+  breakpoint. Inserts an Anthropic `cache_control` marker on the last system
+  message so the static prompt head bills at the cache-read rate (~0.1× input)
+  on repeats; the model still runs. `true` for the 5-minute window,
+  `{ ttl: "1h" }` for the longer one. Only writes the `anthropic` namespace
+  (ignored by other providers, safe on a mixed chain) and steps aside if you set
+  `cacheControl` yourself. Savings surface via the existing `cachedInputTokens` /
+  `cachedSavingUsd`. New exported type `PromptCacheOptions`.
+- `CallRecord` gains **`cacheHit`** and **`cacheHitSavingUsd`** for response-cache
+  hits.
+
+### Compatibility
+
+- Fully backward compatible. Both `cache` and `promptCache` are **off by
+  default** — unset, routing behaves exactly as before.
+
+## [0.6.2] — 2026-06-11
+
+Circuit breaker for persistently-failing providers. Until now the only recovery
+lever was `resetIntervalMs`, which snaps routing back to the cheapest provider on
+a timer — so a provider that's actually down keeps eating one failed attempt
+every window. The breaker remembers the failure and stops sending it traffic.
+
+### Added
+
+- **`createLCR({ cooldown })`.** A provider that fails `maxFailures` times within
+  `windowMs` is *skipped* for `cooldownMs` instead of being re-probed every
+  request; a single success clears its count. `true` enables defaults (3 / 60s →
+  60s); pass `{ maxFailures, windowMs, cooldownMs }` to tune. New exported type
+  `CooldownOptions`.
+- The breaker only **reorders** each request's attempt list (cooling providers go
+  last), so when every provider is cooling a request still tries them all rather
+  than failing outright — it can never turn a recoverable request into a hard
+  failure.
+
+### Changed
+
+- The routing engine now snapshots a per-request **attempt order** once (cheapest
+  ring with cooling providers moved to the back) and threads it through streaming
+  failover, replacing the previous modular index walk. Behavior is identical when
+  `cooldown` is unset.
+
+### Compatibility
+
+- Fully backward compatible. `cooldown` is **off by default** — with it unset no
+  provider is ever skipped and routing behaves exactly as before.
+
 ## [0.6.1] — 2026-06-11
 
 Zero-config pricing for native-maker routes. Until now every priced provider
 needed a hand-typed `cost: { input, output }`; for a vendor's own API that number
-is just the public list price you could look up. 0.7 bundles those.
+is just the public list price you could look up. 0.6.1 bundles those.
 
 ### Added
 

package/README.md

@@ -171,6 +171,53 @@ Look a price up yourself with `getModelPrice("claude-sonnet-4-6")`. The table is
 2. **Fall through on failure.** On any provider failure — rate limit, 5xx, timeout, a **billing cap** (402 / out-of-credit / quota), *and* a client error like a **400** — it advances to the next provider, streaming-safe. A 400 fails over on purpose: across OpenAI-compatible aggregators a 400 is usually "*this* provider won't take this request" (an unsupported param, a model it hasn't listed, a stricter schema), not a universally-broken request — so the next provider may well serve it. If every provider rejects the request it still fails, surfacing the **original** error so a genuine caller bug stays debuggable. The one failure that never fails over is a deliberate caller cancellation (`AbortSignal`). Pass `shouldRetry: isRetryableError` to `createLCR` to restore the stricter "client errors fail fast" behavior.
 3. **Recover.** After an idle window (`resetIntervalMs`, default 60s) it snaps back to the cheapest provider.
 
+For a provider that's *persistently* down, the timer alone keeps re-probing it — one failed attempt every window. Turn on the **circuit breaker** to stop that:
+
+```ts
+const lcr = createLCR({
+  models: { /* … */ },
+  cooldown: true, // skip a provider that keeps failing, instead of re-probing it
+});
+```
+
+With `cooldown` on, a provider that fails enough times in a window is *skipped* for a cooldown period rather than tried every request — and a single success clears it. Defaults are 3 failures / 60s → 60s cooldown; tune with `cooldown: { maxFailures, windowMs, cooldownMs }`. It only ever **reorders** the attempt list (cooling providers go last), so if *every* provider is cooling a request still tries them all rather than failing outright. Off by default — routing is unchanged unless you opt in.
+
+## Cache
+
+There are two completely different "caches" in LLM land, and ai-lcr does both — each off by default, each a pure config flag with no service to run.
+
+### Skip the call entirely (`cache`) — response cache
+
+When a request is byte-for-byte identical to one already answered, replay the stored response and call **no provider at all**: zero latency, `costUsd: 0`. This is the layer Vercel AI Gateway notably *doesn't* offer.
+
+```ts
+const lcr = createLCR({
+  models: { /* … */ },
+  cache: true, // exact-match response cache (in-memory by default)
+});
+```
+
+Storage is pluggable and ai-lcr ships **zero dependencies** for it:
+
+- **`cache: true`** uses a process-local in-memory store. Real on a long-running server, and useful *within* one serverless invocation (an agent loop repeating a sub-call) — but it does **not** survive across serverless requests, because separate function instances don't share memory.
+- **For cross-request hits on serverless**, bring your own store backed by a shared layer (Upstash Redis, Vercel KV): `cache: myStore`. ai-lcr runs no service of its own — any shared store is yours. A custom store is just `{ get, set }` (see `CacheStore`); `createMemoryCacheStore({ maxEntries })` is exported if you want the bundled one with a cap.
+- **`cache: { store?, ttlMs? }`** sets an entry lifetime.
+
+A hit settles a `CallRecord` with `cacheHit: true`, `costUsd: 0`, and the money it avoided on its own `cacheHitSavingUsd` line — a *caching* saving, kept separate from routing savings (`baselineUsd − costUsd`), never folded in. Empty completions and usage-less results are never cached. One caveat worth stating: caching makes identical requests return identical responses — exactly right for idempotent / `temperature: 0` calls, a behavior change for sampled ones.
+
+### Pay less for the call (`promptCache`) — provider prompt cache
+
+Different mechanism: the model still runs, but the **static head of the prompt** (your system prompt) bills at the provider's cache-read rate (~0.1× input) on repeats. Anthropic needs an explicit `cache_control` marker; OpenAI / Gemini / DeepSeek cache the prefix automatically. `promptCache: true` inserts that marker on the last system message for you:
+
+```ts
+const lcr = createLCR({
+  models: { /* … */ },
+  promptCache: true,          // 5-minute window; { ttl: "1h" } for the longer one
+});
+```
+
+It only writes the `anthropic` provider-options namespace, which every other provider ignores — so it's safe on a mixed chain. And it **steps aside entirely** if you set `cacheControl` yourself anywhere in the prompt. The savings then show up exactly as before: `cachedInputTokens` and `cachedSavingUsd` on the `CallRecord` (see [Cache-aware cost](#see-what-happened-oncall) below).
+
 ## See what happened (`onCall`)
 
 `onError`/`onCost` fire separately and uncorrelated, so a failover is hard to read after the fact. `onCall` gives you **one record per request** — the full chain, the winner, the reason for each failed hop, latency, and cost — and `formatCallRecord` turns it into a one-liner you can scan:
@@ -490,6 +537,7 @@ Two OpenAI-compatible providers, same probe, same day. Cells cover both families
 ## Roadmap
 
 - [x] Own failover engine — cheapest-first routing + streaming-safe fallback, no external routing dependency
+- [x] Circuit breaker (`cooldown`) — skip a persistently-failing provider instead of re-probing it every window
 - [x] Real per-call cost accounting (`onCost`)
 - [x] One correlated record per request with the full failover chain (`onCall` + `formatCallRecord`)
 - [x] Auto cheapest-first ordering (`autoSort`) from per-provider `cost`

package/README.zh-CN.md

@@ -144,6 +144,42 @@ DeepInfra 只承载开源权重——没有第一方 Claude / GPT / Gemini。那
 2. **失败时向下穿透。** 遇到任何 provider 失败——限流、5xx、超时、**额度耗尽**（402 / 欠费 / 余额不足），以及 **400** 这类 client 错误——都会前进到下一个 provider，且对流式安全。400 会 failover 是有意为之：在 OpenAI 兼容聚合层里，400 往往是"*这家* provider 不吃这个请求"（不支持的参数、它没上架这个 model、更严格的 schema），而非请求本身坏了——换一家很可能就能服务。若所有 provider 都拒绝，请求仍会失败，并抛出**第一个**（原始）错误，让真正的调用方 bug 保持可调试。唯一永远不 failover 的是调用方主动取消（`AbortSignal`）。想恢复旧的"client 错误立即失败"行为，给 `createLCR` 传 `shouldRetry: isRetryableError`。
 3. **恢复。** 在一段空闲窗口（`resetIntervalMs`，默认 60s）之后，自动回到最便宜的 provider。
 
+## 缓存
+
+LLM 世界里有两种完全不同的"缓存"，ai-lcr 两种都做——都默认关闭，都只是一个配置开关，**不需要你跑任何服务**。
+
+### 整次调用都省掉（`cache`）—— 响应缓存
+
+当一个请求和之前回答过的一模一样时，直接重放已存的响应、**完全不调用任何 provider**：零延迟、`costUsd: 0`。这正是 Vercel AI Gateway 明显没做的那一层。
+
+```ts
+const lcr = createLCR({
+  models: { /* … */ },
+  cache: true, // 精确匹配响应缓存（默认进程内内存）
+});
+```
+
+存储可插拔，且 ai-lcr 为此**零依赖**：
+
+- **`cache: true`** 用进程内内存。在长驻 server 上是真缓存；在单次 serverless 调用内（比如 agent 循环里重复的子调用）也有用——但它**不会跨 serverless 请求存活**，因为不同函数实例之间不共享内存。
+- **想在 serverless 上跨请求命中**，自带一个由共享层（Upstash Redis、Vercel KV）支撑的 store：`cache: myStore`。ai-lcr 自己不跑任何服务——共享层是你的。自定义 store 就是 `{ get, set }`（见 `CacheStore`）；想要带上限的内置实现，用导出的 `createMemoryCacheStore({ maxEntries })`。
+- **`cache: { store?, ttlMs? }`** 可设置过期时间。
+
+命中会落一条 `CallRecord`：`cacheHit: true`、`costUsd: 0`，并把省下的钱单独记在 `cacheHitSavingUsd` 一行——这是**缓存**省的钱，和路由省的钱（`baselineUsd − costUsd`）分开，绝不混在一起。空回复和无 usage 的结果永不缓存。一个要点：缓存会让相同请求返回相同响应——对幂等 / `temperature: 0` 的调用正好，对采样型调用则是行为改变。
+
+### 让这次调用更便宜（`promptCache`）—— provider 提示缓存
+
+另一套机制：模型照样跑，但**提示的静态开头**（你的 system prompt）在重复时按 provider 的缓存读价（约 0.1× input）计费。Anthropic 需要显式 `cache_control` 标记；OpenAI / Gemini / DeepSeek 自动缓存前缀。`promptCache: true` 帮你在最后一条 system 消息上插入这个标记：
+
+```ts
+const lcr = createLCR({
+  models: { /* … */ },
+  promptCache: true,          // 5 分钟窗口；想要更长用 { ttl: "1h" }
+});
+```
+
+它只写 `anthropic` 这个 provider-options 命名空间，其他 provider 一律忽略——所以在混合链路上也安全。而且只要你在 prompt 里自己设了 `cacheControl`，它就**完全让位**。省下的钱照旧体现在 `CallRecord` 的 `cachedInputTokens` 和 `cachedSavingUsd` 上。
+
 ## 看清每次调用发生了什么（`onCall`）
 
 `onError`/`onCost` 各自独立触发、互不关联，事后很难还原一次 failover 的全貌。`onCall` 给你**每个请求一条记录**——完整的尝试链、最终服务者、每跳失败的原因、延迟和成本；`formatCallRecord` 把它变成一行可扫读的日志：