llmshim 0.4.0__tar.gz → 0.6.0__tar.gz

{llmshim-0.4.0 → llmshim-0.6.0}/CLAUDE.md

@@ -53,6 +53,35 @@ refresh in a completion; hold a snapshot for decisions that must agree.
 normalized responses and usage chunks, logs, and proxy usage. Native token
 fields remain readable. `src/usage.rs` owns extraction; the streaming client
 merges Anthropic's start/delta usage before normalizing terminal counts.
+`usage.uncached_input_tokens` joins them: the providers disagree on whether
+their prompt total already includes the cache read (Anthropic's excludes it,
+OpenAI/Chat Completions/Gemini include it), and only the transport boundary
+still knows which convention the body used. The convention is read off the
+cache-read field that actually matched, never a provider-name table.
+
+### USD cost accounting
+
+`src/cost.rs` is the only thing that multiplies a catalog `Cost` (USD per
+million tokens) by those counters. Input is charged on `uncached_input_tokens`
+so a cached prompt is never billed twice; cache reads and writes are charged at
+their own rates; reasoning tokens are already inside `completion_tokens`.
+**An absent price is `None`, never `0.0`** — a positive count in a bucket with
+no rate poisons the whole total rather than producing a partial sum that reads
+as a complete one. `client.rs` stamps `usage.cost_usd` at the transport
+boundary (the last place that knows the dispatch target) for completions and
+for whichever stream chunk carries usage; `log.rs`, `proxy::types::Usage`, both
+native facades and the four bundled clients carry it through as a nullable
+field. `null` means unknown, not free.
+
+`src/gateway/quota.rs` adds a per-identity dollar cap beside the RPM/TPM
+buckets: `budget_usd` + `budget_window_secs` on an `Identity`, checked before
+dispatch and charged after (cost is only knowable once a response exists, so
+one in-flight request can overshoot). Windows tumble rather than slide, because
+the fleet-wide store is one counter per window. `SpendCap::with_store` takes the
+Redis-backed `DistributedGateway` in distributed mode so `$100/day` means one
+hundred dollars fleet-wide, not per replica. **A response the catalog cannot
+price is not charged** — recording zero would let an unpriced model run forever
+under a budget; `cost_usd: null` is the signal that a price is missing.
 The native Chat Completions streams must use their own parser in the client;
 passing them to the Responses parser silently drops all events.
 
@@ -189,6 +218,49 @@ remain provider errors. Callers must configure the final URL directly.
 
 `FallbackConfig` defines an ordered list of models to try. On retryable errors (429, 500, 502, 503, 529), retries with exponential backoff then falls through to the next model. `completion_with_fallback()` is the top-level API. The proxy supports this via `"fallback": ["model1", "model2"]` in the request body.
 
+### Provider health (`src/breaker.rs`, `src/proxy/health.rs`)
+
+**Health is not rate-limit backoff.** The token buckets already slow a provider
+down after a 429 — a 429 means the provider is alive and asking for less. The
+breaker counts what retrying cannot fix: 5xx (500/502/503/504/529) and
+transport failures. Adapted from `rcode-provider`'s `ProviderBreaker`, which we
+own: sliding failure window, open state, and a single half-open probe admitted
+after the cooldown. Config: `LLMSHIM_BREAKER_WINDOW_SECS` (60),
+`LLMSHIM_BREAKER_TRIP_THRESHOLD` (3; `0` disables), `LLMSHIM_BREAKER_COOLDOWN_SECS` (30).
+
+The breaker hangs on the `Router` (`Router::breaker()` / `with_breaker`). Every
+dispatch path *observes* outcomes so health accrues from ordinary traffic; only
+`fallback.rs` *refuses*, and it checks before every attempt rather than once per
+chain entry — the attempt that opens a circuit is usually the chain's own, so a
+per-entry check would still retry into a target it just watched die. A single-target call is still dispatched:
+with no alternative, refusing would only convert an upstream failure into a
+local one. `proxy::health::build_breaker()` attaches a Redis-coordinated
+`SharedHealth` (failure ZSET + open marker + `SET NX` probe) when
+`LLMSHIM_REDIS_URL` is set and `redis-coordination` is compiled in, mirroring
+`build_limiter`. Every shared operation **fails open**.
+
+### Named routes (`src/config.rs`, `src/router.rs`)
+
+A caller-defined name maps to a model plus request settings:
+
+```toml
+[routes.compaction]
+model = "anthropic/claude-haiku-4-5-20251001"
+reasoning_effort = "low"
+max_tokens = 4096
+```
+
+Addressed as `"model": "route/compaction"`, so a route travels through the
+existing `provider/model` grammar — an OpenAI SDK, the CLI and the proxy's
+admission control all handle it without learning a new field. `resolve_key`
+resolves the indirection, so rate limiting never sees an unrecognized string.
+
+**llmshim must not learn harness vocabulary.** The name is opaque: a harness may
+call a route `compaction`, `advisor` or `webSearch`, and llmshim only knows it
+maps to a model. Route settings are defaults — a per-request key always wins —
+and an unknown name is a 400, never a silent fall back to the default model.
+Routes do not chain.
+
 ### Vision (`src/vision.rs`)
 
 Image content blocks are translated between providers automatically. Users can send images in any format (OpenAI `image_url`, Anthropic `image`, Gemini `inline_data`) and the correct provider sees its native format. Base64 data URIs and plain URLs are both handled. Gemini falls back to a text placeholder for URL images (only supports `inline_data`).
@@ -303,7 +375,13 @@ HTTP proxy with our own API spec (not OpenAI-compatible). Built on axum.
 Endpoints:
 - `POST /v1/chat` — non-streaming (or streaming if `stream: true`)
 - `POST /v1/chat/stream` — always SSE streaming with typed events (`content`, `reasoning`, `tool_call`, `usage`, `done`, `error`)
-- `GET /v1/models` — list available models (filtered to configured providers)
+- `GET /v1/models` — list available models (filtered to configured providers).
+  Serves two audiences from one body: an OpenAI SDK reads the `object: "list"` /
+  `data[]` envelope (so `client.models.list()` works unmodified), llmshim's own
+  clients read `models[]`. Both issue the same request, so there is no path to
+  split on — the union *is* the split. `data[].id` is the routing id, requestable
+  back as `model`. Built once in `proxy::convert::models_response`, shared with
+  the gateway.
 - `GET /health` — health check with provider list
 
 Request format uses `config` for provider-agnostic settings and `provider_config` for raw passthrough. OpenAPI 3.1 spec at `api/openapi.yaml`.
@@ -444,6 +522,11 @@ and native endpoints. Keep display messages readable and source type/code/param
 metadata separate; do not move this logic back into a wire-only formatter.
 JSON responses carry native metadata in response extensions, and SSE errors carry
 an optional structured error object so native rendering remains lossless.
+`n > 1` is refused rather than emulated: the OpenAI backend is the Responses
+API (no `n`), Anthropic Messages and Gemini have no `n` either, and the
+single-message proxy projects choice zero. The refusal is a correctly shaped
+`{"error":{type,message,param:"n",code:"unsupported_parameter"}}`, carried
+through `normalize_error` so both facades render it natively.
 Tests: `unit_wire`, gateway `http::native_tests`. Use a temporary receipt directory
 in tests; never put real signatures, credentials, or conversations in fixtures.
 

{llmshim-0.4.0 → llmshim-0.6.0}/Cargo.lock

@@ -1180,7 +1180,7 @@ checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77"
 
 [[package]]
 name = "llmshim"
-version = "0.4.0"
+version = "0.6.0"
 dependencies = [
  "async-stream",
  "async-trait",

{llmshim-0.4.0 → llmshim-0.6.0}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "llmshim"
-version = "0.4.0"
+version = "0.6.0"
 edition = "2021"
 description = "Blazing fast LLM API translation layer in pure Rust"
 license = "MIT OR Apache-2.0"

{llmshim-0.4.0 → llmshim-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: llmshim
-Version: 0.4.0
+Version: 0.6.0
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License

{llmshim-0.4.0 → llmshim-0.6.0}/docs/src/concepts/routing.md

@@ -70,11 +70,57 @@ perform the second lookup.
 Aliases are not currently configurable through the CLI, config file, proxy
 API, or language clients.
 
+## Named routes
+
+An alias renames a model. A **named route** goes further: it maps a
+caller-defined name to a model *plus* request settings, configured in
+`~/.llmshim/config.toml`.
+
+```toml
+[routes.compaction]
+model = "anthropic/claude-haiku-4-5-20251001"
+reasoning_effort = "low"
+max_tokens = 4096
+```
+
+Address it as a model:
+
+```json
+{"model": "route/compaction", "messages": [{"role": "user", "content": "…"}]}
+```
+
+Because it reuses the `provider/model` grammar, a route works everywhere a
+model address does — the Rust API, the CLI, the proxy, the native endpoints and
+an unmodified OpenAI SDK.
+
+The name is **opaque to llmshim**. A harness may call a route `compaction`,
+`advisor` or `webSearch`; llmshim never interprets it and has no built-in role
+vocabulary. The harness decides what a name means; llmshim provides only the
+mechanism.
+
+Three rules:
+
+- **Settings are defaults.** A key the request already carries wins, so a
+  caller can pick a route and still raise `reasoning_effort` for one call.
+- **An unknown name is an error** (HTTP 400), never a silent fall back to a
+  default model.
+- **Routes do not chain.** A route's `model` may not be another `route/…`.
+
+A Rust application can register routes directly:
+
+```rust
+use llmshim::config::Route;
+
+let router = llmshim::router::Router::new()
+    .route("compaction", Route { model: "anthropic/claude-haiku-4-5-20251001".into(), ..Default::default() });
+```
+
 ## Environment variables versus `config.toml`
 
 `Router::from_env()` reads provider environment variables such as
 `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, and `XAI_API_KEY`. It
-does not read `~/.llmshim/config.toml` by itself. It also discovers the selected
+does not read API keys from `~/.llmshim/config.toml` by itself; it does read
+that file's `[routes]` table, which has no environment equivalent. It also discovers the selected
 ChatGPT OAuth cache, whose default location is `~/.llmshim/chatgpt/auth.json`.
 
 The CLI and proxy call `llmshim::env::load_all()` before constructing their

{llmshim-0.4.0 → llmshim-0.6.0}/docs/src/guides/caching.md

@@ -46,6 +46,19 @@ Every response and usage event reports `cache_read_tokens` and
 add repeated streaming snapshots together. Zero means no cache tokens were
 reported, not that the request was free.
 
+They are joined by `uncached_input_tokens` — the prompt minus whatever cache
+read the provider already counted inside it. Providers disagree on that:
+Anthropic's `input_tokens` excludes the cache read, while the OpenAI Responses,
+Chat Completions and Gemini prompt totals include it. llmshim resolves the
+disagreement at the transport boundary so one counter means one thing.
+
+`cost_usd` is the USD charged for the response: uncached input, output, cache
+reads and cache writes each at their own catalog rate, so a cached prompt is
+never billed twice. **`null` means the catalog carries no price for the model —
+it never means free.** A model priced for input but not for the cache reads a
+response actually used also yields `null`, rather than a partial sum that would
+read as a complete one.
+
 `ProviderRequest::can_continue_from` compares endpoint, credential headers,
 settings and the full prior input prefix. `include`, `store`, `reasoning`, tool
 schemas and other settings participate. The library sends full stateless

{llmshim-0.4.0 → llmshim-0.6.0}/docs/src/guides/fallbacks.md

@@ -89,7 +89,19 @@ address. A success returns immediately. If every address fails, Rust returns
 `all_failed` error response.
 
 Each fallback model must resolve to a provider registered on the Router. A
-chain can cross providers, but only when their keys are configured.
+chain can cross providers, but only when their keys are configured. A chain
+entry may also be a [named route](../concepts/routing.md#named-routes).
+
+## Open circuits are skipped, not retried
+
+A chain also consults provider health, before every attempt rather than once per
+entry — the attempt that opens a circuit is usually the chain's own. When a
+provider's circuit is open, because of too many `5xx`/transport failures in the
+window, the chain abandons that entry, records `circuit open for provider …`
+among the collected errors, and moves to the next address instead of spending
+the rest of its retry budget on a target it already knows is dead. A `429` never
+opens a circuit; that is rate limiting, handled by backoff. See
+[Scaling and rate limits](../proxy/scaling.md#provider-health) for the knobs.
 
 ## Fallback is non-streaming only
 

{llmshim-0.4.0 → llmshim-0.6.0}/docs/src/proxy/http-api.md

@@ -146,16 +146,28 @@ consumption patterns, see [Streaming](../guides/streaming.md).
 
 ## Models and health
 
-`GET /v1/models` returns the registry entries for configured providers:
+`GET /v1/models` returns the registry entries for configured providers, in two
+shapes from one body:
 
 ```json
 {
+  "object": "list",
+  "data": [
+    {"id": "openai/gpt-5.6-terra", "object": "model", "created": 0, "owned_by": "openai"}
+  ],
   "models": [
     {"id": "openai/gpt-5.6-terra", "provider": "openai", "name": "gpt-5.6-terra"}
   ]
 }
 ```
 
+`object` + `data` is the OpenAI list envelope, so an OpenAI SDK's
+`client.models.list()` works against llmshim unmodified. `models` is llmshim's
+own shape and is unchanged. Both audiences issue the same `GET /v1/models`, so
+there is no second path to split on. A `data[].id` is the routing id and can be
+sent straight back as `model`; `created` is the catalog release date as a Unix
+timestamp, or `0` when unknown.
+
 This is discovery, not an allowlist: an arbitrary provider model ID can still
 be routed explicitly. See [Model discovery](../reference/models.md).
 

{llmshim-0.4.0 → llmshim-0.6.0}/docs/src/proxy/native-apis.md

@@ -33,7 +33,18 @@ its standard message/function shapes and `response_format`; JSON-object mode als
 gets object validation. Generation fields supported by the destination retain
 their ordinary adapter behavior. The facade supports one completion (`n:1`) and
 custom function tools; it rejects provider-hosted tool definitions rather than
-turning them into client-executed functions. Provider-specific endpoints such as
+turning them into client-executed functions. `n > 1` is refused rather than
+emulated — the OpenAI backend is the Responses API, which has no `n`, and
+neither do Anthropic Messages or Gemini; fanning out N requests would change
+the cost, rate-limit footprint and cache behavior of what was asked for. The
+refusal is a properly shaped OpenAI error naming the parameter:
+
+```json
+{"error": {"message": "Unsupported value: 'n' must be 1. …",
+           "type": "invalid_request_error", "param": "n",
+           "code": "unsupported_parameter"}}
+```
+ Provider-specific endpoints such as
 batches, token counting, uploads and Responses are not served by these aliases.
 
 With `stream:true`, text arrives incrementally. Chat Completions emits

{llmshim-0.4.0 → llmshim-0.6.0}/docs/src/proxy/scaling.md

@@ -75,6 +75,63 @@ When neither RPM nor TPM is set, proactive rate limiting is disabled;
 concurrency backpressure still applies. Token permits are estimates based on
 request size and requested output, not provider billing measurements.
 
+## Provider health
+
+Rate limiting and health are different questions. A `429` means the provider is
+alive and asking for less, and the token buckets already slow it down. A
+circuit breaker counts what retrying cannot fix — `500`, `502`, `503`, `504`,
+`529` and transport failures — over a sliding window, opens the circuit at the
+threshold, and admits a single probe after the cooldown.
+
+| Variable | Default | Meaning |
+|---|---:|---|
+| `LLMSHIM_BREAKER_WINDOW_SECS` | `60` | Sliding window over which failures are counted |
+| `LLMSHIM_BREAKER_TRIP_THRESHOLD` | `3` | Failures that open a circuit; `0` disables the breaker |
+| `LLMSHIM_BREAKER_COOLDOWN_SECS` | `30` | Time an open circuit waits before admitting a probe |
+
+Every dispatch path *observes* outcomes, so health accrues from ordinary
+traffic. Only a [fallback chain](../guides/fallbacks.md) *refuses*: it skips a
+provider with an open circuit instead of spending its retry budget on a target
+it already knows is dead. A single-target request is still dispatched — with no
+alternative, refusing would only convert an upstream failure into a local one.
+
+## Spend caps
+
+The experimental gateway enforces a per-identity USD cap beside the RPM/TPM
+buckets. A gateway key's identity may carry `budget_usd` and an optional
+`budget_window_secs` (default one day); over budget is a `429` with
+`Retry-After` set to the window reset.
+
+```json
+{"sk-example": {"tenant": "acme", "tier": 1, "budget_usd": 100, "budget_window_secs": 86400}}
+```
+
+Cost is only knowable after a response, so the cap is checked before dispatch
+and charged after: one in-flight request can overshoot.
+
+A response the catalog cannot price is **not** charged — recording zero would let
+an unpriced model run forever under a budget. So that a cap cannot silently stop
+binding, a request whose target has **no catalog price is refused before it runs**
+when a budget is set:
+
+```
+400 {"error":{"code":"unpriceable_under_budget","param":"model", …}}
+```
+
+It is deliberately not a `429`: retrying never clears it. Three ways forward —
+use a priced model, add a local price override in the catalog, or accept the risk
+explicitly per key:
+
+```json
+{"sk-example": {"tenant": "acme", "budget_usd": 100, "budget_allow_unpriced": true}}
+```
+
+`budget_allow_unpriced` defaults to `false`. With it set, those requests run and
+are not charged, and each one logs a warning and increments
+`llmshim_gateway_unpriced_under_cap_total{provider,model}` — a non-zero counter
+means the budget is not binding for that target. An accepted risk should stay
+measurable rather than become an assumption.
+
 ## One replica or a coordinated fleet
 
 The default buckets are in memory. With `N` replicas, each replica enforces
@@ -88,8 +145,10 @@ cargo install llmshim --features redis-coordination
 LLMSHIM_REDIS_URL=redis://redis.internal:6379 llmshim proxy
 ```
 
-`redis-coordination` includes the `proxy` feature. Redis is used for rate-limit
-coordination; connection pools and concurrency limits remain per process. If
+`redis-coordination` includes the `proxy` feature. Redis coordinates rate-limit
+buckets, provider health and — on the gateway — spend, so a shared limit, a
+dead provider and a dollar cap all mean the same thing on every replica;
+connection pools and concurrency limits remain per process. If
 Redis becomes unavailable at runtime, limiting fails open so requests continue.
 If the Redis client cannot be initialized—or the binary lacks the feature—the
 proxy warns and falls back to in-memory buckets.

{llmshim-0.4.0 → llmshim-0.6.0}/llmshim/types.py

@@ -11,7 +11,7 @@ the module stays compatible with Python 3.9 (no ``typing.NotRequired``).
 
 from __future__ import annotations
 
-from typing import Any, List, Literal, TypedDict, Union
+from typing import Any, List, Literal, Optional, TypedDict, Union
 
 __all__ = [
     "Role",
@@ -190,6 +190,9 @@ class Usage(TypedDict, total=False):
     total_tokens: int
     cache_read_tokens: int
     cache_write_tokens: int
+    #: USD charged for this response. ``None`` means the server could not price
+    #: the model — it never means free.
+    cost_usd: Optional[float]
 
 
 class _ResponseMessageBase(TypedDict):
@@ -285,6 +288,8 @@ class UsageEvent(TypedDict, total=False):
     total_tokens: int
     cache_read_tokens: int
     cache_write_tokens: int
+    #: ``None`` when the server could not price the model, never ``0.0``.
+    cost_usd: Optional[float]
 
 
 class _DoneBase(TypedDict):