llmshim 0.3.4__tar.gz → 0.3.6__tar.gz

{llmshim-0.3.4 → llmshim-0.3.6}/CLAUDE.md

@@ -14,7 +14,7 @@ This is a public crate on crates.io. Do NOT make breaking changes to `pub` items
 
 - **OpenAI:** `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.5-pro`, `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.4-mini`, `gpt-5.4-nano`
 - **Anthropic:** `claude-opus-5`, `claude-opus-4-8`, `claude-sonnet-5`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`
-- **Gemini:** `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-flash-lite`
+- **Gemini:** `gemini-3.8-flash`, `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-flash-lite`
 - **xAI:** `grok-4.6`, `grok-4.5`, `grok-4.3`, `grok-4.20-multi-agent-beta-0309`, `grok-4.20-beta-0309-reasoning`, `grok-4.20-beta-0309-non-reasoning`
 - **OpenRouter:** not enumerated (huge/dynamic catalog) — any `openrouter/<vendor>/<model>` slug routes through, e.g. `openrouter/anthropic/claude-sonnet-4.5`.
 - **vLLM / SGLang:** not enumerated (self-hosted) — any `vllm/<served-model>` or `sglang/<served-model>` routes through to the configured server, e.g. `sglang/Qwen/Qwen3.6-35B-A3B-FP8`.
@@ -58,12 +58,23 @@ llmshim::completion(router, request)
 
 Every provider implements: `transform_request`, `transform_response`, `transform_stream_chunk`.
 
+Non-streaming OpenAI/xAI Responses, Gemini and Anthropic transformations require
+supported terminal metadata. Do not turn absent or unrecognized status into
+`finish_reason: "stop"`. Rejected metadata returns a fixed 502 provider error
+without copying provider-supplied status/content into the diagnostic. Tests
+live in `tests/support/completion_status.rs`, included by `unit_openai`.
+Streaming status handling is separate and is not changed by this policy.
+
 ### Router (`src/router.rs`)
 
 Parses `"provider/model"` strings by splitting on the **first** `/` only, so an OpenRouter slug's internal slash survives (`openrouter/anthropic/claude-sonnet-4.5` → provider `openrouter`, model `anthropic/claude-sonnet-4.5`). Auto-infers provider from prefix (`gpt*`/`o*` → openai, `claude*` → anthropic, `gemini*` → gemini, `grok*` → xai); **OpenRouter, vLLM, and SGLang have no prefix inference** — their slugs collide with everyone's, so address them explicitly (`openrouter/…`, `vllm/…`, `sglang/…`); the first-slash split also preserves HF-style served-model slugs (`vllm/meta-llama/Llama-3.1-8B-Instruct`). Supports aliases. `Router::from_env()` reads API-key env vars, plus `VLLM_BASE_URL` / `SGLANG_BASE_URL` (+ optional `*_API_KEY`) for the self-hosted providers.
 
 ### HTTP Client (`src/client.rs`)
 
+Automatic redirects are disabled on the shared client. Keep prompts and
+provider-specific credential headers at the configured endpoint; 3xx responses
+remain provider errors. Callers must configure the final URL directly.
+
 `ShimClient` with shared connection pool (`LazyLock`), HTTP/2, gzip/brotli/zstd compression, TCP keepalive + nodelay. Automatic retry (3 attempts by default) on transport errors and 429/500/502/503/504/529 status codes. This is the **reactive** layer: on a retryable *response* it honors the server's `Retry-After` header (integer seconds or HTTP-date) and provider reset hints (OpenAI `x-ratelimit-reset-*`, Anthropic `anthropic-ratelimit-*-reset`), clamped to a cap and nudged with a little jitter; when there's no server hint (or a transport error) it falls back to full-jitter exponential backoff (uniform in `[0, min(cap, base·2^attempt)]`) to avoid a thundering herd. Tunable via `LLMSHIM_MAX_RETRIES` and `LLMSHIM_MAX_BACKOFF_SECS`. `warmup()` pre-establishes TCP+TLS connections. `SseStream` buffers bytes, extracts `data:` lines, routes through provider's `transform_stream_chunk`.
 
 ### Fallback chains (`src/fallback.rs`)
@@ -147,6 +158,30 @@ Env config (all optional, safe defaults; when no RPM/TPM limits are set the limi
 
 Topologies: **sidecar / zero-infra** (default in-memory limiter — set limits to `global / N` when running N replicas) vs. **Redis-coordinated fleet** (one shared global limit across all replicas).
 
+### Experimental: priority-queue gateway (`src/gateway/`, feature `gateway`)
+
+An experimental scheduler that inverts the proxy's admission model: instead of *rejecting* when a provider's token bucket is empty, it **enqueues** each request into a per-provider priority queue and dispatches when capacity frees — ordered by **priority tier** (paying customer > free) then **FIFO** within a tier. Built for a fleet doing thousands of req/s that can't fire every call the instant it arrives. `gateway = ["proxy"]` (reuses the `RateLimiter` token buckets, `Backpressure`, and the proxy's request/response converters — `convert`/`error` are `pub(crate)` for this).
+
+- **`Scheduler`** (`src/gateway/mod.rs`) owns one lane (queue + dispatcher task) per provider, so a rate-limited OpenAI queue never blocks a ready Anthropic one. `submit()` (unary) / `submit_stream()` (SSE) enqueue + await; a client disconnect / `max_wait` timeout cancels the queued job so it never burns a token.
+- **Dispatcher** is event/timer-driven: pops the highest-priority job, acquires a concurrency slot *then* a rate token (so a saturated semaphore never wastes a token), and on a rate-limit miss **requeues** (preserving priority) and sleeps for exactly the `RetryAfter` the limiter reports — waking early on new work. No busy-wait, no `RateLimiter` changes. `max_wait` bounds **queue residence only** (a `started` signal releases it at dispatch), never the upstream call.
+- **Fairness/aging** (`InMemoryQueue`): per-tier FIFO deques; dequeue picks the front with the highest *effective* priority = `tier + min(max_boost, wait/aging_step)`, so a starved low tier eventually overtakes a high-tier flood. Defaults (`aging_step` 5s, `max_boost` 16) keep normal load strictly priority-then-FIFO. `O(#tiers)` dequeue.
+- **Streaming** (`submit_stream`): a `Delivery::Stream` job hands back an `mpsc` receiver once the upstream opens; the dispatcher forwards chunks and holds the concurrency permit for the whole stream (freed when it ends / the client disconnects). HTTP `POST /v1/chat/stream` (and `stream:true`) bridge it to SSE.
+- **`RequestQueue`** trait is the in-process pluggable backend (in-memory default). Cross-process is a *separate* seam, not this trait (a `Job` holds a `oneshot`).
+- **Distributed** (`src/gateway/distributed.rs`, feature `gateway-redis` = `gateway` + `redis-coordination`): a fleet shares a Redis **ZSET priority queue** per provider and a Redis **pub/sub response bus** keyed by request-id, so any instance dispatches any job and routes the result back to the origin's HTTP connection. Reuses the shared `RedisRateLimiter` for fleet-wide rate coordination. `llmshim gateway` auto-selects distributed mode when `LLMSHIM_REDIS_URL` is set and the binary has `gateway-redis`. Supports **unary + streaming** (typed `BusMessage`s over the channel). Full-parity with the in-memory lane:
+  - **Aging** via a *virtual-deadline* score `enqueue_ms − tier·aging_step` popped with `ZPOPMIN` — one static score gives priority, FIFO, and anti-starvation aging with no re-scoring.
+  - **At-least-once** via an atomic **lease** (Lua: `ZPOPMIN` queue → `processing` ZSET with a visibility deadline + recorded score) + ack/release + a background **reaper** (`reap_once`, also public for external cron) that requeues expired leases; streams refresh their lease as they run. A redelivered job may run twice (idempotent upstream calls). Env: `LLMSHIM_GATEWAY_LEASE_TIMEOUT_MS`, `LLMSHIM_GATEWAY_REQUEST_TIMEOUT_MS`.
+- **HTTP** (`src/gateway/http.rs`): serves the proxy's `POST /v1/chat` + `/v1/chat/stream` contract routed through the scheduler; tier from the **`x-llmshim-priority`** header (uint, default 0). `RealDispatch` calls `completion_with_logger`/`stream`, mapping an upstream 429 → bucket penalty. Run: `llmshim gateway` (needs `--features gateway`, or `gateway-redis` for a fleet). Env: `LLMSHIM_GATEWAY_MAX_WAIT_MS`, `LLMSHIM_GATEWAY_QUEUE_DEPTH`, `LLMSHIM_GATEWAY_MAX_CONCURRENCY`, `LLMSHIM_GATEWAY_AGING_STEP_MS`, `LLMSHIM_GATEWAY_MAX_BOOST`, `LLMSHIM_GATEWAY_REQUEST_TIMEOUT_MS`, `LLMSHIM_GATEWAY_LEASE_TIMEOUT_MS`, `LLMSHIM_GATEWAY_MAX_ATTEMPTS`, `LLMSHIM_GATEWAY_IDEMPOTENCY_TTL_SECS`, plus all the proxy rate-limit vars.
+
+#### Production hardening (gateway)
+
+- **Auth + tier from identity** (`src/gateway/auth.rs`): set `LLMSHIM_GATEWAY_KEYS_FILE` to a JSON map `{ "<api-key>": {"tenant","tier","rpm","tpm"} }`. Then `Authorization: Bearer <key>` is **required** and tier/tenant come from the key — the client `x-llmshim-priority` header is ignored (closes the queue-jump exploit); missing/invalid key → 401. Unset ⇒ open dev mode (header trusted, tenant `anonymous`).
+- **Per-tenant quotas** (`src/gateway/quota.rs`): per-`(tenant,provider)` RPM/TPM token buckets from the caller's identity, enforced in the HTTP layer (proactive 429 + `Retry-After`) on top of the global provider limits — one tenant can't monopolize shared capacity.
+- **Idempotency** (`src/gateway/idempotency.rs`): `Idempotency-Key` header → cache-after-completion so a client retry returns the first result (no second billed call); in-memory for local, Redis for the fleet. Distributed at-least-once also has a **done-marker** (a completed job that gets redelivered is skipped) and a **dead-letter queue** after `max_attempts` (poison-job guard).
+- **Observability**: `GET /metrics` (Prometheus, `src/gateway/metrics.rs`, dependency-free) — requests/dispatched/rejected counters, in-flight + queue-depth gauges, queue-wait + upstream-latency histograms; `GET /ready` (503 if Redis is down in distributed mode); `GET /v1/gateway/stats` (queue depths, dead-letter counts); every response carries `x-request-id`.
+- **Graceful shutdown**: `gateway`/`proxy` drain in-flight on SIGTERM/Ctrl-C.
+- **Load test**: `cargo run --release --features gateway --example gateway_loadtest` (throughput + zero-loss + priority-under-load asserts). **Deploy**: `Dockerfile` builds `--features gateway` by default (`--build-arg FEATURES=gateway-redis` for a fleet); `docker run … llmshim gateway`.
+- Intentionally **not** done: per-tier (as opposed to per-tenant) queue-depth caps — the global depth cap + per-tenant quotas cover it; single-flight idempotency for *concurrent* same-key requests (only cache-after-completion).
+
 ## Client libraries (`clients/`)
 
 Thin clients that speak the proxy's HTTP API. They are faithful to the OpenAPI contract in `api/openapi.yaml` — when you change the proxy's request/response shapes, update that spec and keep the clients in sync. All publish in lockstep with the crate version on every release.

{llmshim-0.3.4 → llmshim-0.3.6}/Cargo.lock

@@ -958,7 +958,7 @@ checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77"
 
 [[package]]
 name = "llmshim"
-version = "0.3.4"
+version = "0.3.6"
 dependencies = [
  "async-stream",
  "async-trait",

{llmshim-0.3.4 → llmshim-0.3.6}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "llmshim"
-version = "0.3.4"
+version = "0.3.6"
 edition = "2021"
 description = "Blazing fast LLM API translation layer in pure Rust"
 license = "MIT OR Apache-2.0"
@@ -8,12 +8,16 @@ repository = "https://github.com/sanjay920/llmshim"
 homepage = "https://github.com/sanjay920/llmshim"
 keywords = ["llm", "openai", "anthropic", "gemini", "ai"]
 categories = ["api-bindings", "web-programming"]
-exclude = ["clients/", "api/", ".claude/", "Dockerfile", ".dockerignore"]
+exclude = ["clients/", "api/", ".claude/", "Dockerfile", ".dockerignore", "gateway-test-ui.html"]
 readme = "README.md"
 
 [features]
 default = []
 proxy = ["dep:axum", "dep:tower", "dep:tower-http", "dep:async-stream", "dep:async-trait"]
+# Experimental priority-queue gateway. Reuses the proxy's RateLimiter/Backpressure.
+gateway = ["proxy"]
+# Gateway + Redis-backed distributed queue & response bus (cross-instance fleet).
+gateway-redis = ["gateway", "redis-coordination"]
 # Opt-in distributed rate-limit coordination via Redis. Keeps the default proxy
 # binary lean — redis is only pulled in when this feature is explicitly enabled.
 redis-coordination = ["proxy", "dep:redis"]
@@ -67,3 +71,8 @@ path = "benchmarks/bench.rs"
 name = "loadtest"
 path = "benchmarks/loadtest.rs"
 required-features = ["proxy"]
+
+[[example]]
+name = "gateway_loadtest"
+path = "benchmarks/gateway_loadtest.rs"
+required-features = ["gateway"]

{llmshim-0.3.4 → llmshim-0.3.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: llmshim
-Version: 0.3.4
+Version: 0.3.6
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License

{llmshim-0.3.4 → llmshim-0.3.6}/README.md

@@ -76,8 +76,23 @@ llmshim configure          # interactive prompt
 
 ---
 
+## Endpoint redirects
+
+The shared HTTP client does not follow redirects. Configure the final API URL
+directly: a 3xx response is returned as a provider error instead of forwarding
+the prompt and provider-specific credential headers to another endpoint.
+This applies to both streaming and non-streaming requests.
+
 ## Use it from Rust
 
+Non-streaming OpenAI Responses, xAI Responses, Gemini, and Anthropic results
+require supported terminal metadata. Missing, malformed, or unsupported
+statuses return a `ProviderError` (502) with a fixed diagnostic instead of
+being interpreted as successful completion. Known completion, output-limit,
+filtering and Anthropic tool-call mappings remain available. Additional native
+reasons require an explicit mapping; streaming transforms are unchanged.
+Provider mocks should include the native terminal status/stop reason.
+
 ```bash
 cargo add llmshim tokio serde_json
 ```

llmshim-0.3.6/benchmarks/gateway_loadtest.rs

@@ -0,0 +1,203 @@
+//! Gateway load test — drives the in-process priority [`Scheduler`] with tens of
+//! thousands of concurrent mixed-priority requests against a fake (no-network)
+//! dispatcher, so it exercises the scheduler itself, not any provider.
+//!
+//! Run: `cargo run --release --features gateway --example gateway_loadtest`
+//!
+//! Reports throughput + latency percentiles and asserts two invariants:
+//!   1. **Zero loss** under a burst far larger than the concurrency limit.
+//!   2. **Priority holds under load** — high-tier p50 latency ≪ low-tier p50
+//!      when the concurrency slots are saturated.
+//! Exits non-zero if either invariant fails.
+
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use async_trait::async_trait;
+use llmshim::gateway::{Dispatch, DispatchError, GatewayConfig, GatewayRequest, Scheduler};
+use llmshim::proxy::ratelimit::{InMemoryRateLimiter, RateLimitConfig};
+use serde_json::{json, Value};
+
+/// A no-network dispatcher with a fixed simulated latency.
+struct FakeDispatch {
+    latency: Duration,
+    served: Arc<AtomicU64>,
+}
+
+#[async_trait]
+impl Dispatch for FakeDispatch {
+    async fn dispatch(&self, _provider: &str, payload: Value) -> Result<Value, DispatchError> {
+        if !self.latency.is_zero() {
+            tokio::time::sleep(self.latency).await;
+        }
+        self.served.fetch_add(1, Ordering::Relaxed);
+        Ok(payload)
+    }
+}
+
+fn unlimited() -> Arc<InMemoryRateLimiter> {
+    Arc::new(InMemoryRateLimiter::new(RateLimitConfig::default()))
+}
+
+fn percentile(sorted_us: &[u128], p: f64) -> f64 {
+    if sorted_us.is_empty() {
+        return 0.0;
+    }
+    let idx = ((sorted_us.len() as f64 - 1.0) * p).round() as usize;
+    sorted_us[idx] as f64 / 1000.0 // → ms
+}
+
+fn report(name: &str, mut latencies_us: Vec<u128>, wall: Duration) {
+    latencies_us.sort_unstable();
+    let n = latencies_us.len();
+    let rps = n as f64 / wall.as_secs_f64();
+    println!(
+        "{name}: {n} reqs in {:.2}s = {rps:.0} req/s | p50={:.1}ms p95={:.1}ms p99={:.1}ms max={:.1}ms",
+        wall.as_secs_f64(),
+        percentile(&latencies_us, 0.50),
+        percentile(&latencies_us, 0.95),
+        percentile(&latencies_us, 0.99),
+        percentile(&latencies_us, 1.00),
+    );
+}
+
+#[tokio::main(flavor = "multi_thread")]
+async fn main() {
+    let mut failures = 0;
+
+    // ---- Scenario A: throughput + zero loss under a large burst ------------
+    {
+        let served = Arc::new(AtomicU64::new(0));
+        let config = GatewayConfig {
+            max_queue_depth: 500_000,
+            max_concurrency_per_provider: 512,
+            max_wait: Duration::from_secs(60),
+            ..Default::default()
+        };
+        let sched = Scheduler::new(
+            config,
+            unlimited(),
+            Arc::new(FakeDispatch {
+                latency: Duration::from_millis(2),
+                served: served.clone(),
+            }),
+        );
+
+        let n = 20_000u64;
+        let ok = Arc::new(AtomicU64::new(0));
+        let errs = Arc::new(AtomicU64::new(0));
+        let start = Instant::now();
+        let mut handles = Vec::with_capacity(n as usize);
+        for i in 0..n {
+            let s = sched.clone();
+            let ok = ok.clone();
+            let errs = errs.clone();
+            handles.push(tokio::spawn(async move {
+                let t = Instant::now();
+                let r = s
+                    .submit(GatewayRequest {
+                        provider: "bench".into(),
+                        tier: (i % 3) as u8,
+                        permits: 1,
+                        payload: json!({ "id": i }),
+                    })
+                    .await;
+                match r {
+                    Ok(_) => ok.fetch_add(1, Ordering::Relaxed),
+                    Err(_) => errs.fetch_add(1, Ordering::Relaxed),
+                };
+                t.elapsed().as_micros()
+            }));
+        }
+        let mut lat = Vec::with_capacity(n as usize);
+        for h in handles {
+            lat.push(h.await.unwrap());
+        }
+        let wall = start.elapsed();
+        report("throughput", lat, wall);
+        let (ok, errs) = (ok.load(Ordering::Relaxed), errs.load(Ordering::Relaxed));
+        println!(
+            "  ok={ok} errors={errs} served={}",
+            served.load(Ordering::Relaxed)
+        );
+        if errs != 0 || ok != n {
+            eprintln!("  FAIL: expected {n} successful, zero lost — got ok={ok} errs={errs}");
+            failures += 1;
+        } else {
+            println!("  PASS: zero loss under burst");
+        }
+    }
+
+    // ---- Scenario B: priority holds when concurrency is saturated ----------
+    {
+        let served = Arc::new(AtomicU64::new(0));
+        let config = GatewayConfig {
+            max_queue_depth: 100_000,
+            max_concurrency_per_provider: 8, // the bottleneck
+            max_wait: Duration::from_secs(120),
+            ..Default::default()
+        };
+        let sched = Scheduler::new(
+            config,
+            unlimited(),
+            Arc::new(FakeDispatch {
+                latency: Duration::from_millis(25),
+                served: served.clone(),
+            }),
+        );
+
+        let per_tier = 800u64;
+        let low_lat = Arc::new(std::sync::Mutex::new(Vec::<u128>::new()));
+        let high_lat = Arc::new(std::sync::Mutex::new(Vec::<u128>::new()));
+        let start = Instant::now();
+        let mut handles = Vec::new();
+        // Interleave low- and high-tier submissions.
+        for i in 0..per_tier {
+            for (tier, bucket) in [(0u8, &low_lat), (5u8, &high_lat)] {
+                let s = sched.clone();
+                let bucket = bucket.clone();
+                handles.push(tokio::spawn(async move {
+                    let t = Instant::now();
+                    let _ = s
+                        .submit(GatewayRequest {
+                            provider: "bench".into(),
+                            tier,
+                            permits: 1,
+                            payload: json!({ "id": i }),
+                        })
+                        .await;
+                    bucket.lock().unwrap().push(t.elapsed().as_micros());
+                }));
+            }
+        }
+        for h in handles {
+            h.await.unwrap();
+        }
+        let wall = start.elapsed();
+        let low = Arc::try_unwrap(low_lat).unwrap().into_inner().unwrap();
+        let high = Arc::try_unwrap(high_lat).unwrap().into_inner().unwrap();
+        report("priority(low tier)", low.clone(), wall);
+        report("priority(high tier)", high.clone(), wall);
+        let mut lo = low;
+        let mut hi = high;
+        lo.sort_unstable();
+        hi.sort_unstable();
+        let low_p50 = percentile(&lo, 0.50);
+        let high_p50 = percentile(&hi, 0.50);
+        if high_p50 < low_p50 {
+            println!("  PASS: high-tier p50 {high_p50:.1}ms < low-tier p50 {low_p50:.1}ms");
+        } else {
+            eprintln!(
+                "  FAIL: priority not honored — high p50 {high_p50:.1}ms >= low p50 {low_p50:.1}ms"
+            );
+            failures += 1;
+        }
+    }
+
+    if failures > 0 {
+        eprintln!("\n{failures} scenario(s) FAILED");
+        std::process::exit(1);
+    }
+    println!("\nAll load-test invariants held.");
+}

{llmshim-0.3.4 → llmshim-0.3.6}/docs/src/reference/errors.md

@@ -23,6 +23,18 @@ failure can occur after streaming begins.
 
 ## What the shared client retries
 
+Provider requests do not follow HTTP redirects. A `3xx` response returns a
+`ProviderError` with that status; configure the final endpoint URL directly.
+This keeps prompts and provider authentication headers at the configured
+destination for both completion and streaming requests.
+
+Non-streaming native normalizers require supported terminal metadata. Missing,
+malformed, nonterminal or unrecognized status/finish-reason values produce a
+`ProviderError` with status `502` and a fixed diagnostic, rather than labeling
+partial text as a successful `stop`. The diagnostic excludes response text
+and the provider-supplied reason. Known completion, length, filtering and tool
+mappings remain supported. Streaming transformations are unchanged.
+
 The shared provider client automatically retries:
 
 - transport connect, timeout, request, and body failures;

{llmshim-0.3.4 → llmshim-0.3.6}/src/client.rs

@@ -65,6 +65,9 @@ impl ShimClient {
     pub fn new() -> Self {
         Self {
             http: Client::builder()
+                // Prompts and custom provider credentials belong only at the
+                // configured endpoint, never an HTTP Location target.
+                .redirect(reqwest::redirect::Policy::none())
                 .pool_idle_timeout(Duration::from_secs(90))
                 .pool_max_idle_per_host(4)
                 .tcp_keepalive(Duration::from_secs(30))
@@ -670,6 +673,49 @@ mod tests {
 
     // --- Integration (mockito, local only — no provider API calls) ----------
 
+    #[tokio::test]
+    async fn redirects_do_not_forward_prompts_or_credentials() {
+        for status in [301, 302, 303, 307, 308] {
+            for same_origin in [false, true] {
+                let mut origin = mockito::Server::new_async().await;
+                let mut other = mockito::Server::new_async().await;
+                let destination = if same_origin { &mut origin } else { &mut other };
+                let location = format!("{}/moved", destination.url());
+                let forwarded = destination
+                    .mock(if status <= 303 { "GET" } else { "POST" }, "/moved")
+                    .with_status(200)
+                    .with_body("unexpected forwarding")
+                    .expect(0)
+                    .create_async()
+                    .await;
+                let redirect = origin
+                    .mock("POST", "/v1/chat/completions")
+                    .match_header("x-api-key", "test-secret")
+                    .match_body(mockito::Matcher::Json(serde_json::json!({
+                        "messages": [{"role": "user", "content": "private transcript"}]
+                    })))
+                    .with_status(status)
+                    .with_header("location", &location)
+                    .expect(1)
+                    .create_async()
+                    .await;
+                let request = ProviderRequest {
+                    url: format!("{}/v1/chat/completions", origin.url()),
+                    headers: vec![("x-api-key".into(), "test-secret".into())],
+                    body: serde_json::json!({
+                        "messages": [{"role": "user", "content": "private transcript"}]
+                    }),
+                };
+                let result = ShimClient::new().send(&request).await;
+                redirect.assert_async().await;
+                forwarded.assert_async().await;
+                assert!(
+                    matches!(result, Err(ShimError::ProviderError { status: actual, .. }) if actual == status as u16)
+                );
+            }
+        }
+    }
+
     #[tokio::test]
     async fn honors_retry_after_then_succeeds() {
         let mut server = mockito::Server::new_async().await;

llmshim-0.3.6/src/gateway/auth.rs

@@ -0,0 +1,199 @@
+//! Gateway authentication and per-key identity (tenant, tier, per-tenant quotas).
+//!
+//! The priority tier must come from an **authenticated** identity, not a
+//! client-supplied header — otherwise any caller sets `x-llmshim-priority: 255`
+//! and jumps the queue. When an API-key file is configured, the gateway requires
+//! `Authorization: Bearer <key>` and derives tier + tenant from the key (the
+//! header is ignored). When no keys are configured it stays **open** for local
+//! dev and honors the header, so nothing breaks out of the box.
+
+use std::collections::HashMap;
+
+use axum::http::HeaderMap;
+use serde::Deserialize;
+
+fn anonymous() -> String {
+    "anonymous".to_string()
+}
+
+/// Who a request belongs to, resolved from its API key (or the header in open
+/// mode). `rpm`/`tpm` are optional per-tenant limits enforced by the gateway.
+#[derive(Clone, Debug, Deserialize)]
+pub struct Identity {
+    #[serde(default = "anonymous")]
+    pub tenant: String,
+    #[serde(default)]
+    pub tier: u8,
+    #[serde(default)]
+    pub rpm: Option<u32>,
+    #[serde(default)]
+    pub tpm: Option<u32>,
+}
+
+/// Authentication failure — both map to HTTP 401.
+#[derive(Debug)]
+pub enum AuthError {
+    MissingKey,
+    InvalidKey,
+}
+
+/// API-key registry. `Open` = no auth (dev); `Enforced` = Bearer required.
+pub enum KeyStore {
+    Open,
+    Enforced(HashMap<String, Identity>),
+}
+
+impl KeyStore {
+    /// Load from `LLMSHIM_GATEWAY_KEYS_FILE` (a JSON object mapping API key →
+    /// identity). Unset → open mode. A set-but-unreadable/invalid file is a boot
+    /// error and **exits** rather than silently running unauthenticated.
+    pub fn from_env() -> Self {
+        match std::env::var("LLMSHIM_GATEWAY_KEYS_FILE") {
+            Ok(path) if !path.trim().is_empty() => {
+                let data = std::fs::read_to_string(&path).unwrap_or_else(|e| {
+                    eprintln!("gateway: cannot read LLMSHIM_GATEWAY_KEYS_FILE {path}: {e}");
+                    std::process::exit(1);
+                });
+                let map: HashMap<String, Identity> =
+                    serde_json::from_str(&data).unwrap_or_else(|e| {
+                        eprintln!("gateway: invalid keys file {path}: {e}");
+                        std::process::exit(1);
+                    });
+                eprintln!(
+                    "  Auth: ENFORCED ({} API key(s); tier from key, x-llmshim-priority ignored)",
+                    map.len()
+                );
+                KeyStore::Enforced(map)
+            }
+            _ => {
+                eprintln!(
+                    "  Auth: OPEN (no LLMSHIM_GATEWAY_KEYS_FILE; x-llmshim-priority trusted)"
+                );
+                KeyStore::Open
+            }
+        }
+    }
+
+    /// Build an enforced store directly (tests / embedding).
+    pub fn enforced(keys: HashMap<String, Identity>) -> Self {
+        KeyStore::Enforced(keys)
+    }
+
+    /// Whether Bearer auth is required.
+    pub fn is_enforced(&self) -> bool {
+        matches!(self, KeyStore::Enforced(_))
+    }
+
+    /// Resolve the caller's identity. Open mode reads the tier from the client
+    /// header; enforced mode requires a valid Bearer key and takes tier + tenant
+    /// from it (header ignored, so callers can't self-escalate).
+    pub fn identify(&self, headers: &HeaderMap) -> Result<Identity, AuthError> {
+        match self {
+            KeyStore::Open => Ok(Identity {
+                tenant: anonymous(),
+                tier: header_tier(headers),
+                rpm: None,
+                tpm: None,
+            }),
+            KeyStore::Enforced(map) => {
+                let key = bearer_token(headers).ok_or(AuthError::MissingKey)?;
+                map.get(key).cloned().ok_or(AuthError::InvalidKey)
+            }
+        }
+    }
+}
+
+/// Parse the client `x-llmshim-priority` header into a tier (default `0`).
+pub fn header_tier(headers: &HeaderMap) -> u8 {
+    headers
+        .get("x-llmshim-priority")
+        .and_then(|v| v.to_str().ok())
+        .and_then(|s| s.trim().parse::<u8>().ok())
+        .unwrap_or(0)
+}
+
+/// Extract the `Authorization: Bearer <token>` value.
+fn bearer_token(headers: &HeaderMap) -> Option<&str> {
+    let h = headers
+        .get(axum::http::header::AUTHORIZATION)?
+        .to_str()
+        .ok()?;
+    h.strip_prefix("Bearer ")
+        .or_else(|| h.strip_prefix("bearer "))
+        .map(str::trim)
+        .filter(|s| !s.is_empty())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use axum::http::{HeaderMap, HeaderName, HeaderValue};
+
+    fn hdrs(pairs: &[(&str, &str)]) -> HeaderMap {
+        let mut h = HeaderMap::new();
+        for (k, v) in pairs {
+            h.insert(
+                HeaderName::from_bytes(k.as_bytes()).unwrap(),
+                HeaderValue::from_str(v).unwrap(),
+            );
+        }
+        h
+    }
+
+    #[test]
+    fn open_mode_trusts_the_priority_header() {
+        let store = KeyStore::Open;
+        let id = store
+            .identify(&hdrs(&[("x-llmshim-priority", "7")]))
+            .unwrap();
+        assert_eq!(id.tier, 7);
+        assert_eq!(id.tenant, "anonymous");
+    }
+
+    #[test]
+    fn enforced_mode_ignores_the_header_and_uses_the_key() {
+        let mut keys = HashMap::new();
+        keys.insert(
+            "sk-paid".to_string(),
+            Identity {
+                tenant: "acme".into(),
+                tier: 5,
+                rpm: Some(100),
+                tpm: None,
+            },
+        );
+        let store = KeyStore::enforced(keys);
+
+        // A valid key → its tier, regardless of the (spoofed) header.
+        let id = store
+            .identify(&hdrs(&[
+                ("authorization", "Bearer sk-paid"),
+                ("x-llmshim-priority", "255"),
+            ]))
+            .unwrap();
+        assert_eq!(id.tier, 5, "tier must come from the key, not the header");
+        assert_eq!(id.tenant, "acme");
+
+        // Missing / bad key → rejected.
+        assert!(matches!(
+            store.identify(&hdrs(&[("x-llmshim-priority", "255")])),
+            Err(AuthError::MissingKey)
+        ));
+        assert!(matches!(
+            store.identify(&hdrs(&[("authorization", "Bearer nope")])),
+            Err(AuthError::InvalidKey)
+        ));
+    }
+
+    #[test]
+    fn identity_json_defaults() {
+        let id: Identity = serde_json::from_str(r#"{"tenant":"t"}"#).unwrap();
+        assert_eq!(id.tier, 0);
+        assert!(id.rpm.is_none());
+        let full: Identity =
+            serde_json::from_str(r#"{"tenant":"t","tier":3,"rpm":50,"tpm":9000}"#).unwrap();
+        assert_eq!(full.tier, 3);
+        assert_eq!(full.rpm, Some(50));
+        assert_eq!(full.tpm, Some(9000));
+    }
+}