llmshim 0.3.3__tar.gz → 0.3.5__tar.gz

{llmshim-0.3.3 → llmshim-0.3.5}/CLAUDE.md

@@ -14,7 +14,7 @@ This is a public crate on crates.io. Do NOT make breaking changes to `pub` items
 
 - **OpenAI:** `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.5-pro`, `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.4-mini`, `gpt-5.4-nano`
 - **Anthropic:** `claude-opus-5`, `claude-opus-4-8`, `claude-sonnet-5`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`
-- **Gemini:** `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`
+- **Gemini:** `gemini-3.8-flash`, `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-flash-lite`
 - **xAI:** `grok-4.6`, `grok-4.5`, `grok-4.3`, `grok-4.20-multi-agent-beta-0309`, `grok-4.20-beta-0309-reasoning`, `grok-4.20-beta-0309-non-reasoning`
 - **OpenRouter:** not enumerated (huge/dynamic catalog) — any `openrouter/<vendor>/<model>` slug routes through, e.g. `openrouter/anthropic/claude-sonnet-4.5`.
 - **vLLM / SGLang:** not enumerated (self-hosted) — any `vllm/<served-model>` or `sglang/<served-model>` routes through to the configured server, e.g. `sglang/Qwen/Qwen3.6-35B-A3B-FP8`.
@@ -147,6 +147,30 @@ Env config (all optional, safe defaults; when no RPM/TPM limits are set the limi
 
 Topologies: **sidecar / zero-infra** (default in-memory limiter — set limits to `global / N` when running N replicas) vs. **Redis-coordinated fleet** (one shared global limit across all replicas).
 
+### Experimental: priority-queue gateway (`src/gateway/`, feature `gateway`)
+
+An experimental scheduler that inverts the proxy's admission model: instead of *rejecting* when a provider's token bucket is empty, it **enqueues** each request into a per-provider priority queue and dispatches when capacity frees — ordered by **priority tier** (paying customer > free) then **FIFO** within a tier. Built for a fleet doing thousands of req/s that can't fire every call the instant it arrives. `gateway = ["proxy"]` (reuses the `RateLimiter` token buckets, `Backpressure`, and the proxy's request/response converters — `convert`/`error` are `pub(crate)` for this).
+
+- **`Scheduler`** (`src/gateway/mod.rs`) owns one lane (queue + dispatcher task) per provider, so a rate-limited OpenAI queue never blocks a ready Anthropic one. `submit()` (unary) / `submit_stream()` (SSE) enqueue + await; a client disconnect / `max_wait` timeout cancels the queued job so it never burns a token.
+- **Dispatcher** is event/timer-driven: pops the highest-priority job, acquires a concurrency slot *then* a rate token (so a saturated semaphore never wastes a token), and on a rate-limit miss **requeues** (preserving priority) and sleeps for exactly the `RetryAfter` the limiter reports — waking early on new work. No busy-wait, no `RateLimiter` changes. `max_wait` bounds **queue residence only** (a `started` signal releases it at dispatch), never the upstream call.
+- **Fairness/aging** (`InMemoryQueue`): per-tier FIFO deques; dequeue picks the front with the highest *effective* priority = `tier + min(max_boost, wait/aging_step)`, so a starved low tier eventually overtakes a high-tier flood. Defaults (`aging_step` 5s, `max_boost` 16) keep normal load strictly priority-then-FIFO. `O(#tiers)` dequeue.
+- **Streaming** (`submit_stream`): a `Delivery::Stream` job hands back an `mpsc` receiver once the upstream opens; the dispatcher forwards chunks and holds the concurrency permit for the whole stream (freed when it ends / the client disconnects). HTTP `POST /v1/chat/stream` (and `stream:true`) bridge it to SSE.
+- **`RequestQueue`** trait is the in-process pluggable backend (in-memory default). Cross-process is a *separate* seam, not this trait (a `Job` holds a `oneshot`).
+- **Distributed** (`src/gateway/distributed.rs`, feature `gateway-redis` = `gateway` + `redis-coordination`): a fleet shares a Redis **ZSET priority queue** per provider and a Redis **pub/sub response bus** keyed by request-id, so any instance dispatches any job and routes the result back to the origin's HTTP connection. Reuses the shared `RedisRateLimiter` for fleet-wide rate coordination. `llmshim gateway` auto-selects distributed mode when `LLMSHIM_REDIS_URL` is set and the binary has `gateway-redis`. Supports **unary + streaming** (typed `BusMessage`s over the channel). Full-parity with the in-memory lane:
+  - **Aging** via a *virtual-deadline* score `enqueue_ms − tier·aging_step` popped with `ZPOPMIN` — one static score gives priority, FIFO, and anti-starvation aging with no re-scoring.
+  - **At-least-once** via an atomic **lease** (Lua: `ZPOPMIN` queue → `processing` ZSET with a visibility deadline + recorded score) + ack/release + a background **reaper** (`reap_once`, also public for external cron) that requeues expired leases; streams refresh their lease as they run. A redelivered job may run twice (idempotent upstream calls). Env: `LLMSHIM_GATEWAY_LEASE_TIMEOUT_MS`, `LLMSHIM_GATEWAY_REQUEST_TIMEOUT_MS`.
+- **HTTP** (`src/gateway/http.rs`): serves the proxy's `POST /v1/chat` + `/v1/chat/stream` contract routed through the scheduler; tier from the **`x-llmshim-priority`** header (uint, default 0). `RealDispatch` calls `completion_with_logger`/`stream`, mapping an upstream 429 → bucket penalty. Run: `llmshim gateway` (needs `--features gateway`, or `gateway-redis` for a fleet). Env: `LLMSHIM_GATEWAY_MAX_WAIT_MS`, `LLMSHIM_GATEWAY_QUEUE_DEPTH`, `LLMSHIM_GATEWAY_MAX_CONCURRENCY`, `LLMSHIM_GATEWAY_AGING_STEP_MS`, `LLMSHIM_GATEWAY_MAX_BOOST`, `LLMSHIM_GATEWAY_REQUEST_TIMEOUT_MS`, `LLMSHIM_GATEWAY_LEASE_TIMEOUT_MS`, `LLMSHIM_GATEWAY_MAX_ATTEMPTS`, `LLMSHIM_GATEWAY_IDEMPOTENCY_TTL_SECS`, plus all the proxy rate-limit vars.
+
+#### Production hardening (gateway)
+
+- **Auth + tier from identity** (`src/gateway/auth.rs`): set `LLMSHIM_GATEWAY_KEYS_FILE` to a JSON map `{ "<api-key>": {"tenant","tier","rpm","tpm"} }`. Then `Authorization: Bearer <key>` is **required** and tier/tenant come from the key — the client `x-llmshim-priority` header is ignored (closes the queue-jump exploit); missing/invalid key → 401. Unset ⇒ open dev mode (header trusted, tenant `anonymous`).
+- **Per-tenant quotas** (`src/gateway/quota.rs`): per-`(tenant,provider)` RPM/TPM token buckets from the caller's identity, enforced in the HTTP layer (proactive 429 + `Retry-After`) on top of the global provider limits — one tenant can't monopolize shared capacity.
+- **Idempotency** (`src/gateway/idempotency.rs`): `Idempotency-Key` header → cache-after-completion so a client retry returns the first result (no second billed call); in-memory for local, Redis for the fleet. Distributed at-least-once also has a **done-marker** (a completed job that gets redelivered is skipped) and a **dead-letter queue** after `max_attempts` (poison-job guard).
+- **Observability**: `GET /metrics` (Prometheus, `src/gateway/metrics.rs`, dependency-free) — requests/dispatched/rejected counters, in-flight + queue-depth gauges, queue-wait + upstream-latency histograms; `GET /ready` (503 if Redis is down in distributed mode); `GET /v1/gateway/stats` (queue depths, dead-letter counts); every response carries `x-request-id`.
+- **Graceful shutdown**: `gateway`/`proxy` drain in-flight on SIGTERM/Ctrl-C.
+- **Load test**: `cargo run --release --features gateway --example gateway_loadtest` (throughput + zero-loss + priority-under-load asserts). **Deploy**: `Dockerfile` builds `--features gateway` by default (`--build-arg FEATURES=gateway-redis` for a fleet); `docker run … llmshim gateway`.
+- Intentionally **not** done: per-tier (as opposed to per-tenant) queue-depth caps — the global depth cap + per-tenant quotas cover it; single-flight idempotency for *concurrent* same-key requests (only cache-after-completion).
+
 ## Client libraries (`clients/`)
 
 Thin clients that speak the proxy's HTTP API. They are faithful to the OpenAPI contract in `api/openapi.yaml` — when you change the proxy's request/response shapes, update that spec and keep the clients in sync. All publish in lockstep with the crate version on every release.

{llmshim-0.3.3 → llmshim-0.3.5}/Cargo.lock

@@ -958,7 +958,7 @@ checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77"
 
 [[package]]
 name = "llmshim"
-version = "0.3.3"
+version = "0.3.5"
 dependencies = [
  "async-stream",
  "async-trait",

{llmshim-0.3.3 → llmshim-0.3.5}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "llmshim"
-version = "0.3.3"
+version = "0.3.5"
 edition = "2021"
 description = "Blazing fast LLM API translation layer in pure Rust"
 license = "MIT OR Apache-2.0"
@@ -8,12 +8,16 @@ repository = "https://github.com/sanjay920/llmshim"
 homepage = "https://github.com/sanjay920/llmshim"
 keywords = ["llm", "openai", "anthropic", "gemini", "ai"]
 categories = ["api-bindings", "web-programming"]
-exclude = ["clients/", "api/", ".claude/", "Dockerfile", ".dockerignore"]
+exclude = ["clients/", "api/", ".claude/", "Dockerfile", ".dockerignore", "gateway-test-ui.html"]
 readme = "README.md"
 
 [features]
 default = []
 proxy = ["dep:axum", "dep:tower", "dep:tower-http", "dep:async-stream", "dep:async-trait"]
+# Experimental priority-queue gateway. Reuses the proxy's RateLimiter/Backpressure.
+gateway = ["proxy"]
+# Gateway + Redis-backed distributed queue & response bus (cross-instance fleet).
+gateway-redis = ["gateway", "redis-coordination"]
 # Opt-in distributed rate-limit coordination via Redis. Keeps the default proxy
 # binary lean — redis is only pulled in when this feature is explicitly enabled.
 redis-coordination = ["proxy", "dep:redis"]
@@ -67,3 +71,8 @@ path = "benchmarks/bench.rs"
 name = "loadtest"
 path = "benchmarks/loadtest.rs"
 required-features = ["proxy"]
+
+[[example]]
+name = "gateway_loadtest"
+path = "benchmarks/gateway_loadtest.rs"
+required-features = ["gateway"]

{llmshim-0.3.3 → llmshim-0.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: llmshim
-Version: 0.3.3
+Version: 0.3.5
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -37,11 +37,30 @@ llmshim.configure(
     openai="sk-...",
     gemini="AIza...",
     xai="xai-...",
+    openrouter="sk-or-...",
 )
 ```
 
 Or from the CLI: `llmshim configure`
 
+### Self-hosted servers (vLLM / SGLang)
+
+vLLM and SGLang are configured via **environment variables** (not
+`config.toml`) — the auto-spawned proxy inherits them from your Python
+process. Set them before your first call:
+
+```python
+import os
+
+os.environ["VLLM_BASE_URL"] = "http://localhost:8000/v1"
+os.environ["VLLM_API_KEY"] = "..."       # optional
+os.environ["SGLANG_BASE_URL"] = "http://localhost:30000/v1"
+os.environ["SGLANG_API_KEY"] = "..."     # optional
+```
+
+Then address them via the model string — `vllm/<served-model>` or
+`sglang/<served-model>` (see the model table below).
+
 ## Chat
 
 ```python
@@ -106,17 +125,63 @@ print(f"GPT: {r2['message']['content']}")
 
 ## Reasoning / Thinking
 
+Two provider-agnostic knobs control reasoning; both are clamped to the
+nearest tier the target model supports:
+
+- `reasoning_effort` — `"none"`, `"low"`, `"medium"`, `"high"`, `"xhigh"`, or `"max"`
+- `reasoning_mode` — `"standard"` (default) or `"pro"` (requests substantially
+  more model work; native on OpenAI gpt-5.6/-pro, emulated as an effort bump
+  elsewhere)
+
 ```python
 resp = llmshim.chat(
-    "claude-sonnet-4-6",
+    "claude-sonnet-5",
     "Solve: x^2 - 5x + 6 = 0",
     max_tokens=4000,
     reasoning_effort="high",
+    reasoning_mode="pro",
 )
 print(resp["reasoning"])        # thinking content
 print(resp["message"]["content"])  # answer
 ```
 
+For full native control, bypass the unified mapping with a namespaced
+`provider_config` (see below), e.g.
+`provider_config={"x-anthropic": {"thinking": {"type": "enabled", "budget_tokens": 4000}}}`.
+
+## Provider-Specific Controls (`provider_config`)
+
+`provider_config` merges into the request **root** and carries anything the
+unified `config` doesn't cover. Native provider controls MUST be **namespaced**
+per provider (`x-anthropic`, `x-openai`, `x-gemini`, `x-openrouter`, `x-vllm`,
+`x-sglang`); it also carries the top-level `tools`, `response_format`, and
+`reasoning_summary` keys.
+
+```python
+resp = llmshim.chat(
+    "anthropic/claude-sonnet-5",
+    "Solve this step by step: 17 * 23",
+    max_tokens=4000,
+    provider_config={
+        # native Anthropic extended-thinking control
+        "x-anthropic": {"thinking": {"type": "enabled", "budget_tokens": 4000}},
+        # structured output
+        "response_format": {"type": "json_object"},
+    },
+)
+```
+
+OpenRouter routing preferences use the `x-openrouter` namespace:
+
+```python
+resp = llmshim.chat(
+    "openrouter/anthropic/claude-sonnet-4.5",
+    "Hello",
+    max_tokens=200,
+    provider_config={"x-openrouter": {"provider": {"sort": "throughput"}}},
+)
+```
+
 ## Tool Use / Function Calling
 
 ```python
@@ -147,7 +212,7 @@ resp = llmshim.chat(
     "anthropic/claude-sonnet-4-6",
     "Hello",
     max_tokens=100,
-    fallback=["openai/gpt-5.5", "gemini/gemini-3-flash-preview"],
+    fallback=["openai/gpt-5.6-sol", "gemini/gemini-3.5-flash"],
 )
 ```
 
@@ -220,10 +285,21 @@ billed provider calls; run it only when you deliberately want to hit real APIs.
 
 | Provider | Models |
 |----------|--------|
-| OpenAI | `gpt-5.5`, `gpt-5.5-pro`, `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.4-mini`, `gpt-5.4-nano` |
+| OpenAI | `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.5-pro`, `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.4-mini`, `gpt-5.4-nano` |
 | Anthropic | `claude-opus-5`, `claude-opus-4-8`, `claude-sonnet-5`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001` |
-| Gemini | `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite` |
+| Gemini | `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-flash-lite` |
 | xAI | `grok-4.6`, `grok-4.5`, `grok-4.3`, `grok-4.20-multi-agent-beta-0309`, `grok-4.20-beta-0309-reasoning`, `grok-4.20-beta-0309-non-reasoning` |
 
 Call `llmshim.models()` for the live list filtered to your configured providers.
 
+### OpenRouter & self-hosted (vLLM / SGLang)
+
+These providers are addressed by the model string plus environment variables —
+any model the upstream serves is reachable, so they aren't in the table above.
+
+| Provider | Address as | Env vars | Native controls |
+|----------|-----------|----------|-----------------|
+| OpenRouter | `openrouter/<vendor>/<model>` (e.g. `openrouter/anthropic/claude-sonnet-4.5`) | `OPENROUTER_API_KEY` (or `llmshim.configure(openrouter=...)`) | `provider_config={"x-openrouter": {...}}` (`provider`, `models`, `transforms`) |
+| vLLM | `vllm/<served-model>` | `VLLM_BASE_URL` (+ optional `VLLM_API_KEY`) | `provider_config={"x-vllm": {...}}` |
+| SGLang | `sglang/<served-model>` | `SGLANG_BASE_URL` (+ optional `SGLANG_API_KEY`) | `provider_config={"x-sglang": {...}}` |
+

{llmshim-0.3.3 → llmshim-0.3.5}/README.md

@@ -364,7 +364,7 @@ Standard library only. Full docs: [`clients/ruby/README.md`](clients/ruby/README
 |----------|--------|-------------------|
 | **OpenAI** | `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.5-pro`, `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.4-mini`, `gpt-5.4-nano` | Yes (summaries) |
 | **Anthropic** | `claude-opus-5`, `claude-opus-4-8`, `claude-sonnet-5`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001` | Yes (full thinking) |
-| **Google Gemini** | `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite` | Yes (thought summaries) |
+| **Google Gemini** | `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-flash-lite` | Yes (thought summaries) |
 | **xAI** | `grok-4.6`, `grok-4.5`, `grok-4.3`, `grok-4.20-multi-agent-beta-0309`, `grok-4.20-beta-0309-reasoning`, `grok-4.20-beta-0309-non-reasoning` | No (hidden) |
 
 Use a bare model name (auto-detected by prefix) or an explicit `provider/model` string.

llmshim-0.3.5/benchmarks/gateway_loadtest.rs

@@ -0,0 +1,203 @@
+//! Gateway load test — drives the in-process priority [`Scheduler`] with tens of
+//! thousands of concurrent mixed-priority requests against a fake (no-network)
+//! dispatcher, so it exercises the scheduler itself, not any provider.
+//!
+//! Run: `cargo run --release --features gateway --example gateway_loadtest`
+//!
+//! Reports throughput + latency percentiles and asserts two invariants:
+//!   1. **Zero loss** under a burst far larger than the concurrency limit.
+//!   2. **Priority holds under load** — high-tier p50 latency ≪ low-tier p50
+//!      when the concurrency slots are saturated.
+//! Exits non-zero if either invariant fails.
+
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use async_trait::async_trait;
+use llmshim::gateway::{Dispatch, DispatchError, GatewayConfig, GatewayRequest, Scheduler};
+use llmshim::proxy::ratelimit::{InMemoryRateLimiter, RateLimitConfig};
+use serde_json::{json, Value};
+
+/// A no-network dispatcher with a fixed simulated latency.
+struct FakeDispatch {
+    latency: Duration,
+    served: Arc<AtomicU64>,
+}
+
+#[async_trait]
+impl Dispatch for FakeDispatch {
+    async fn dispatch(&self, _provider: &str, payload: Value) -> Result<Value, DispatchError> {
+        if !self.latency.is_zero() {
+            tokio::time::sleep(self.latency).await;
+        }
+        self.served.fetch_add(1, Ordering::Relaxed);
+        Ok(payload)
+    }
+}
+
+fn unlimited() -> Arc<InMemoryRateLimiter> {
+    Arc::new(InMemoryRateLimiter::new(RateLimitConfig::default()))
+}
+
+fn percentile(sorted_us: &[u128], p: f64) -> f64 {
+    if sorted_us.is_empty() {
+        return 0.0;
+    }
+    let idx = ((sorted_us.len() as f64 - 1.0) * p).round() as usize;
+    sorted_us[idx] as f64 / 1000.0 // → ms
+}
+
+fn report(name: &str, mut latencies_us: Vec<u128>, wall: Duration) {
+    latencies_us.sort_unstable();
+    let n = latencies_us.len();
+    let rps = n as f64 / wall.as_secs_f64();
+    println!(
+        "{name}: {n} reqs in {:.2}s = {rps:.0} req/s | p50={:.1}ms p95={:.1}ms p99={:.1}ms max={:.1}ms",
+        wall.as_secs_f64(),
+        percentile(&latencies_us, 0.50),
+        percentile(&latencies_us, 0.95),
+        percentile(&latencies_us, 0.99),
+        percentile(&latencies_us, 1.00),
+    );
+}
+
+#[tokio::main(flavor = "multi_thread")]
+async fn main() {
+    let mut failures = 0;
+
+    // ---- Scenario A: throughput + zero loss under a large burst ------------
+    {
+        let served = Arc::new(AtomicU64::new(0));
+        let config = GatewayConfig {
+            max_queue_depth: 500_000,
+            max_concurrency_per_provider: 512,
+            max_wait: Duration::from_secs(60),
+            ..Default::default()
+        };
+        let sched = Scheduler::new(
+            config,
+            unlimited(),
+            Arc::new(FakeDispatch {
+                latency: Duration::from_millis(2),
+                served: served.clone(),
+            }),
+        );
+
+        let n = 20_000u64;
+        let ok = Arc::new(AtomicU64::new(0));
+        let errs = Arc::new(AtomicU64::new(0));
+        let start = Instant::now();
+        let mut handles = Vec::with_capacity(n as usize);
+        for i in 0..n {
+            let s = sched.clone();
+            let ok = ok.clone();
+            let errs = errs.clone();
+            handles.push(tokio::spawn(async move {
+                let t = Instant::now();
+                let r = s
+                    .submit(GatewayRequest {
+                        provider: "bench".into(),
+                        tier: (i % 3) as u8,
+                        permits: 1,
+                        payload: json!({ "id": i }),
+                    })
+                    .await;
+                match r {
+                    Ok(_) => ok.fetch_add(1, Ordering::Relaxed),
+                    Err(_) => errs.fetch_add(1, Ordering::Relaxed),
+                };
+                t.elapsed().as_micros()
+            }));
+        }
+        let mut lat = Vec::with_capacity(n as usize);
+        for h in handles {
+            lat.push(h.await.unwrap());
+        }
+        let wall = start.elapsed();
+        report("throughput", lat, wall);
+        let (ok, errs) = (ok.load(Ordering::Relaxed), errs.load(Ordering::Relaxed));
+        println!(
+            "  ok={ok} errors={errs} served={}",
+            served.load(Ordering::Relaxed)
+        );
+        if errs != 0 || ok != n {
+            eprintln!("  FAIL: expected {n} successful, zero lost — got ok={ok} errs={errs}");
+            failures += 1;
+        } else {
+            println!("  PASS: zero loss under burst");
+        }
+    }
+
+    // ---- Scenario B: priority holds when concurrency is saturated ----------
+    {
+        let served = Arc::new(AtomicU64::new(0));
+        let config = GatewayConfig {
+            max_queue_depth: 100_000,
+            max_concurrency_per_provider: 8, // the bottleneck
+            max_wait: Duration::from_secs(120),
+            ..Default::default()
+        };
+        let sched = Scheduler::new(
+            config,
+            unlimited(),
+            Arc::new(FakeDispatch {
+                latency: Duration::from_millis(25),
+                served: served.clone(),
+            }),
+        );
+
+        let per_tier = 800u64;
+        let low_lat = Arc::new(std::sync::Mutex::new(Vec::<u128>::new()));
+        let high_lat = Arc::new(std::sync::Mutex::new(Vec::<u128>::new()));
+        let start = Instant::now();
+        let mut handles = Vec::new();
+        // Interleave low- and high-tier submissions.
+        for i in 0..per_tier {
+            for (tier, bucket) in [(0u8, &low_lat), (5u8, &high_lat)] {
+                let s = sched.clone();
+                let bucket = bucket.clone();
+                handles.push(tokio::spawn(async move {
+                    let t = Instant::now();
+                    let _ = s
+                        .submit(GatewayRequest {
+                            provider: "bench".into(),
+                            tier,
+                            permits: 1,
+                            payload: json!({ "id": i }),
+                        })
+                        .await;
+                    bucket.lock().unwrap().push(t.elapsed().as_micros());
+                }));
+            }
+        }
+        for h in handles {
+            h.await.unwrap();
+        }
+        let wall = start.elapsed();
+        let low = Arc::try_unwrap(low_lat).unwrap().into_inner().unwrap();
+        let high = Arc::try_unwrap(high_lat).unwrap().into_inner().unwrap();
+        report("priority(low tier)", low.clone(), wall);
+        report("priority(high tier)", high.clone(), wall);
+        let mut lo = low;
+        let mut hi = high;
+        lo.sort_unstable();
+        hi.sort_unstable();
+        let low_p50 = percentile(&lo, 0.50);
+        let high_p50 = percentile(&hi, 0.50);
+        if high_p50 < low_p50 {
+            println!("  PASS: high-tier p50 {high_p50:.1}ms < low-tier p50 {low_p50:.1}ms");
+        } else {
+            eprintln!(
+                "  FAIL: priority not honored — high p50 {high_p50:.1}ms >= low p50 {low_p50:.1}ms"
+            );
+            failures += 1;
+        }
+    }
+
+    if failures > 0 {
+        eprintln!("\n{failures} scenario(s) FAILED");
+        std::process::exit(1);
+    }
+    println!("\nAll load-test invariants held.");
+}

{llmshim-0.3.3 → llmshim-0.3.5}/docs/src/guides/reasoning.md

@@ -117,7 +117,7 @@ token budget scaled from `max_tokens` and floored at 1024:
 Gemini uses the four-rung
 `generationConfig.thinkingConfig.thinkingLevel` enum:
 
-| unified | gemini-3.5-flash / 3.6-flash / 3.5-flash-lite | gemini-3.7-flash (and gemini-3.1-pro) |
+| unified | gemini-3.5-flash / 3.6-flash / 3.5-flash-lite / 3.1-flash-lite | gemini-3.7-flash (and gemini-3.1-pro) |
 |---|---|---|
 | `none` | `minimal` (zero thinking tokens) | **`low`** (this model cannot disable thinking) |
 | `low` | `low` | `low` |

{llmshim-0.3.3 → llmshim-0.3.5}/docs/src/reference/models.md

@@ -14,7 +14,7 @@ the ID and display label.
 
 ## Registered catalog
 
-The current registry contains 26 entries, newest first within each provider.
+The current registry contains 27 entries, newest first within each provider.
 This page mirrors `src/models.rs`; use runtime discovery rather than parsing
 this table in applications.
 
@@ -52,6 +52,7 @@ this table in applications.
 | `gemini/gemini-3.6-flash` | Gemini 3.6 Flash |
 | `gemini/gemini-3.5-flash` | Gemini 3.5 Flash |
 | `gemini/gemini-3.5-flash-lite` | Gemini 3.5 Flash Lite |
+| `gemini/gemini-3.1-flash-lite` | Gemini 3.1 Flash Lite |
 
 ### xAI
 

{llmshim-0.3.3 → llmshim-0.3.5}/llmshim/__init__.py

@@ -14,6 +14,8 @@ Spec-faithful TypedDicts live in ``llmshim.types`` (also re-exported here) for
 static type-checking of requests, responses, and stream events.
 """
 
+from importlib.metadata import PackageNotFoundError, version
+
 from llmshim import types
 from llmshim._client import LlmShimError, chat, configure, health, models, stream
 from llmshim.types import (
@@ -55,4 +57,7 @@ __all__ = [
     "ToolCall",
     "Usage",
 ]
-__version__ = "0.1.22"
+try:
+    __version__ = version("llmshim")
+except PackageNotFoundError:  # not installed (e.g. running from source tree)
+    __version__ = "0.0.0"

{llmshim-0.3.3 → llmshim-0.3.5}/llmshim/_client.py

@@ -89,6 +89,7 @@ def configure(
     anthropic: Optional[str] = None,
     gemini: Optional[str] = None,
     xai: Optional[str] = None,
+    openrouter: Optional[str] = None,
 ) -> None:
     """Configure API keys. Writes to ~/.llmshim/config.toml.
 
@@ -98,6 +99,19 @@ def configure(
     Usage:
         import llmshim
         llmshim.configure(anthropic="sk-ant-...", openai="sk-...")
+
+    Self-hosted servers (vLLM, SGLang) are configured via environment
+    variables — NOT config.toml — which the auto-spawned proxy inherits from
+    the Python process. Set them before your first call, e.g.::
+
+        import os
+        os.environ["VLLM_BASE_URL"] = "http://localhost:8000/v1"
+        os.environ["VLLM_API_KEY"] = "..."      # optional
+        os.environ["SGLANG_BASE_URL"] = "http://localhost:30000/v1"
+        os.environ["SGLANG_API_KEY"] = "..."    # optional
+
+    Then address them via the model string, e.g. ``vllm/<served-model>`` or
+    ``sglang/<served-model>``.
     """
     config_dir = Path.home() / ".llmshim"
     config_path = config_dir / "config.toml"
@@ -125,6 +139,8 @@ def configure(
         keys["gemini"] = gemini
     if xai is not None:
         keys["xai"] = xai
+    if openrouter is not None:
+        keys["openrouter"] = openrouter
 
     # Write back
     config_dir.mkdir(parents=True, exist_ok=True)
@@ -152,6 +168,8 @@ def configure(
         os.environ["GEMINI_API_KEY"] = gemini
     if xai:
         os.environ["XAI_API_KEY"] = xai
+    if openrouter:
+        os.environ["OPENROUTER_API_KEY"] = openrouter
 
     # If server is already running, it won't pick up new keys until restart.
     # Force restart on next call.