switchboard-llm 0.1.0__tar.gz

switchboard_llm-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+.venv/
+__pycache__/
+*.pyc
+.secrets/
+.env
+.env.local
+bench/data/
+dist/
+build/
+*.egg-info/
+.ruff_cache/
+.pytest_cache/
+uv.lock

switchboard_llm-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Archit Dwivedi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

switchboard_llm-0.1.0/PKG-INFO

@@ -0,0 +1,186 @@
+Metadata-Version: 2.4
+Name: switchboard-llm
+Version: 0.1.0
+Summary: An OpenAI-compatible LLM router that saves cost without losing quality.
+Project-URL: Homepage, https://github.com/archit0/switchboard
+Project-URL: Repository, https://github.com/archit0/switchboard
+Project-URL: Issues, https://github.com/archit0/switchboard/issues
+Author: Archit Dwivedi
+License-Expression: MIT
+License-File: LICENSE
+Keywords: anthropic,cost,frugalgpt,gateway,gemini,llm,mixture-of-agents,openai,router
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: fastapi>=0.110
+Requires-Dist: httpx>=0.27
+Requires-Dist: uvicorn[standard]>=0.29
+Description-Content-Type: text/markdown
+
+# switchboard
+
+[![CI](https://github.com/archit0/switchboard/actions/workflows/ci.yml/badge.svg)](https://github.com/archit0/switchboard/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/switchboard-llm.svg)](https://pypi.org/project/switchboard-llm/)
+
+An **OpenAI-compatible LLM router** that saves cost without losing quality. Point
+any OpenAI client at it and it routes each request to the cheapest model that can
+handle it — easy prompts to a small model, hard ones to a parallel
+**Mixture-of-Agents** — trading a little latency for large savings while holding
+(or beating) frontier-model quality on a representative workload.
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="anything")
+client.chat.completions.create(model="router-cost", messages=[{"role": "user", "content": "..."}])
+```
+
+It works on top of **any OpenAI-compatible gateway that fronts multiple providers**
+behind one key (e.g. a LiteLLM proxy) — so one client can reach OpenAI, Anthropic,
+and Google models just by changing the `model` field. The router is a thin policy
+on top of that.
+
+---
+
+## Install
+
+```bash
+pip install switchboard-llm        # or: uv add switchboard-llm
+```
+
+Configure your gateway (any OpenAI-compatible endpoint):
+
+```bash
+export OPENAI_API_KEY=...                  # your gateway key
+export OPENAI_BASE_URL=https://.../v1      # your endpoint
+```
+
+## Use it
+
+**As a server** (drop-in for any OpenAI client):
+
+```bash
+switchboard serve                          # http://localhost:8000/v1  (use --port to change)
+```
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="anything")
+r = client.chat.completions.create(model="router", messages=[{"role": "user", "content": "Hi"}])
+print(r.model_extra["switchboard"])        # route, cost, savings telemetry
+```
+
+**As a library:**
+
+```python
+import asyncio
+from switchboard import Engine
+
+async def main():
+    eng = Engine()
+    rr = await eng.answer([{"role": "user", "content": "What is 17 * 23?"}], mode="cost")
+    print(rr.content, f"${rr.cost:.6f}", f"{rr.savings_pct:.0f}% cheaper than Opus")
+    await eng.aclose()
+
+asyncio.run(main())
+```
+
+**From the CLI:**
+
+```bash
+switchboard ask "Prove sqrt(2) is irrational" --mode quality
+switchboard models                         # probe which gateway models are actually live
+```
+
+---
+
+## The honest thesis (read this first)
+
+The goal is a router that is **cheaper than a frontier model (e.g. Opus) and
+matches-or-beats it on benchmarks**. That is achievable — but only as a
+**portfolio result over a realistic workload**, not a per-query miracle. The iron
+law:
+
+> On a *single hard query*, you cannot both beat the frontier model **and** be
+> cheaper than it on that same query.
+
+What you *can* do, and what this does:
+
+| Traffic | What the router does | Outcome |
+|---|---|---|
+| **Easy queries** (most real traffic) | route to a cheap model | quality ties Opus, **5–50× cheaper** |
+| **Hard queries** (the minority) | **Mixture-of-Agents**: several cheap/mid models answer in parallel, a synthesizer fuses them | quality can **match or exceed** a single Opus call, still **< Opus cost** |
+| **Repeats** | exact-match cache | **free** |
+
+Averaged over the workload, total spend is well below always-Opus and mean
+accuracy is **equal-or-better**. Grounded in **RouteLLM**, **FrugalGPT** (cascade
+with a judge), and **Mixture-of-Agents**.
+
+---
+
+## Modes
+
+Pick the strategy via the `model` field:
+
+| `model` | strategy |
+|---|---|
+| `router` / `router-balanced` | triage → single cheap (easy) / single mid (moderate) / Mixture-of-Agents (hard) |
+| `router-cost` | **FrugalGPT cascade** — answer cheap, a judge scores it, escalate only if low |
+| `router-quality` | bias one tier up — best quality while staying under Opus cost |
+
+Any **real** model id (`claude-opus-4-8`, `gpt-5.5`, …) passes straight through, so
+this also works as a plain multi-provider proxy.
+
+## How it works
+
+```
+request ─► [cache] ─► [triage: how hard?] ─► [policy] ──► single cheap model      (easy)
+                                                      └─► single mid model        (moderate)
+                                                      └─► Mixture-of-Agents        (hard)
+                                                            proposers ∥ ─► synthesizer
+```
+
+- **Triage** (`src/switchboard/classify.py`) — free heuristics (length, code/math
+  markers, multi-step verbs) decide obvious cases; a tiny LLM classifier scores the
+  ambiguous middle. Output: difficulty 1–5 → tier.
+- **Policy / execution** (`src/switchboard/engine.py`) — `single`, `moa` (parallel
+  proposers + synthesizer), or `cascade` (cheap → judge → escalate).
+- **Cost accounting** — every response carries its internal cost, an estimate of
+  what always-Opus would have cost, and the savings %, under a `switchboard` key.
+
+---
+
+## Results
+
+On **GSM8K (50 items, exact numeric grading)**, baseline = always `claude-opus-4-8`:
+
+| config | accuracy | total cost | vs Opus |
+|---|---|---|---|
+| always-Opus | 100.0% | $0.3674 | baseline |
+| `router-cost` | **100.0%** | $0.0064 | **57× cheaper — Pareto win** |
+| `router-quality` | 100.0% | $0.2781 | 1.3× cheaper |
+| `router-balanced` | 92.0% | $0.0611 | 6× cheaper but lost accuracy |
+
+Reproduce: `python -m bench.run_gsm8k --n 50 --seed 0`. Full write-up and honest
+caveats in [`RESULTS.md`](RESULTS.md). (The verifier is what makes routing safe —
+`router-balanced` has none and lost 8 points; `router-cost`'s judge is the fix.)
+
+---
+
+## Limitations & next steps
+
+- **Pricing is a list-price proxy** (`src/switchboard/config.py`). Drop your real
+  rate card into `pricing.json` (`{"model": [in_per_1M, out_per_1M]}`) to override.
+- **Triage under-detects "deceptively simple" trap questions** — `router-cost`/
+  `router-quality` compensate via the judge/MoA.
+- **Streaming is simulated** (full answer computed, then chunked) — MoA can't
+  token-stream; only the single-model path could truly stream.
+- **Semantic cache** (embed prompt → nearest neighbour) is not yet wired.
+- **The gateway's `/v1/models` list may be stale** — trust `switchboard models`.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

switchboard_llm-0.1.0/README.md

@@ -0,0 +1,162 @@
+# switchboard
+
+[![CI](https://github.com/archit0/switchboard/actions/workflows/ci.yml/badge.svg)](https://github.com/archit0/switchboard/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/switchboard-llm.svg)](https://pypi.org/project/switchboard-llm/)
+
+An **OpenAI-compatible LLM router** that saves cost without losing quality. Point
+any OpenAI client at it and it routes each request to the cheapest model that can
+handle it — easy prompts to a small model, hard ones to a parallel
+**Mixture-of-Agents** — trading a little latency for large savings while holding
+(or beating) frontier-model quality on a representative workload.
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="anything")
+client.chat.completions.create(model="router-cost", messages=[{"role": "user", "content": "..."}])
+```
+
+It works on top of **any OpenAI-compatible gateway that fronts multiple providers**
+behind one key (e.g. a LiteLLM proxy) — so one client can reach OpenAI, Anthropic,
+and Google models just by changing the `model` field. The router is a thin policy
+on top of that.
+
+---
+
+## Install
+
+```bash
+pip install switchboard-llm        # or: uv add switchboard-llm
+```
+
+Configure your gateway (any OpenAI-compatible endpoint):
+
+```bash
+export OPENAI_API_KEY=...                  # your gateway key
+export OPENAI_BASE_URL=https://.../v1      # your endpoint
+```
+
+## Use it
+
+**As a server** (drop-in for any OpenAI client):
+
+```bash
+switchboard serve                          # http://localhost:8000/v1  (use --port to change)
+```
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="anything")
+r = client.chat.completions.create(model="router", messages=[{"role": "user", "content": "Hi"}])
+print(r.model_extra["switchboard"])        # route, cost, savings telemetry
+```
+
+**As a library:**
+
+```python
+import asyncio
+from switchboard import Engine
+
+async def main():
+    eng = Engine()
+    rr = await eng.answer([{"role": "user", "content": "What is 17 * 23?"}], mode="cost")
+    print(rr.content, f"${rr.cost:.6f}", f"{rr.savings_pct:.0f}% cheaper than Opus")
+    await eng.aclose()
+
+asyncio.run(main())
+```
+
+**From the CLI:**
+
+```bash
+switchboard ask "Prove sqrt(2) is irrational" --mode quality
+switchboard models                         # probe which gateway models are actually live
+```
+
+---
+
+## The honest thesis (read this first)
+
+The goal is a router that is **cheaper than a frontier model (e.g. Opus) and
+matches-or-beats it on benchmarks**. That is achievable — but only as a
+**portfolio result over a realistic workload**, not a per-query miracle. The iron
+law:
+
+> On a *single hard query*, you cannot both beat the frontier model **and** be
+> cheaper than it on that same query.
+
+What you *can* do, and what this does:
+
+| Traffic | What the router does | Outcome |
+|---|---|---|
+| **Easy queries** (most real traffic) | route to a cheap model | quality ties Opus, **5–50× cheaper** |
+| **Hard queries** (the minority) | **Mixture-of-Agents**: several cheap/mid models answer in parallel, a synthesizer fuses them | quality can **match or exceed** a single Opus call, still **< Opus cost** |
+| **Repeats** | exact-match cache | **free** |
+
+Averaged over the workload, total spend is well below always-Opus and mean
+accuracy is **equal-or-better**. Grounded in **RouteLLM**, **FrugalGPT** (cascade
+with a judge), and **Mixture-of-Agents**.
+
+---
+
+## Modes
+
+Pick the strategy via the `model` field:
+
+| `model` | strategy |
+|---|---|
+| `router` / `router-balanced` | triage → single cheap (easy) / single mid (moderate) / Mixture-of-Agents (hard) |
+| `router-cost` | **FrugalGPT cascade** — answer cheap, a judge scores it, escalate only if low |
+| `router-quality` | bias one tier up — best quality while staying under Opus cost |
+
+Any **real** model id (`claude-opus-4-8`, `gpt-5.5`, …) passes straight through, so
+this also works as a plain multi-provider proxy.
+
+## How it works
+
+```
+request ─► [cache] ─► [triage: how hard?] ─► [policy] ──► single cheap model      (easy)
+                                                      └─► single mid model        (moderate)
+                                                      └─► Mixture-of-Agents        (hard)
+                                                            proposers ∥ ─► synthesizer
+```
+
+- **Triage** (`src/switchboard/classify.py`) — free heuristics (length, code/math
+  markers, multi-step verbs) decide obvious cases; a tiny LLM classifier scores the
+  ambiguous middle. Output: difficulty 1–5 → tier.
+- **Policy / execution** (`src/switchboard/engine.py`) — `single`, `moa` (parallel
+  proposers + synthesizer), or `cascade` (cheap → judge → escalate).
+- **Cost accounting** — every response carries its internal cost, an estimate of
+  what always-Opus would have cost, and the savings %, under a `switchboard` key.
+
+---
+
+## Results
+
+On **GSM8K (50 items, exact numeric grading)**, baseline = always `claude-opus-4-8`:
+
+| config | accuracy | total cost | vs Opus |
+|---|---|---|---|
+| always-Opus | 100.0% | $0.3674 | baseline |
+| `router-cost` | **100.0%** | $0.0064 | **57× cheaper — Pareto win** |
+| `router-quality` | 100.0% | $0.2781 | 1.3× cheaper |
+| `router-balanced` | 92.0% | $0.0611 | 6× cheaper but lost accuracy |
+
+Reproduce: `python -m bench.run_gsm8k --n 50 --seed 0`. Full write-up and honest
+caveats in [`RESULTS.md`](RESULTS.md). (The verifier is what makes routing safe —
+`router-balanced` has none and lost 8 points; `router-cost`'s judge is the fix.)
+
+---
+
+## Limitations & next steps
+
+- **Pricing is a list-price proxy** (`src/switchboard/config.py`). Drop your real
+  rate card into `pricing.json` (`{"model": [in_per_1M, out_per_1M]}`) to override.
+- **Triage under-detects "deceptively simple" trap questions** — `router-cost`/
+  `router-quality` compensate via the judge/MoA.
+- **Streaming is simulated** (full answer computed, then chunked) — MoA can't
+  token-stream; only the single-model path could truly stream.
+- **Semantic cache** (embed prompt → nearest neighbour) is not yet wired.
+- **The gateway's `/v1/models` list may be stale** — trust `switchboard models`.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

switchboard_llm-0.1.0/RESULTS.md

@@ -0,0 +1,83 @@
+# Measured results
+
+## GSM8K — 50 items, seed 0 (recognized benchmark)
+
+`.venv/bin/python -m bench.run_gsm8k --n 50 --seed 0`. Zero-shot chain-of-thought,
+exact numeric grading (gold = number after `####`), no LLM judge. Baseline =
+always `claude-opus-4-8`, same 50 items for every config.
+
+```
+config                   acc     total$   $/correct   mean_ms
+------------------------------------------------------------------------------
+claude-opus-4-8       100.0%    0.36744    0.007349      2094
+router-balanced        92.0%    0.06106    0.001327      4025
+router-cost           100.0%    0.00640    0.000128      2898
+router-quality        100.0%    0.27812    0.005562      6784
+------------------------------------------------------------------------------
+router-cost        acc 100.0% (+0.0pt)  cost $0.00640  (57.4x cheaper)  PARETO-WIN
+router-quality     acc 100.0% (+0.0pt)  cost $0.27812  ( 1.3x cheaper)  PARETO-WIN
+router-balanced    acc  92.0% (-8.0pt)  cost $0.06106  ( 6.0x cheaper)  cheaper, LOWER acc
+```
+
+**Findings (honest):**
+- **`router-cost` (FrugalGPT cascade) is Pareto-dominant: ties Opus at 100% for
+  57× less cost.** The cheap model + judge resolves most items; only low-confidence
+  ones escalate. This is the headline result on a recognized benchmark.
+- **`router-balanced` LOST 8 points** (4/50 items). This is the real cost of naive
+  single-model routing with *no verification* — when the mid model errs, nothing
+  catches it. The cascade's judge is precisely the fix, and the data shows it.
+  Reported, not hidden. (Actionable next step: add a light verifier to the
+  balanced mid tier.)
+- **Magnitude caveat:** GSM8K is now *easy* for modern small models, so 57× is
+  near the optimistic end. Expect smaller-but-real savings on harder suites
+  (MMLU-Pro, GPQA) where the cheap tier carries less of the load.
+
+---
+
+## Mixed smoke set — 15 items
+
+Run of `.venv/bin/python -m bench.run_bench` on the 15-item mixed set
+(easy factual/math, CRT "trap" questions, open-ended reasoning & code).
+Baseline = always `claude-opus-4-8`. Grader = `gemini-3.1-pro-preview`
+(independent of the MoA pool). Costs use the list-price proxies in
+`src/switchboard/config.py`.
+
+```
+config                  acc     total$   $/correct   mean_ms
+------------------------------------------------------------------------------
+claude-opus-4-8       93.3%    0.10338     0.00738      3084
+router-balanced       93.3%    0.01162     0.00083      3138
+router-cost          100.0%    0.01106     0.00074      4534
+router-quality       100.0%    0.04278     0.00285      5502
+------------------------------------------------------------------------------
+router-balanced    acc  93.3% (+0.0pt)  cost $0.01162  ( 8.9x cheaper)  PARETO-WIN
+router-cost        acc 100.0% (+6.7pt)  cost $0.01106  ( 9.3x cheaper)  PARETO-WIN
+router-quality     acc 100.0% (+6.7pt)  cost $0.04278  ( 2.4x cheaper)  PARETO-WIN
+```
+
+## Reading this honestly
+
+- **All three router modes are Pareto wins**: equal-or-better accuracy than
+  always-Opus at 2.4×–9.3× lower cost. `router-cost` is strictly better on
+  *both* axes here.
+- **`router-balanced` matches Opus at near-Opus latency** (3.1s vs 3.1s) because
+  most items route to a single cheap model. `router-quality` is slower (5.5s)
+  because it fires Mixture-of-Agents on more items — that is the latency you pay
+  for the parallel internal calls you asked about.
+- **Caveats (don't over-read a small sample):**
+  - 15 items, single-vote judge → the 3 open-ended items carry grader noise.
+    Opus's (correct) √2 proof was marked wrong by the judge in this run; a
+    majority-vote judge would smooth this out.
+  - The 12 exact-match items are noise-free and every config answered all 12
+    correctly — the accuracy separation comes entirely from the judged items.
+  - Costs are **list-price proxies**, not your gateway's actual rates. Swap in the real
+    rate card via `pricing.json` to get true dollar savings.
+
+## How to reproduce / scale up
+
+```bash
+.venv/bin/python -m bench.run_bench                 # this run
+.venv/bin/python -m bench.run_bench --n 8           # quick smoke
+# add your own items to bench/dataset.py to test on YOUR workload mix —
+# the savings depend heavily on how much of your traffic is genuinely easy.
+```

switchboard_llm-0.1.0/pyproject.toml

@@ -0,0 +1,67 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "switchboard-llm"
+description = "An OpenAI-compatible LLM router that saves cost without losing quality."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.11"
+authors = [{ name = "Archit Dwivedi" }]
+keywords = ["llm", "router", "openai", "anthropic", "gemini", "mixture-of-agents", "frugalgpt", "cost", "gateway"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dynamic = ["version"]
+dependencies = [
+  "httpx>=0.27",
+  "fastapi>=0.110",
+  "uvicorn[standard]>=0.29",
+]
+
+[project.scripts]
+switchboard = "switchboard.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/archit0/switchboard"
+Repository = "https://github.com/archit0/switchboard"
+Issues = "https://github.com/archit0/switchboard/issues"
+
+[dependency-groups]
+dev = [
+  "pytest>=8",
+  "ruff>=0.6",
+]
+
+[tool.hatch.version]
+path = "src/switchboard/__init__.py"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/switchboard"]
+
+[tool.hatch.build.targets.sdist]
+include = ["src/switchboard", "README.md", "RESULTS.md", "LICENSE"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120
+src = ["src", "tests", "bench", "tools"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "W", "C4"]
+ignore = [
+  "E501",  # long lines (lots of explanatory comments / prompts)
+  "B008",  # function calls in argument defaults (FastAPI-style)
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["S101"]

switchboard_llm-0.1.0/src/switchboard/__init__.py

@@ -0,0 +1,57 @@
+"""switchboard — an OpenAI-compatible LLM router that saves cost without losing quality.
+
+Point any OpenAI client at the switchboard server and it routes each request to
+the cheapest model that can handle it — easy prompts to a small model, hard ones
+to a parallel Mixture-of-Agents — trading a little latency for large savings while
+holding (or beating) frontier-model quality on a representative workload.
+
+Quickstart (library)::
+
+    import asyncio
+    from switchboard import Engine
+
+    async def main():
+        eng = Engine()
+        result = await eng.answer(
+            [{"role": "user", "content": "What is 17 * 23?"}],
+            mode="cost",
+        )
+        print(result.content, result.cost, result.savings_pct)
+        await eng.aclose()
+
+    asyncio.run(main())
+
+Quickstart (OpenAI-compatible server)::
+
+    $ switchboard serve            # http://localhost:8000/v1
+
+    from openai import OpenAI
+    client = OpenAI(base_url="http://localhost:8000/v1", api_key="anything")
+    client.chat.completions.create(model="router-cost", messages=[...])
+
+The gateway is configured via the ``OPENAI_BASE_URL`` and ``OPENAI_API_KEY``
+environment variables (any OpenAI-compatible endpoint that fronts multiple
+providers — e.g. a LiteLLM proxy — works).
+"""
+
+from switchboard.cache import ResponseCache
+from switchboard.classify import Triage, triage
+from switchboard.config import GatewayConfig, cost_usd, price_of
+from switchboard.engine import Engine, RouteResult
+from switchboard.gateway import Completion, Gateway
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "Completion",
+    "Engine",
+    "Gateway",
+    "GatewayConfig",
+    "ResponseCache",
+    "RouteResult",
+    "Triage",
+    "__version__",
+    "cost_usd",
+    "price_of",
+    "triage",
+]

switchboard_llm-0.1.0/src/switchboard/cache.py

@@ -0,0 +1,56 @@
+"""Exact-match response cache.
+
+The cheapest API call is the one you never make. Repeated/identical prompts
+(very common in agent loops and eval harnesses) return instantly at zero cost.
+A semantic cache (embed the prompt, nearest-neighbour over past prompts) is the
+natural next step — the gateway exposes `gemini-embedding-*` and
+`text-embedding-3-*` for exactly this — but exact-match already captures the
+biggest, safest wins without false hits.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import threading
+import time
+from typing import Any
+
+
+def _key(messages: list[dict], mode: str) -> str:
+    blob = json.dumps({"m": messages, "mode": mode}, sort_keys=True, ensure_ascii=False)
+    return hashlib.sha256(blob.encode("utf-8")).hexdigest()
+
+
+class ResponseCache:
+    def __init__(self, max_items: int = 4096, ttl_seconds: float | None = None):
+        self._store: dict[str, tuple[float, Any]] = {}
+        self._lock = threading.Lock()
+        self._max = max_items
+        self._ttl = ttl_seconds
+        self.hits = 0
+        self.misses = 0
+
+    def get(self, messages: list[dict], mode: str) -> Any | None:
+        k = _key(messages, mode)
+        with self._lock:
+            item = self._store.get(k)
+            if item is None:
+                self.misses += 1
+                return None
+            ts, val = item
+            if self._ttl is not None and (time.time() - ts) > self._ttl:
+                del self._store[k]
+                self.misses += 1
+                return None
+            self.hits += 1
+            return val
+
+    def put(self, messages: list[dict], mode: str, value: Any) -> None:
+        k = _key(messages, mode)
+        with self._lock:
+            if len(self._store) >= self._max and k not in self._store:
+                # drop oldest
+                oldest = min(self._store, key=lambda x: self._store[x][0])
+                del self._store[oldest]
+            self._store[k] = (time.time(), value)