cachellm-proxy 0.2.1__tar.gz → 0.2.2__tar.gz

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cachellm-proxy
-Version: 0.2.1
+Version: 0.2.2
 Summary: A drop-in semantic cache for OpenAI-compatible LLM APIs: match requests by meaning, serve repeats instantly, cut spend.
 Keywords: llm,cache,semantic-cache,openai,bedrock,proxy,vector-search,redis,fastapi,llmops,cost-optimization
 Author: Adarsh Dwivedi
@@ -53,13 +53,13 @@ Description-Content-Type: text/markdown
 [![CI](https://github.com/adarshcod30/CacheLLM/actions/workflows/ci.yml/badge.svg)](https://github.com/adarshcod30/CacheLLM/actions/workflows/ci.yml)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-265%20passing-brightgreen)](tests/)
+[![Tests](https://img.shields.io/badge/tests-327%20passing-brightgreen)](tests/)
 [![PyPI](https://img.shields.io/pypi/v/cachellm-proxy)](https://pypi.org/project/cachellm-proxy/)
 [![Hit rate](https://img.shields.io/badge/hit%20rate-77%25%20on%20Bedrock-orange)](docs/evaluation.md)
 
 **A drop-in semantic cache for OpenAI-compatible LLM APIs. Change one base URL, and questions your model has already answered come back in milliseconds instead of seconds.**
 
-On a 2,000-request replay against **AWS Bedrock** it served **77% of traffic from cache** with **zero false positives** on genuinely new questions, cutting spend by **78%** and p95 latency from 1,023 ms to **5.7 ms**.
+On a 2,000-request replay against **AWS Bedrock** it served **77% of traffic from cache** with **zero false positives** on genuinely new questions, cutting spend by **78%** and p95 latency from 1,023 ms to **5.7 ms** on a cache hit, or 9.7 ms when the question was reworded.
 
 [Quick start](#quick-start) · [How it works](#how-it-works) · [Evaluation](docs/evaluation.md) · [API reference](#api-reference) · [Deployment](#deployment-and-infrastructure)
 
@@ -97,7 +97,8 @@ Everything else stays the same: same request shape, same response shape, same er
 | False positives on genuinely new questions | **0 of 368** |
 | Reworded repeats served from cache | 95.2% |
 | Exact repeats served from cache | 99.3% |
-| p95 latency, cache hit | **5.7 ms** |
+| p95 latency, any cache hit | **5.7 ms** |
+| p95 latency, reworded hit | 9.7 ms |
 | p95 latency, cache miss | 1,022.8 ms |
 | p95 speedup | **178.8x** |
 | Cost reduction | **78.0%** |
@@ -257,7 +258,7 @@ This is where a caching project usually hand-waves. The full write-up is in [doc
 
 **Result 2: thresholds do not transfer between models.** The safe threshold ranges from 0.89 for MiniLM to 0.98 for Arctic-embed. Every model tested had negative separation, meaning the mean duplicate score sat below the worst hard negative. Bigger and slower did not fix it.
 
-| Model | Safe threshold | Recall there | Embed ms |
+| Model | Safe threshold | Recall there | Embed ms, short probe |
 | --- | ---: | ---: | ---: |
 | `all-MiniLM-L6-v2` | **0.89** | **35.0%** | 5.7 |
 | `gte-base` | 0.96 | 26.0% | 20.6 |
@@ -266,7 +267,7 @@ This is where a caching project usually hand-waves. The full write-up is in [doc
 | `snowflake-arctic-embed-s` | 0.98 | 11.4% | 3.2 |
 | `bge-small-en-v1.5` | 0.96 | 8.1% | 3.8 |
 
-MiniLM gives four times the safe recall of bge-small at a third of the download size, so it is the default. The whole table ships in code as `CALIBRATED_THRESHOLDS`, and the proxy warns at startup if you configure a model it has never measured.
+MiniLM gives four times the safe recall of bge-small, for a slightly larger download of 86 MB against 63 MB, so it is the default. The embed column is a ranking from short synthetic strings, not what a real question costs. The load test measures that directly. The whole table ships in code as `CALIBRATED_THRESHOLDS`, and the proxy warns at startup if you configure a model it has never measured.
 
 **Result 3, a negative one: a lexical guard does not rescue it.** The obvious fix is to require matched prompts to share content words. Measured, hard negatives have *higher* token overlap (0.42 mean) than genuine paraphrases share vocabulary, because they differ by exactly one decisive word. The guard rejects good matches and keeps dangerous ones. It was measured and dropped rather than shipped.
 
@@ -366,7 +367,7 @@ One environment variable per host. Everything except Bedrock speaks the OpenAI p
 | Host | `CACHELLM_OPENAI_BASE_URL` | Example model |
 | --- | --- | --- |
 | OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` |
-| Groq | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile` |
+| Groq | `https://api.groq.com/openai/v1` | `openai/gpt-oss-20b` |
 | Google Gemini | `https://generativelanguage.googleapis.com/v1beta/openai/` | `gemini-2.5-flash` |
 | Anthropic | `https://api.anthropic.com/v1` | `claude-haiku-4-5` |
 | OpenRouter | `https://openrouter.ai/api/v1` | `anthropic/claude-3.5-sonnet` |
@@ -380,6 +381,10 @@ CACHELLM_OPENAI_API_KEY=gsk_your_key \
 cachellm serve
 ```
 
+If the key is already in its usual variable, such as `GROQ_API_KEY` or `GEMINI_API_KEY`, you can skip both lines: the proxy finds it at startup and says which host it picked.
+
+**Checked live, not just in tests.** On 2026-09-11 the proxy was run against real **Google Gemini** (`gemini-2.5-flash`, `gemini-3.5-flash`) and **Groq** (`openai/gpt-oss-20b`, `openai/gpt-oss-120b`), driven by the official OpenAI SDK, alongside the AWS Bedrock benchmark. Each run covers a miss, an exact hit, a reworded hit, a similar question that must not hit, a stream and its replay, a hot-temperature bypass and an unknown model's error, and checks that the API key never reaches the log. Results are in [docs/evaluation.md](docs/evaluation.md#live-provider-checks), and `bench/live_check.py` reruns them with your own key. The other hosts share the same adapter and its tests, but have not yet been run against the real service.
+
 **AWS Bedrock** is the one exception, because it does not speak the OpenAI protocol. It needs the `aws` extra, and then uses your existing AWS credentials with no vendor API key at all:
 
 ```bash
@@ -401,7 +406,7 @@ Three rules, in order. You never configure a model list.
 2. **A recognisable vendor convention.** Bedrock ids are always `vendor.model`, so `amazon.nova-lite-v1:0` and `us.anthropic.claude-3-haiku-20240307-v1:0` are identified with no prefix. OpenAI's own families (`gpt-`, `o1`, `o3`, `text-embedding-`) are identified the same way, whatever the default is.
 3. **Otherwise the configured default**, which is the OpenAI-compatible adapter. Names like `llama3.2`, `mixtral-8x7b-32768` and `qwen2.5-coder:7b` are served by Groq, Ollama, Together and OpenRouter alike, so the endpoint you configured is the only sensible answer.
 
-Model ids that legitimately contain a slash, which OpenRouter and Together both use, are forwarded whole. Only this proxy's own prefixes are stripped. Note that `anthropic.claude-…` with a dot is a Bedrock id while `anthropic/claude-…` with a slash is an OpenRouter id, and the router tells them apart.
+Model ids that legitimately contain a slash, which OpenRouter and Together both use, are forwarded whole. Only this proxy's own prefixes are stripped, and `openai/` only when the upstream is OpenAI itself: Groq, OpenRouter and Together name OpenAI's models `openai/gpt-oss-120b`, so to them the prefix is part of the id. Note that `anthropic.claude-…` with a dot is a Bedrock id while `anthropic/claude-…` with a slash is an OpenRouter id, and the router tells them apart.
 
 Ask it directly if you are unsure:
 
@@ -410,6 +415,39 @@ curl -s localhost:8080/admin/route/meta-llama/Llama-3.3-70B-Instruct-Turbo
 curl -s localhost:8080/admin/providers
 ```
 
+### Where the cache lives
+
+One setting, `CACHELLM_BACKEND`, with three values. Most people never touch it.
+
+| Value | What happens | Choose it when |
+| --- | --- | --- |
+| `auto`, the default | Redis if the extra is installed and a usable server answers, the in-process cache otherwise | You have no opinion |
+| `memory` | A numpy matrix inside the proxy. Nothing to install | One process: a laptop, a side project, a single server |
+| `redis` | One Redis server shared by every copy of the proxy | Several processes must share one cache |
+
+Memory is not the slow option. Below roughly 100,000 entries it is the faster one: scanning 20,000 cached prompts takes 0.85 ms, while a Redis round trip alone costs 2 to 3 ms. Redis earns its place when several processes need one shared cache. Run four copies of the proxy on memory and you have four separate caches, each a quarter as warm.
+
+Memory can survive a restart too. Give it a file, and it saves there on a clean shutdown and loads it back on start:
+
+```bash
+CACHELLM_MEMORY_SNAPSHOT_PATH=$HOME/.cachellm/cache.npz cachellm serve
+```
+
+**Setting up Redis.** Install the extra, then run a Redis 8 server:
+
+```bash
+pip install "cachellm-proxy[redis]"
+brew install redis && brew services start redis    # macOS
+docker run -d -p 6379:6379 redis:8-alpine          # anywhere with Docker
+```
+
+The server has two requirements, and both are checked at startup:
+
+- **It needs the search module**, which is what stores and searches vectors. Redis 8 from Homebrew or the official Docker image includes it. Many Linux distribution packages ship an older Redis without it, so prefer the Docker image there. A hosted Redis works if its plan includes search.
+- **It has to be database 0**, because Redis search cannot index any other. Keep several apps apart with `CACHELLM_INDEX_NAME` instead.
+
+If either is missing, `auto` uses memory and says exactly why, both in a startup warning and at the top of `cachellm stats`. With `CACHELLM_BACKEND=redis` the proxy never switches storage behind your back. It keeps answering without a cache and reports itself degraded until Redis is fixed.
+
 ## API reference
 
 ### Chat completions
@@ -506,7 +544,7 @@ Every setting is an environment variable prefixed `CACHELLM_`, or a line in `.en
 | Variable | Default | Notes |
 | --- | --- | --- |
 | `CACHELLM_BACKEND` | `auto` | `memory` needs nothing, `redis` shares one cache across workers, `auto` uses Redis when reachable and memory when not |
-| `CACHELLM_REDIS_URL` | `redis://localhost:6379/0` | Must be database 0. Redis Search cannot index any other |
+| `CACHELLM_REDIS_URL` | `redis://localhost:6379/0` | Must be database 0, and the server needs the search module. See [Where the cache lives](#where-the-cache-lives) |
 | `CACHELLM_MEMORY_MAX_ENTRIES` | `50000` | Cap for the in-memory store. 50k of 384-dim vectors is about 73 MB |
 | `CACHELLM_MEMORY_SNAPSHOT_PATH` | unset | Persist the in-memory cache to this file so a restart does not start cold |
 | `CACHELLM_API_KEYS` | empty | Comma-separated client keys. Empty disables auth, which is local development only |
@@ -517,7 +555,7 @@ Every setting is an environment variable prefixed `CACHELLM_`, or a line in `.en
 | `CACHELLM_TTL_VOLATILE` | `900` | Fifteen minutes for anything about now |
 | `CACHELLM_MAX_CACHEABLE_TEMPERATURE` | `0.3` | Above this, nothing is cached |
 | `CACHELLM_PII_GUARD` | `true` | Refuse to store prompts that look personal |
-| `CACHELLM_DEFAULT_PROVIDER` | `bedrock` | `bedrock`, `openai` or `fake` |
+| `CACHELLM_DEFAULT_PROVIDER` | `openai` | `openai` covers every OpenAI-compatible host. Also `bedrock` or `fake`. Left unset, the proxy picks from the keys it finds |
 | `CACHELLM_LOG_PROMPTS` | `false` | Prompt text stays out of logs unless you opt in |
 
 ## Deployment and infrastructure
@@ -586,7 +624,7 @@ cachellm/
 ## Testing
 
 ```bash
-make test          # 126 tests
+make test          # 327 tests
 make test-cov      # with coverage
 make lint          # ruff and mypy
 ```

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/README.md

@@ -3,13 +3,13 @@
 [![CI](https://github.com/adarshcod30/CacheLLM/actions/workflows/ci.yml/badge.svg)](https://github.com/adarshcod30/CacheLLM/actions/workflows/ci.yml)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-265%20passing-brightgreen)](tests/)
+[![Tests](https://img.shields.io/badge/tests-327%20passing-brightgreen)](tests/)
 [![PyPI](https://img.shields.io/pypi/v/cachellm-proxy)](https://pypi.org/project/cachellm-proxy/)
 [![Hit rate](https://img.shields.io/badge/hit%20rate-77%25%20on%20Bedrock-orange)](docs/evaluation.md)
 
 **A drop-in semantic cache for OpenAI-compatible LLM APIs. Change one base URL, and questions your model has already answered come back in milliseconds instead of seconds.**
 
-On a 2,000-request replay against **AWS Bedrock** it served **77% of traffic from cache** with **zero false positives** on genuinely new questions, cutting spend by **78%** and p95 latency from 1,023 ms to **5.7 ms**.
+On a 2,000-request replay against **AWS Bedrock** it served **77% of traffic from cache** with **zero false positives** on genuinely new questions, cutting spend by **78%** and p95 latency from 1,023 ms to **5.7 ms** on a cache hit, or 9.7 ms when the question was reworded.
 
 [Quick start](#quick-start) · [How it works](#how-it-works) · [Evaluation](docs/evaluation.md) · [API reference](#api-reference) · [Deployment](#deployment-and-infrastructure)
 
@@ -47,7 +47,8 @@ Everything else stays the same: same request shape, same response shape, same er
 | False positives on genuinely new questions | **0 of 368** |
 | Reworded repeats served from cache | 95.2% |
 | Exact repeats served from cache | 99.3% |
-| p95 latency, cache hit | **5.7 ms** |
+| p95 latency, any cache hit | **5.7 ms** |
+| p95 latency, reworded hit | 9.7 ms |
 | p95 latency, cache miss | 1,022.8 ms |
 | p95 speedup | **178.8x** |
 | Cost reduction | **78.0%** |
@@ -207,7 +208,7 @@ This is where a caching project usually hand-waves. The full write-up is in [doc
 
 **Result 2: thresholds do not transfer between models.** The safe threshold ranges from 0.89 for MiniLM to 0.98 for Arctic-embed. Every model tested had negative separation, meaning the mean duplicate score sat below the worst hard negative. Bigger and slower did not fix it.
 
-| Model | Safe threshold | Recall there | Embed ms |
+| Model | Safe threshold | Recall there | Embed ms, short probe |
 | --- | ---: | ---: | ---: |
 | `all-MiniLM-L6-v2` | **0.89** | **35.0%** | 5.7 |
 | `gte-base` | 0.96 | 26.0% | 20.6 |
@@ -216,7 +217,7 @@ This is where a caching project usually hand-waves. The full write-up is in [doc
 | `snowflake-arctic-embed-s` | 0.98 | 11.4% | 3.2 |
 | `bge-small-en-v1.5` | 0.96 | 8.1% | 3.8 |
 
-MiniLM gives four times the safe recall of bge-small at a third of the download size, so it is the default. The whole table ships in code as `CALIBRATED_THRESHOLDS`, and the proxy warns at startup if you configure a model it has never measured.
+MiniLM gives four times the safe recall of bge-small, for a slightly larger download of 86 MB against 63 MB, so it is the default. The embed column is a ranking from short synthetic strings, not what a real question costs. The load test measures that directly. The whole table ships in code as `CALIBRATED_THRESHOLDS`, and the proxy warns at startup if you configure a model it has never measured.
 
 **Result 3, a negative one: a lexical guard does not rescue it.** The obvious fix is to require matched prompts to share content words. Measured, hard negatives have *higher* token overlap (0.42 mean) than genuine paraphrases share vocabulary, because they differ by exactly one decisive word. The guard rejects good matches and keeps dangerous ones. It was measured and dropped rather than shipped.
 
@@ -316,7 +317,7 @@ One environment variable per host. Everything except Bedrock speaks the OpenAI p
 | Host | `CACHELLM_OPENAI_BASE_URL` | Example model |
 | --- | --- | --- |
 | OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` |
-| Groq | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile` |
+| Groq | `https://api.groq.com/openai/v1` | `openai/gpt-oss-20b` |
 | Google Gemini | `https://generativelanguage.googleapis.com/v1beta/openai/` | `gemini-2.5-flash` |
 | Anthropic | `https://api.anthropic.com/v1` | `claude-haiku-4-5` |
 | OpenRouter | `https://openrouter.ai/api/v1` | `anthropic/claude-3.5-sonnet` |
@@ -330,6 +331,10 @@ CACHELLM_OPENAI_API_KEY=gsk_your_key \
 cachellm serve
 ```
 
+If the key is already in its usual variable, such as `GROQ_API_KEY` or `GEMINI_API_KEY`, you can skip both lines: the proxy finds it at startup and says which host it picked.
+
+**Checked live, not just in tests.** On 2026-09-11 the proxy was run against real **Google Gemini** (`gemini-2.5-flash`, `gemini-3.5-flash`) and **Groq** (`openai/gpt-oss-20b`, `openai/gpt-oss-120b`), driven by the official OpenAI SDK, alongside the AWS Bedrock benchmark. Each run covers a miss, an exact hit, a reworded hit, a similar question that must not hit, a stream and its replay, a hot-temperature bypass and an unknown model's error, and checks that the API key never reaches the log. Results are in [docs/evaluation.md](docs/evaluation.md#live-provider-checks), and `bench/live_check.py` reruns them with your own key. The other hosts share the same adapter and its tests, but have not yet been run against the real service.
+
 **AWS Bedrock** is the one exception, because it does not speak the OpenAI protocol. It needs the `aws` extra, and then uses your existing AWS credentials with no vendor API key at all:
 
 ```bash
@@ -351,7 +356,7 @@ Three rules, in order. You never configure a model list.
 2. **A recognisable vendor convention.** Bedrock ids are always `vendor.model`, so `amazon.nova-lite-v1:0` and `us.anthropic.claude-3-haiku-20240307-v1:0` are identified with no prefix. OpenAI's own families (`gpt-`, `o1`, `o3`, `text-embedding-`) are identified the same way, whatever the default is.
 3. **Otherwise the configured default**, which is the OpenAI-compatible adapter. Names like `llama3.2`, `mixtral-8x7b-32768` and `qwen2.5-coder:7b` are served by Groq, Ollama, Together and OpenRouter alike, so the endpoint you configured is the only sensible answer.
 
-Model ids that legitimately contain a slash, which OpenRouter and Together both use, are forwarded whole. Only this proxy's own prefixes are stripped. Note that `anthropic.claude-…` with a dot is a Bedrock id while `anthropic/claude-…` with a slash is an OpenRouter id, and the router tells them apart.
+Model ids that legitimately contain a slash, which OpenRouter and Together both use, are forwarded whole. Only this proxy's own prefixes are stripped, and `openai/` only when the upstream is OpenAI itself: Groq, OpenRouter and Together name OpenAI's models `openai/gpt-oss-120b`, so to them the prefix is part of the id. Note that `anthropic.claude-…` with a dot is a Bedrock id while `anthropic/claude-…` with a slash is an OpenRouter id, and the router tells them apart.
 
 Ask it directly if you are unsure:
 
@@ -360,6 +365,39 @@ curl -s localhost:8080/admin/route/meta-llama/Llama-3.3-70B-Instruct-Turbo
 curl -s localhost:8080/admin/providers
 ```
 
+### Where the cache lives
+
+One setting, `CACHELLM_BACKEND`, with three values. Most people never touch it.
+
+| Value | What happens | Choose it when |
+| --- | --- | --- |
+| `auto`, the default | Redis if the extra is installed and a usable server answers, the in-process cache otherwise | You have no opinion |
+| `memory` | A numpy matrix inside the proxy. Nothing to install | One process: a laptop, a side project, a single server |
+| `redis` | One Redis server shared by every copy of the proxy | Several processes must share one cache |
+
+Memory is not the slow option. Below roughly 100,000 entries it is the faster one: scanning 20,000 cached prompts takes 0.85 ms, while a Redis round trip alone costs 2 to 3 ms. Redis earns its place when several processes need one shared cache. Run four copies of the proxy on memory and you have four separate caches, each a quarter as warm.
+
+Memory can survive a restart too. Give it a file, and it saves there on a clean shutdown and loads it back on start:
+
+```bash
+CACHELLM_MEMORY_SNAPSHOT_PATH=$HOME/.cachellm/cache.npz cachellm serve
+```
+
+**Setting up Redis.** Install the extra, then run a Redis 8 server:
+
+```bash
+pip install "cachellm-proxy[redis]"
+brew install redis && brew services start redis    # macOS
+docker run -d -p 6379:6379 redis:8-alpine          # anywhere with Docker
+```
+
+The server has two requirements, and both are checked at startup:
+
+- **It needs the search module**, which is what stores and searches vectors. Redis 8 from Homebrew or the official Docker image includes it. Many Linux distribution packages ship an older Redis without it, so prefer the Docker image there. A hosted Redis works if its plan includes search.
+- **It has to be database 0**, because Redis search cannot index any other. Keep several apps apart with `CACHELLM_INDEX_NAME` instead.
+
+If either is missing, `auto` uses memory and says exactly why, both in a startup warning and at the top of `cachellm stats`. With `CACHELLM_BACKEND=redis` the proxy never switches storage behind your back. It keeps answering without a cache and reports itself degraded until Redis is fixed.
+
 ## API reference
 
 ### Chat completions
@@ -456,7 +494,7 @@ Every setting is an environment variable prefixed `CACHELLM_`, or a line in `.en
 | Variable | Default | Notes |
 | --- | --- | --- |
 | `CACHELLM_BACKEND` | `auto` | `memory` needs nothing, `redis` shares one cache across workers, `auto` uses Redis when reachable and memory when not |
-| `CACHELLM_REDIS_URL` | `redis://localhost:6379/0` | Must be database 0. Redis Search cannot index any other |
+| `CACHELLM_REDIS_URL` | `redis://localhost:6379/0` | Must be database 0, and the server needs the search module. See [Where the cache lives](#where-the-cache-lives) |
 | `CACHELLM_MEMORY_MAX_ENTRIES` | `50000` | Cap for the in-memory store. 50k of 384-dim vectors is about 73 MB |
 | `CACHELLM_MEMORY_SNAPSHOT_PATH` | unset | Persist the in-memory cache to this file so a restart does not start cold |
 | `CACHELLM_API_KEYS` | empty | Comma-separated client keys. Empty disables auth, which is local development only |
@@ -467,7 +505,7 @@ Every setting is an environment variable prefixed `CACHELLM_`, or a line in `.en
 | `CACHELLM_TTL_VOLATILE` | `900` | Fifteen minutes for anything about now |
 | `CACHELLM_MAX_CACHEABLE_TEMPERATURE` | `0.3` | Above this, nothing is cached |
 | `CACHELLM_PII_GUARD` | `true` | Refuse to store prompts that look personal |
-| `CACHELLM_DEFAULT_PROVIDER` | `bedrock` | `bedrock`, `openai` or `fake` |
+| `CACHELLM_DEFAULT_PROVIDER` | `openai` | `openai` covers every OpenAI-compatible host. Also `bedrock` or `fake`. Left unset, the proxy picks from the keys it finds |
 | `CACHELLM_LOG_PROMPTS` | `false` | Prompt text stays out of logs unless you opt in |
 
 ## Deployment and infrastructure
@@ -536,7 +574,7 @@ cachellm/
 ## Testing
 
 ```bash
-make test          # 126 tests
+make test          # 327 tests
 make test-cov      # with coverage
 make lint          # ruff and mypy
 ```

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "cachellm-proxy"
-version = "0.2.1"
+version = "0.2.2"
 description = "A drop-in semantic cache for OpenAI-compatible LLM APIs: match requests by meaning, serve repeats instantly, cut spend."
 readme = "README.md"
 requires-python = ">=3.11"

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "cachellm-proxy"
-version = "0.2.1"
+version = "0.2.2"
 description = "A drop-in semantic cache for OpenAI-compatible LLM APIs: match requests by meaning, serve repeats instantly, cut spend."
 readme = "README.md"
 authors = [

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/__init__.py

@@ -2,7 +2,7 @@
 
 from __future__ import annotations
 
-__version__ = "0.2.1"
+__version__ = "0.2.2"
 
 __all__ = ["__version__", "main"]
 

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/api/deps.py

@@ -16,8 +16,10 @@ cache that takes an application down when it breaks is worse than no cache.
 
 from __future__ import annotations
 
+import contextlib
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING, Any
+from urllib.parse import urlparse
 
 import structlog
 from fastapi import Request
@@ -46,6 +48,44 @@ REDIS_MISSING = (
 )
 
 
+class RedisUnusableError(RuntimeError):
+    """Redis answered, but it cannot hold this cache.
+
+    Different in kind from "no Redis here". Nobody runs a Redis server by
+    accident, so when one answers and still cannot be used, the operator almost
+    certainly meant to use it and deserves to hear why it was passed over.
+    """
+
+
+def _where(url: str) -> str:
+    """Host and port only. The URL may carry a password, and this gets logged."""
+    parsed = urlparse(url)
+    if parsed.hostname:
+        return f"{parsed.hostname}:{parsed.port or 6379}"
+    return "the configured address"
+
+
+async def _require_search(redis: Any, settings: Settings) -> None:
+    """Fail with a plain explanation before RedisVL fails with a cryptic one."""
+    where = _where(settings.redis_url)
+    db = int(redis.connection_pool.connection_kwargs.get("db", 0) or 0)
+    if db != 0:
+        raise RedisUnusableError(
+            f"Redis at {where} answered, but CACHELLM_REDIS_URL selects database {db}. "
+            "Redis search can only index database 0, so end the URL with /0."
+        )
+    try:
+        await redis.execute_command("FT._LIST")
+    except Exception as exc:
+        if "unknown command" not in str(exc).lower():
+            raise
+        raise RedisUnusableError(
+            f"Redis at {where} answered, but it has no search module, so it cannot "
+            "store vectors. Redis 8 from Homebrew or the official Docker image includes "
+            "it. Many Linux distribution packages do not."
+        ) from exc
+
+
 @dataclass
 class AppState:
     settings: Settings
@@ -62,6 +102,8 @@ class AppState:
     degraded_reason: str = ""
     #: Which backend actually started: "redis" or "memory".
     backend: str = "none"
+    #: Why a reachable Redis was passed over, shown by `cachellm stats`.
+    backend_note: str = ""
 
     @property
     def caching_on(self) -> bool:
@@ -96,8 +138,12 @@ async def build_state(
         except Exception as exc:
             redis_error = f"{type(exc).__name__}: {exc}"
             if settings.backend == "redis":
-                state.degraded_reason = redis_error
-                log.error("redis_unavailable_failing_open", error=redis_error)
+                unusable = isinstance(exc, RedisUnusableError)
+                state.degraded_reason = str(exc) if unusable else redis_error
+                log.error("redis_unavailable_failing_open", error=state.degraded_reason)
+            elif isinstance(exc, RedisUnusableError):
+                state.backend_note = f"{exc} Using the in-process cache instead."
+                log.warning("redis_unusable_using_memory", reason=str(exc))
             else:
                 log.info(
                     "redis_unavailable_using_memory",
@@ -135,9 +181,15 @@ async def _attach_redis(state: AppState, settings: Settings) -> None:
         raise ImportError(REDIS_MISSING) from exc
 
     redis = build_redis(settings)
-    await redis.ping()
-    vectors = VectorStore(redis, settings)
-    await vectors.connect()
+    try:
+        await redis.ping()
+        await _require_search(redis, settings)
+        vectors = VectorStore(redis, settings)
+        await vectors.connect()
+    except Exception:
+        with contextlib.suppress(Exception):
+            await redis.aclose()
+        raise
     state.redis = redis
     _wire(
         state, settings, vectors, ExactStore(redis, settings), Analytics(redis, settings), "redis"

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/api/routes_admin.py

@@ -76,6 +76,8 @@ async def stats(request: Request) -> dict[str, Any]:
     data = await state.cache.stats()
     data["cache_available"] = True
     data["backend"] = state.backend
+    if state.backend_note:
+        data["backend_note"] = state.backend_note
     data["caching_enabled"] = state.settings.enabled
     data["shadow_mode"] = state.settings.shadow_mode
     data["in_flight"] = state.singleflight.in_flight

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/api/routes_chat.py

@@ -32,7 +32,7 @@ from cachellm.models import (
 )
 from cachellm.observability import span
 from cachellm.pricing import estimate_cost
-from cachellm.providers.base import Provider
+from cachellm.providers.base import Provider, StreamEvent
 
 log = structlog.get_logger(__name__)
 router = APIRouter()
@@ -240,6 +240,12 @@ async def chat_completions(request: Request, body: ChatCompletionRequest) -> Any
         return await _proxy_once(state, body, lookup, provider, provider_name, started)
 
 
+async def _count_provider_error(state: AppState, provider_name: str) -> None:
+    state.metrics.provider_errors.labels(provider=provider_name).inc()
+    if state.analytics is not None:
+        await state.analytics.incr("provider_errors")
+
+
 async def _proxy_once(
     state: AppState,
     body: ChatCompletionRequest,
@@ -264,9 +270,7 @@ async def _proxy_once(
         else:
             result, coalesced = await call(), False
     except UpstreamError:
-        state.metrics.provider_errors.labels(provider=provider_name).inc()
-        if state.analytics is not None:
-            await state.analytics.incr("provider_errors")
+        await _count_provider_error(state, provider_name)
         raise
 
     if coalesced:
@@ -361,6 +365,25 @@ async def _proxy_stream(
     """
     stream_id = sse.new_stream_id()
     created = int(time.time())
+    upstream = aiter(provider.stream(body))
+
+    # Hold the response until the upstream sends its first event. A wrong model
+    # name or a bad key fails right here, so the caller gets the upstream's real
+    # status code. Answering first used to turn every such failure into a 200
+    # stream with an empty answer and nothing to say why.
+    try:
+        first: StreamEvent | None = await anext(upstream)
+    except StopAsyncIteration:
+        first = None
+    except UpstreamError:
+        await _count_provider_error(state, provider_name)
+        raise
+
+    async def events() -> AsyncIterator[StreamEvent]:
+        if first is not None:
+            yield first
+        async for event in upstream:
+            yield event
 
     async def generate() -> AsyncIterator[str]:
         buffer: list[str] = []
@@ -369,7 +392,7 @@ async def _proxy_stream(
         completion_tokens = 0
         yield sse.role_chunk(stream_id, body.model, created)
         try:
-            async for event in provider.stream(body):
+            async for event in events():
                 if event.delta:
                     buffer.append(event.delta)
                     yield sse.text_chunk(stream_id, body.model, created, event.delta)
@@ -379,11 +402,9 @@ async def _proxy_stream(
                     prompt_tokens = event.prompt_tokens or prompt_tokens
                     completion_tokens = event.completion_tokens or completion_tokens
         except UpstreamError as exc:
-            state.metrics.provider_errors.labels(provider=provider_name).inc()
-            if state.analytics is not None:
-                await state.analytics.incr("provider_errors")
-            log.warning("stream_failed", error=str(exc)[:200])
-            yield sse.final_chunk(stream_id, body.model, created, "error", None)
+            await _count_provider_error(state, provider_name)
+            log.warning("stream_failed", error=exc.message[:200])
+            yield sse.error_event(exc.message, exc.err_type, exc.code)
             yield sse.DONE
             return
 

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/api/sse.py

@@ -71,6 +71,17 @@ def final_chunk(
     )
 
 
+def error_event(message: str, err_type: str = "api_error", code: str | None = None) -> str:
+    """A failure after the stream has started, in OpenAI's shape.
+
+    The status line has already gone out as 200 by then, so this event is the
+    only way left to say something broke. The official SDKs raise an error when
+    they read it, instead of ending quietly with a half-written answer.
+    """
+    payload = {"error": {"message": message, "type": err_type, "param": None, "code": code}}
+    return f"data: {json.dumps(payload, separators=(',', ':'))}\n\n"
+
+
 def split_for_replay(text: str, max_chars: int = 24) -> list[str]:
     """Chop a cached answer into believable stream chunks.
 

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/embeddings/fastembed_backend.py

@@ -64,12 +64,24 @@ class FastEmbedEmbedder(Embedder):
         return vectors[0]
 
     async def embed_batch(self, texts: list[str]) -> list[np.ndarray]:
+        """Embed many texts, running the model once per distinct uncached text.
+
+        Results are assembled from what was just computed, never read back out
+        of the cache. The cache is bounded, so reading back used to fail two
+        ways: a batch larger than the cache evicted its own first results, and
+        a cache size of 0 made every call fail, which quietly turned caching off.
+        """
         if not texts:
             return []
         model = await self._ensure_model()
-        pending = [t for t in texts if self._cache_get(t) is None]
+        found: dict[str, np.ndarray] = {}
+        for text in texts:
+            if (hit := self._cache_get(text)) is not None:
+                found[text] = hit
+        pending = list(dict.fromkeys(t for t in texts if t not in found))
         if pending:
             raw = await asyncio.to_thread(lambda: list(model.embed(pending)))
             for text, vector in zip(pending, raw, strict=True):
-                self._cache_put(text, self.normalise(np.asarray(vector, dtype=np.float32)))
-        return [self._cache[t] for t in texts]
+                found[text] = self.normalise(np.asarray(vector, dtype=np.float32))
+                self._cache_put(text, found[text])
+        return [found[t] for t in texts]

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/pricing.py

@@ -50,6 +50,18 @@ PRICES: dict[str, ModelPrice] = {
     "gpt-4o-mini": ModelPrice(0.15, 0.60),
     "gpt-4o": ModelPrice(2.50, 10.00),
     "gpt-4.1-mini": ModelPrice(0.40, 1.60),
+    # --- Google Gemini, paid tier, output includes thinking (checked 2026-09-11) ---
+    "gemini-2.5-flash": ModelPrice(0.30, 2.50),
+    "gemini-2.5-flash-lite": ModelPrice(0.10, 0.40),
+    "gemini-2.5-pro": ModelPrice(1.25, 10.00),
+    "gemini-3.5-flash": ModelPrice(1.50, 9.00),
+    "gemini-3.5-flash-lite": ModelPrice(0.30, 2.50),
+    # --- Groq, keyed without the vendor segment (checked 2026-09-11) ---
+    "gpt-oss-20b": ModelPrice(0.075, 0.30),
+    "gpt-oss-120b": ModelPrice(0.15, 0.60),
+    "gpt-oss-safeguard-20b": ModelPrice(0.075, 0.30),
+    "qwen3.6-27b": ModelPrice(0.60, 3.00),
+    "qwen3.8-27b": ModelPrice(0.80, 4.00),
     # --- embeddings (input only) ---
     "amazon.titan-embed-text-v2:0": ModelPrice(0.02, 0.0),
     "text-embedding-3-small": ModelPrice(0.02, 0.0),
@@ -97,10 +109,13 @@ def price_for(model: str) -> ModelPrice:
     key = normalise_model_id(model)
     if key in PRICES:
         return PRICES[key]
-    # Prefix match so undated model revisions still price sensibly.
-    for known, price in PRICES.items():
-        if key.startswith(known.split("-2024")[0].split("-v1:")[0]):
-            return price
+    # Prefix match so dated or preview revisions still price sensibly. The
+    # longest match wins: `gemini-2.5-flash-lite-preview` is a Flash-Lite, and
+    # taking the first match in table order priced it as the dearer Flash.
+    stems = {known.split("-2024")[0].split("-v1:")[0]: known for known in PRICES}
+    matches = [stem for stem in stems if key.startswith(stem)]
+    if matches:
+        return PRICES[stems[max(matches, key=len)]]
     return _FALLBACK
 
 

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/providers/catalog.py

@@ -77,12 +77,10 @@ OPENAI_FAMILIES: tuple[str, ...] = (
 #: Together both use `vendor/model` ids, and stripping their vendor segment
 #: makes the upstream reject the request as an unknown model.
 #:
-#: These three names are therefore reserved. A model id that genuinely begins
-#: `openai/`, `bedrock/` or `fake/` is read as a routing instruction: Groq's
-#: `openai/gpt-oss-120b` reaches the upstream as `gpt-oss-120b`. That is the one
-#: known collision, it affects one model family on one host, and the workaround
-#: is to point CACHELLM_OPENAI_BASE_URL at Groq and send the bare id if the host
-#: accepts it.
+#: A model id starting with one of these always routes to that adapter. What
+#: reaches the upstream is the adapter's call: `openai/` is removed only when the
+#: upstream is OpenAI's own API, because Groq, OpenRouter and Together name
+#: OpenAI's models `openai/gpt-oss-120b` and would reject the bare id.
 ROUTING_PREFIXES: tuple[str, ...] = ("bedrock", "openai", "fake")
 
 
@@ -181,7 +179,7 @@ HOSTS: tuple[Host, ...] = (
         "https://generativelanguage.googleapis.com/v1beta/openai/",
         "GEMINI_API_KEY",
         None,
-        ("gemini-2.5-flash", "gemini-2.0-flash", "gemini-1.5-pro"),
+        ("gemini-2.5-flash", "gemini-2.5-flash-lite", "gemini-3.5-flash"),
         "Also reads GOOGLE_API_KEY. Generous free tier.",
     ),
     Host(
@@ -201,7 +199,7 @@ HOSTS: tuple[Host, ...] = (
         "https://api.groq.com/openai/v1",
         "GROQ_API_KEY",
         None,
-        ("llama-3.3-70b-versatile", "llama-3.1-8b-instant", "gemma2-9b-it"),
+        ("openai/gpt-oss-20b", "openai/gpt-oss-120b", "qwen/qwen3.6-27b"),
         "Very fast, useful free tier.",
     ),
     Host(

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/providers/detect.py

@@ -131,17 +131,21 @@ def apply(settings: Any) -> str | None:
     CACHELLM_OPENAI_API_KEY being set means the decision is already made, and
     detection must not second-guess it.
 
+    "Set" means set anywhere settings are read from, not just exported. A base
+    URL written in a .env file used to be replaced by whichever vendor key
+    happened to be in the shell, because only the process environment was
+    checked. Settings already records which fields came from any source.
+
     Returns a one-line description of what it configured, or None.
     """
-    explicit = any(
-        os.environ.get(name, "").strip()
-        for name in (
-            "CACHELLM_DEFAULT_PROVIDER",
-            "CACHELLM_OPENAI_BASE_URL",
-            "CACHELLM_OPENAI_API_KEY",
-        )
+    fields = ("default_provider", "openai_base_url", "openai_api_key")
+    exported = any(os.environ.get(f"CACHELLM_{name.upper()}", "").strip() for name in fields)
+    # An empty `CACHELLM_OPENAI_API_KEY=` line is a placeholder, not a decision.
+    written = set(getattr(settings, "model_fields_set", ()))
+    configured = any(
+        name in written and str(getattr(settings, name, "") or "").strip() for name in fields
     )
-    if explicit:
+    if exported or configured:
         return None
 
     options = available(include_fake=False)

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/providers/openai_compat.py

@@ -11,6 +11,7 @@ from __future__ import annotations
 import json
 from collections.abc import AsyncIterator
 from typing import Any
+from urllib.parse import urlparse
 
 import httpx
 import structlog
@@ -32,12 +33,32 @@ class OpenAICompatProvider(Provider):
     name = "openai"
 
     def __init__(
-        self, settings: Settings, base_url: str | None = None, api_key: str | None = None
+        self,
+        settings: Settings,
+        base_url: str | None = None,
+        api_key: str | None = None,
+        transport: httpx.AsyncBaseTransport | None = None,
     ) -> None:
         self._settings = settings
         self._base_url = (base_url or settings.openai_base_url).rstrip("/")
         self._api_key = api_key if api_key is not None else settings.openai_api_key
+        # Injectable so tests exercise the real client construction, including
+        # the auth header, against a fake upstream rather than the network.
+        self._transport = transport
         self._client: httpx.AsyncClient | None = None
+        self._is_openai_itself = urlparse(self._base_url).hostname == "api.openai.com"
+
+    def resolve_model(self, model: str) -> str:
+        """Drop our `openai/` routing prefix only when the upstream is OpenAI.
+
+        OpenAI's own API wants `gpt-4o-mini`. Groq, OpenRouter and Together all
+        name OpenAI's models `openai/gpt-oss-120b`, so to them the prefix is part
+        of the model id. Stripping it there turned Groq's two main chat models
+        into 404s, which only showed up against the live API.
+        """
+        if model.startswith("openai/") and not self._is_openai_itself:
+            return model
+        return super().resolve_model(model)
 
     def _http(self) -> httpx.AsyncClient:
         if self._client is None:
@@ -48,6 +69,7 @@ class OpenAICompatProvider(Provider):
                 base_url=self._base_url,
                 headers=headers,
                 timeout=httpx.Timeout(self._settings.request_timeout, connect=10.0),
+                transport=self._transport,
             )
         return self._client
 

{cachellm_proxy-0.2.1 → cachellm_proxy-0.2.2}/src/cachellm/report.py

@@ -12,6 +12,7 @@ from __future__ import annotations
 
 import os
 import sys
+import textwrap
 import time
 from typing import Any
 
@@ -73,7 +74,12 @@ def header(stats: dict[str, Any], embedding_model: str = "") -> list[str]:
         bits.append(f"{MAGENTA}shadow mode{RESET}")
     if not stats.get("caching_enabled", True):
         bits.append(f"{AMBER}caching disabled{RESET}")
-    return ["", "  " + f" {DIM}·{RESET} ".join(bits), ""]
+    lines = ["", "  " + f" {DIM}·{RESET} ".join(bits)]
+    # A reachable Redis that was passed over is worth a line of its own:
+    # whoever started it expected it to be used.
+    if note := stats.get("backend_note"):
+        lines += [f"  {AMBER}{row}{RESET}" for row in textwrap.wrap(str(note), 74)]
+    return [*lines, ""]
 
 
 def summary(stats: dict[str, Any], embedding_model: str = "") -> str: