llm-cachecheck 0.2.2__tar.gz → 0.2.3__tar.gz

{llm_cachecheck-0.2.2 → llm_cachecheck-0.2.3}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.5
 Name: llm-cachecheck
-Version: 0.2.2
-Summary: Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, plus an experimental two-server mode for vLLM and SGLang)
+Version: 0.2.3
+Summary: Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, vLLM and SGLang)
 Project-URL: Homepage, https://github.com/aditi-p31/cache-divergence-study/tree/main/cachecheck
 Project-URL: Source, https://github.com/aditi-p31/cache-divergence-study
 Project-URL: Paper, https://doi.org/10.1109/ACCESS.2026.3749922
@@ -75,23 +75,38 @@ llm-cachecheck --base-url http://127.0.0.1:8080
 
 **vLLM, SGLang or another OpenAI-compatible server.** The cache is a server
 setting, so serve the same model twice, once with prefix caching on and once
-with it off. Both servers must report cached tokens. For vLLM on one GPU:
+with it off. Both servers must report cached tokens. For vLLM on one 24 GB GPU
+(a 3B model, so two copies fit):
 
 ```bash
-vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
+vllm serve Qwen/Qwen2.5-3B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
     --generation-config vllm --enable-prompt-tokens-details
-vllm serve Qwen/Qwen2.5-7B-Instruct --port 8001 --gpu-memory-utilization 0.45 \
+vllm serve Qwen/Qwen2.5-3B-Instruct --port 8100 --gpu-memory-utilization 0.45 \
     --generation-config vllm --enable-prompt-tokens-details --no-enable-prefix-caching
 llm-cachecheck --mode two-servers --base-url http://127.0.0.1:8000 \
-    --nocache-url http://127.0.0.1:8001 --model Qwen/Qwen2.5-7B-Instruct
+    --nocache-url http://127.0.0.1:8100 --model Qwen/Qwen2.5-3B-Instruct
 ```
 
-For SGLang, add `--enable-cache-report` to both servers and
-`--disable-radix-cache` to the second one.
+For SGLang, add `--enable-cache-report` and `--sampling-defaults openai` to
+both servers, and `--disable-radix-cache` to the second one.
 
-The two-server mode is covered by the unit tests against a simulated server.
-It has not yet been run against a real vLLM or SGLang server. If you run it,
-an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
+Make sure each server really started on its own port. If another process
+already holds a port, it can answer for it: RunPod's standard images, for
+example, run an nginx proxy on port 8001 that forwards to port 8000, so both
+URLs would reach the same server.
+The tool then reports the recompute path reusing cached tokens and marks every
+probe invalid, so check the server logs if that happens.
+
+Tested on real servers on 2026-11-11 and 11-12 (UTC; one RTX 4090; the tool's 20
+conversations; details in the [currency note](https://github.com/aditi-p31/cache-divergence-study/blob/main/CURRENCY_CHECK_2026-11.md)):
+
+| Server | Model | Conversations with a different turn-2 answer | Both caches off |
+|---|---|---|---|
+| vLLM 0.33.0, two servers | Qwen2.5-3B-Instruct, BF16 | 14 of 20, in each of 5 runs | 0 of 20 |
+| SGLang 0.5.23, two servers (CUDA graphs disabled) | Qwen2.5-3B-Instruct, BF16 | 12 of 20, in each of 2 runs | 0 of 20 |
+| llama.cpp v0.9.0 | Qwen2.5-7B-Instruct, Q4_K_M | 11 of 20, in each of 2 runs | |
+
+If you run it, an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
 
 Two server processes can also differ for other reasons. To check that, start
 both with prefix caching off and run with `--skip-hit-check`. Every probe

{llm_cachecheck-0.2.2 → llm_cachecheck-0.2.3}/README.md

@@ -51,23 +51,38 @@ llm-cachecheck --base-url http://127.0.0.1:8080
 
 **vLLM, SGLang or another OpenAI-compatible server.** The cache is a server
 setting, so serve the same model twice, once with prefix caching on and once
-with it off. Both servers must report cached tokens. For vLLM on one GPU:
+with it off. Both servers must report cached tokens. For vLLM on one 24 GB GPU
+(a 3B model, so two copies fit):
 
 ```bash
-vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
+vllm serve Qwen/Qwen2.5-3B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
     --generation-config vllm --enable-prompt-tokens-details
-vllm serve Qwen/Qwen2.5-7B-Instruct --port 8001 --gpu-memory-utilization 0.45 \
+vllm serve Qwen/Qwen2.5-3B-Instruct --port 8100 --gpu-memory-utilization 0.45 \
     --generation-config vllm --enable-prompt-tokens-details --no-enable-prefix-caching
 llm-cachecheck --mode two-servers --base-url http://127.0.0.1:8000 \
-    --nocache-url http://127.0.0.1:8001 --model Qwen/Qwen2.5-7B-Instruct
+    --nocache-url http://127.0.0.1:8100 --model Qwen/Qwen2.5-3B-Instruct
 ```
 
-For SGLang, add `--enable-cache-report` to both servers and
-`--disable-radix-cache` to the second one.
+For SGLang, add `--enable-cache-report` and `--sampling-defaults openai` to
+both servers, and `--disable-radix-cache` to the second one.
 
-The two-server mode is covered by the unit tests against a simulated server.
-It has not yet been run against a real vLLM or SGLang server. If you run it,
-an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
+Make sure each server really started on its own port. If another process
+already holds a port, it can answer for it: RunPod's standard images, for
+example, run an nginx proxy on port 8001 that forwards to port 8000, so both
+URLs would reach the same server.
+The tool then reports the recompute path reusing cached tokens and marks every
+probe invalid, so check the server logs if that happens.
+
+Tested on real servers on 2026-11-11 and 11-12 (UTC; one RTX 4090; the tool's 20
+conversations; details in the [currency note](https://github.com/aditi-p31/cache-divergence-study/blob/main/CURRENCY_CHECK_2026-11.md)):
+
+| Server | Model | Conversations with a different turn-2 answer | Both caches off |
+|---|---|---|---|
+| vLLM 0.33.0, two servers | Qwen2.5-3B-Instruct, BF16 | 14 of 20, in each of 5 runs | 0 of 20 |
+| SGLang 0.5.23, two servers (CUDA graphs disabled) | Qwen2.5-3B-Instruct, BF16 | 12 of 20, in each of 2 runs | 0 of 20 |
+| llama.cpp v0.9.0 | Qwen2.5-7B-Instruct, Q4_K_M | 11 of 20, in each of 2 runs | |
+
+If you run it, an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
 
 Two server processes can also differ for other reasons. To check that, start
 both with prefix caching off and run with `--skip-hit-check`. Every probe

{llm_cachecheck-0.2.2 → llm_cachecheck-0.2.3}/cachecheck.py

@@ -22,7 +22,8 @@ Two modes:
   --mode two-servers  vLLM, SGLang or another OpenAI-compatible stack:
                       --base-url serves with prefix caching on, and
                       --nocache-url serves the same model with it off
-                      (tested against a simulated server only)
+                      (tested on real vLLM 0.33.0 and SGLang 0.5.23
+                      servers; results in the README)
 
 Standard library only. Python 3.8 or newer.
 
@@ -30,7 +31,7 @@ Usage (from PyPI: pip install llm-cachecheck, then run llm-cachecheck
 in place of python cachecheck.py):
   python cachecheck.py --base-url http://127.0.0.1:8080
   python cachecheck.py --mode two-servers --base-url http://127.0.0.1:8000 \
-      --nocache-url http://127.0.0.1:8001 --model Qwen/Qwen2.5-7B-Instruct
+      --nocache-url http://127.0.0.1:8100 --model Qwen/Qwen2.5-3B-Instruct
 
 Exit status: 0 no difference and at least 80% of probes valid, 1 at
 least one valid probe differs, 2 inconclusive.
@@ -45,7 +46,7 @@ import time
 import urllib.error
 import urllib.request
 
-__version__ = "0.2.2"
+__version__ = "0.2.3"
 
 CITATION = (
     "Patodiya, A. Same Request, Different Answer: Quantization Amplifies "
@@ -351,8 +352,8 @@ def main(argv=None):
                     default="llamacpp",
                     help="llamacpp (default): one llama.cpp server. "
                          "two-servers: vLLM, SGLang or another OpenAI-"
-                         "compatible stack; tested against a simulated "
-                         "server only, not yet against a real one")
+                         "compatible stack, with a second server that has "
+                         "prefix caching off")
     ap.add_argument("--nocache-url",
                     help="two-servers mode: same model, prefix caching off")
     ap.add_argument("--model", default=None,

{llm_cachecheck-0.2.2 → llm_cachecheck-0.2.3}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "hatchling.build"
 [project]
 name = "llm-cachecheck"
 dynamic = ["version"]
-description = "Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, plus an experimental two-server mode for vLLM and SGLang)"
+description = "Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, vLLM and SGLang)"
 readme = "README.md"
 requires-python = ">=3.8"
 license = "MIT"