llm-cachecheck 0.2.1__tar.gz → 0.2.3__tar.gz

{llm_cachecheck-0.2.1 → llm_cachecheck-0.2.3}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.5
 Name: llm-cachecheck
-Version: 0.2.1
-Summary: Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, plus an experimental two-server mode for vLLM and SGLang)
+Version: 0.2.3
+Summary: Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, vLLM and SGLang)
 Project-URL: Homepage, https://github.com/aditi-p31/cache-divergence-study/tree/main/cachecheck
 Project-URL: Source, https://github.com/aditi-p31/cache-divergence-study
 Project-URL: Paper, https://doi.org/10.1109/ACCESS.2026.3749922
@@ -75,23 +75,38 @@ llm-cachecheck --base-url http://127.0.0.1:8080
 
 **vLLM, SGLang or another OpenAI-compatible server.** The cache is a server
 setting, so serve the same model twice, once with prefix caching on and once
-with it off. Both servers must report cached tokens. For vLLM on one GPU:
+with it off. Both servers must report cached tokens. For vLLM on one 24 GB GPU
+(a 3B model, so two copies fit):
 
 ```bash
-vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
+vllm serve Qwen/Qwen2.5-3B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
     --generation-config vllm --enable-prompt-tokens-details
-vllm serve Qwen/Qwen2.5-7B-Instruct --port 8001 --gpu-memory-utilization 0.45 \
+vllm serve Qwen/Qwen2.5-3B-Instruct --port 8100 --gpu-memory-utilization 0.45 \
     --generation-config vllm --enable-prompt-tokens-details --no-enable-prefix-caching
 llm-cachecheck --mode two-servers --base-url http://127.0.0.1:8000 \
-    --nocache-url http://127.0.0.1:8001 --model Qwen/Qwen2.5-7B-Instruct
+    --nocache-url http://127.0.0.1:8100 --model Qwen/Qwen2.5-3B-Instruct
 ```
 
-For SGLang, add `--enable-cache-report` to both servers and
-`--disable-radix-cache` to the second one.
+For SGLang, add `--enable-cache-report` and `--sampling-defaults openai` to
+both servers, and `--disable-radix-cache` to the second one.
 
-The two-server mode is covered by the unit tests against a simulated server.
-It has not yet been run against a real vLLM or SGLang server. If you run it,
-an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
+Make sure each server really started on its own port. If another process
+already holds a port, it can answer for it: RunPod's standard images, for
+example, run an nginx proxy on port 8001 that forwards to port 8000, so both
+URLs would reach the same server.
+The tool then reports the recompute path reusing cached tokens and marks every
+probe invalid, so check the server logs if that happens.
+
+Tested on real servers on 2026-11-11 and 11-12 (UTC; one RTX 4090; the tool's 20
+conversations; details in the [currency note](https://github.com/aditi-p31/cache-divergence-study/blob/main/CURRENCY_CHECK_2026-11.md)):
+
+| Server | Model | Conversations with a different turn-2 answer | Both caches off |
+|---|---|---|---|
+| vLLM 0.33.0, two servers | Qwen2.5-3B-Instruct, BF16 | 14 of 20, in each of 5 runs | 0 of 20 |
+| SGLang 0.5.23, two servers (CUDA graphs disabled) | Qwen2.5-3B-Instruct, BF16 | 12 of 20, in each of 2 runs | 0 of 20 |
+| llama.cpp v0.9.0 | Qwen2.5-7B-Instruct, Q4_K_M | 11 of 20, in each of 2 runs | |
+
+If you run it, an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
 
 Two server processes can also differ for other reasons. To check that, start
 both with prefix caching off and run with `--skip-hit-check`. Every probe
@@ -152,6 +167,13 @@ your results:
 A difference is not a bug in your model. It shows that the answer depends on
 the server's cache state as well as on the request.
 
+## Share your result
+
+At the end of a run the tool prints a short result block. It leaves out your
+server URL and any key. If you would like to share it, open a
+[result issue](https://github.com/aditi-p31/cache-divergence-study/issues/new?template=cachecheck-result.yml)
+and paste the block. Results where no probe differed are just as useful.
+
 ## Background and citation
 
 `cachecheck` adapts the paired cache-on and cache-off design of the study in

{llm_cachecheck-0.2.1 → llm_cachecheck-0.2.3}/README.md

@@ -51,23 +51,38 @@ llm-cachecheck --base-url http://127.0.0.1:8080
 
 **vLLM, SGLang or another OpenAI-compatible server.** The cache is a server
 setting, so serve the same model twice, once with prefix caching on and once
-with it off. Both servers must report cached tokens. For vLLM on one GPU:
+with it off. Both servers must report cached tokens. For vLLM on one 24 GB GPU
+(a 3B model, so two copies fit):
 
 ```bash
-vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
+vllm serve Qwen/Qwen2.5-3B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
     --generation-config vllm --enable-prompt-tokens-details
-vllm serve Qwen/Qwen2.5-7B-Instruct --port 8001 --gpu-memory-utilization 0.45 \
+vllm serve Qwen/Qwen2.5-3B-Instruct --port 8100 --gpu-memory-utilization 0.45 \
     --generation-config vllm --enable-prompt-tokens-details --no-enable-prefix-caching
 llm-cachecheck --mode two-servers --base-url http://127.0.0.1:8000 \
-    --nocache-url http://127.0.0.1:8001 --model Qwen/Qwen2.5-7B-Instruct
+    --nocache-url http://127.0.0.1:8100 --model Qwen/Qwen2.5-3B-Instruct
 ```
 
-For SGLang, add `--enable-cache-report` to both servers and
-`--disable-radix-cache` to the second one.
+For SGLang, add `--enable-cache-report` and `--sampling-defaults openai` to
+both servers, and `--disable-radix-cache` to the second one.
 
-The two-server mode is covered by the unit tests against a simulated server.
-It has not yet been run against a real vLLM or SGLang server. If you run it,
-an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
+Make sure each server really started on its own port. If another process
+already holds a port, it can answer for it: RunPod's standard images, for
+example, run an nginx proxy on port 8001 that forwards to port 8000, so both
+URLs would reach the same server.
+The tool then reports the recompute path reusing cached tokens and marks every
+probe invalid, so check the server logs if that happens.
+
+Tested on real servers on 2026-11-11 and 11-12 (UTC; one RTX 4090; the tool's 20
+conversations; details in the [currency note](https://github.com/aditi-p31/cache-divergence-study/blob/main/CURRENCY_CHECK_2026-11.md)):
+
+| Server | Model | Conversations with a different turn-2 answer | Both caches off |
+|---|---|---|---|
+| vLLM 0.33.0, two servers | Qwen2.5-3B-Instruct, BF16 | 14 of 20, in each of 5 runs | 0 of 20 |
+| SGLang 0.5.23, two servers (CUDA graphs disabled) | Qwen2.5-3B-Instruct, BF16 | 12 of 20, in each of 2 runs | 0 of 20 |
+| llama.cpp v0.9.0 | Qwen2.5-7B-Instruct, Q4_K_M | 11 of 20, in each of 2 runs | |
+
+If you run it, an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
 
 Two server processes can also differ for other reasons. To check that, start
 both with prefix caching off and run with `--skip-hit-check`. Every probe
@@ -128,6 +143,13 @@ your results:
 A difference is not a bug in your model. It shows that the answer depends on
 the server's cache state as well as on the request.
 
+## Share your result
+
+At the end of a run the tool prints a short result block. It leaves out your
+server URL and any key. If you would like to share it, open a
+[result issue](https://github.com/aditi-p31/cache-divergence-study/issues/new?template=cachecheck-result.yml)
+and paste the block. Results where no probe differed are just as useful.
+
 ## Background and citation
 
 `cachecheck` adapts the paired cache-on and cache-off design of the study in

{llm_cachecheck-0.2.1 → llm_cachecheck-0.2.3}/cachecheck.py

@@ -22,7 +22,8 @@ Two modes:
   --mode two-servers  vLLM, SGLang or another OpenAI-compatible stack:
                       --base-url serves with prefix caching on, and
                       --nocache-url serves the same model with it off
-                      (tested against a simulated server only)
+                      (tested on real vLLM 0.33.0 and SGLang 0.5.23
+                      servers; results in the README)
 
 Standard library only. Python 3.8 or newer.
 
@@ -30,7 +31,7 @@ Usage (from PyPI: pip install llm-cachecheck, then run llm-cachecheck
 in place of python cachecheck.py):
   python cachecheck.py --base-url http://127.0.0.1:8080
   python cachecheck.py --mode two-servers --base-url http://127.0.0.1:8000 \
-      --nocache-url http://127.0.0.1:8001 --model Qwen/Qwen2.5-7B-Instruct
+      --nocache-url http://127.0.0.1:8100 --model Qwen/Qwen2.5-3B-Instruct
 
 Exit status: 0 no difference and at least 80% of probes valid, 1 at
 least one valid probe differs, 2 inconclusive.
@@ -41,10 +42,11 @@ import binascii
 import json
 import os
 import sys
+import time
 import urllib.error
 import urllib.request
 
-__version__ = "0.2.1"
+__version__ = "0.2.3"
 
 CITATION = (
     "Patodiya, A. Same Request, Different Answer: Quantization Amplifies "
@@ -54,6 +56,9 @@ CITATION = (
 
 MIN_VALID_SHARE = 0.8
 
+SHARE_URL = ("https://github.com/aditi-p31/cache-divergence-study/issues/new"
+             "?template=cachecheck-result.yml")
+
 # Two-turn probes. Turn 1 asks for a worked answer of a few sentences, so
 # the server writes a useful amount of KV state while generating it.
 # Turn 2 builds on that answer, so it is processed on top of that state.
@@ -305,6 +310,29 @@ def server_props(args):
             "model_file": os.path.basename(props.get("model_path") or "")}
 
 
+def result_block(args, props, summary, when):
+    """A short, copyable summary of the run. It leaves out the server URL
+    and any key, so it is safe to paste in public."""
+    build = props.get("build_info") or "(fill in engine and version)"
+    model = args.model or props.get("model_file") or "(fill in model)"
+    lines = [
+        "---- llm-cachecheck result ----",
+        "tool: llm-cachecheck %s" % __version__,
+        "date (UTC): %s" % when,
+        "mode: %s" % args.mode,
+        "engine build: %s" % build,
+        "model: %s" % model,
+        "GPU or CPU: (fill in)",
+        "probes: %d planned, %d valid, %d differed" % (
+            summary["probes_planned"], summary["probes_valid"],
+            summary["probes_diverged"]),
+        "max tokens: %d, seed: %d, hit check: %s" % (
+            args.max_tokens, args.seed, summary["hit_check"]),
+        "-------------------------------",
+    ]
+    return "\n".join(lines)
+
+
 def status_line(r):
     if not r["valid"]:
         return "inconclusive (%s)" % "; ".join(r["problems"])
@@ -324,8 +352,8 @@ def main(argv=None):
                     default="llamacpp",
                     help="llamacpp (default): one llama.cpp server. "
                          "two-servers: vLLM, SGLang or another OpenAI-"
-                         "compatible stack; tested against a simulated "
-                         "server only, not yet against a real one")
+                         "compatible stack, with a second server that has "
+                         "prefix caching off")
     ap.add_argument("--nocache-url",
                     help="two-servers mode: same model, prefix caching off")
     ap.add_argument("--model", default=None,
@@ -414,6 +442,10 @@ def main(argv=None):
     if args.skip_hit_check:
         print("Cache hits were assumed, not confirmed (--skip-hit-check).")
     print("Report written to %s" % args.out)
+    print()
+    print(result_block(args, props, summary,
+                       time.strftime("%Y-%m-%d %H:%M", time.gmtime())))
+    print("To share this result (optional), open: " + SHARE_URL)
     print("Method: " + CITATION)
 
     if diverged:

{llm_cachecheck-0.2.1 → llm_cachecheck-0.2.3}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "hatchling.build"
 [project]
 name = "llm-cachecheck"
 dynamic = ["version"]
-description = "Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, plus an experimental two-server mode for vLLM and SGLang)"
+description = "Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, vLLM and SGLang)"
 readme = "README.md"
 requires-python = ">=3.8"
 license = "MIT"

{llm_cachecheck-0.2.1 → llm_cachecheck-0.2.3}/test_cachecheck.py

@@ -162,6 +162,16 @@ class CacheCheckTests(unittest.TestCase):
         self.assertIn("DIFFERENT (from character", out)
         self.assertIn("3 of 3 valid probes gave a different", out)
 
+    def test_result_block_is_printed_and_safe_to_share(self):
+        server = self.server(diverge=True)
+        code, report, out = self.run_tool(server)
+        self.assertIn("---- llm-cachecheck result ----", out)
+        self.assertIn("engine build: b-fake", out)
+        self.assertIn("probes: 3 planned, 3 valid, 3 differed", out)
+        self.assertIn(cachecheck.SHARE_URL, out)
+        block = out.split("---- llm-cachecheck result ----")[1]
+        self.assertNotIn(server.url, block)
+
     def test_divergence_in_reasoning_only(self):
         code, report, out = self.run_tool(self.server(diverge_reasoning=True))
         self.assertEqual(code, 1)