llm-cachecheck 0.2.1__tar.gz

llm_cachecheck-0.2.1/.gitignore

@@ -0,0 +1,5 @@
+__pycache__/
+*.pyc
+.venv/
+.DS_Store
+dist/

llm_cachecheck-0.2.1/LICENSE

@@ -0,0 +1,27 @@
+MIT License
+
+Copyright (c) 2026 Aditi Patodiya
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+Scope note: the measurement dataset in this repository (the results/,
+results-repair/ and results-revision/ directories, revision/orders/,
+findings.json, repair_findings.json, and the JSON result files in analysis/) is
+licensed separately under the Creative Commons Attribution 4.0
+International License (CC BY 4.0).

llm_cachecheck-0.2.1/PKG-INFO

@@ -0,0 +1,180 @@
+Metadata-Version: 2.5
+Name: llm-cachecheck
+Version: 0.2.1
+Summary: Measure how often the prompt cache changes greedy answers on your LLM server (llama.cpp, plus an experimental two-server mode for vLLM and SGLang)
+Project-URL: Homepage, https://github.com/aditi-p31/cache-divergence-study/tree/main/cachecheck
+Project-URL: Source, https://github.com/aditi-p31/cache-divergence-study
+Project-URL: Paper, https://doi.org/10.1109/ACCESS.2026.3749922
+Project-URL: Preprint, https://arxiv.org/abs/2609.04748
+Project-URL: Issues, https://github.com/aditi-p31/cache-divergence-study/issues
+Author: Aditi Patodiya
+License-Expression: MIT
+License-File: LICENSE
+Keywords: inference,kv cache,llama.cpp,llm,nondeterminism,prefix caching,prompt cache,reproducibility,sglang,vllm
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+# llm-cachecheck
+
+On PyPI this tool is `llm-cachecheck`. The PyPI package named `cachecheck` is
+a different project.
+
+How often does the prompt cache change your server's answers?
+
+Serving stacks reuse the cached key-value state of a shared prompt prefix. A
+cache hit and a recomputation run different operations, so with greedy
+decoding a small numeric change can flip a token. After that the answer goes
+its own way. This is known behavior: the llama.cpp server README says that
+`cache_prompt` "can cause nondeterministic results". What varies is how often
+it happens on your model, weights and build. `cachecheck` measures that.
+
+It sends short two-turn conversations with greedy decoding and compares the
+second answer on two paths:
+
+- **cache path:** turn 2 reuses the state the server wrote while it generated
+  turn 1
+- **recompute path:** turn 2 is processed from scratch
+
+Each path runs twice. A probe counts as valid only when all of these hold:
+
+- both paths repeat on their own
+- the server's own counts show that the cache hit covered the turn-1 answer
+- the server's own counts show that the recompute path reused nothing
+
+## Run it
+
+Python 3.8 or newer, no dependencies. Install it from PyPI:
+
+```bash
+pip install llm-cachecheck
+llm-cachecheck --help
+```
+
+Or download the single file
+[`cachecheck.py`](https://github.com/aditi-p31/cache-divergence-study/blob/main/cachecheck/cachecheck.py)
+and run `python cachecheck.py` in place of `llm-cachecheck`. Both take the
+same options. A full run
+sends 120 requests.
+
+**llama.cpp.** The cache is switched per request. Start the server with one
+slot, because the default is several slots that share a cache. Turn the
+host-memory prompt cache off to match the study's controlled runs:
+
+```bash
+llama-server -m model.gguf -np 1 --cache-ram 0 --port 8080
+llm-cachecheck --base-url http://127.0.0.1:8080
+```
+
+**vLLM, SGLang or another OpenAI-compatible server.** The cache is a server
+setting, so serve the same model twice, once with prefix caching on and once
+with it off. Both servers must report cached tokens. For vLLM on one GPU:
+
+```bash
+vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
+    --generation-config vllm --enable-prompt-tokens-details
+vllm serve Qwen/Qwen2.5-7B-Instruct --port 8001 --gpu-memory-utilization 0.45 \
+    --generation-config vllm --enable-prompt-tokens-details --no-enable-prefix-caching
+llm-cachecheck --mode two-servers --base-url http://127.0.0.1:8000 \
+    --nocache-url http://127.0.0.1:8001 --model Qwen/Qwen2.5-7B-Instruct
+```
+
+For SGLang, add `--enable-cache-report` to both servers and
+`--disable-radix-cache` to the second one.
+
+The two-server mode is covered by the unit tests against a simulated server.
+It has not yet been run against a real vLLM or SGLang server. If you run it,
+an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
+
+Two server processes can also differ for other reasons. To check that, start
+both with prefix caching off and run with `--skip-hit-check`. Every probe
+should come out the same.
+
+Options: `--n` sets the number of probes (default 20), `--max-tokens` the
+answer length (default 256), `--out` the JSON report path.
+
+Models that think before answering need thinking turned off for this check.
+Their templates often drop past thinking from the history, which shortens the
+cache hit, and the tool marks those probes as inconclusive.
+
+## Reading the result
+
+A real run on Apple Silicon (Metal) with Qwen2.5-3B-Instruct at Q4_K_M,
+llama.cpp b12138, `-np 1 --cache-ram 0`, default options:
+
+```
+probe  1/20: same
+...
+probe 10/20: DIFFERENT (from character 578)
+...
+probe 13/20: DIFFERENT (from character 235)
+...
+probe 15/20: DIFFERENT (from character 376)
+...
+3 of 20 valid probes gave a different turn-2 answer on the cache path.
+```
+
+All 20 probes were valid. On every probe the server reported that the cache
+hit covered all of turn 1 and its answer, and that the recompute path reused
+nothing. The full report (made with a pre-release build of 0.2.0) is in
+[`examples/qwen2.5-3b-q4km_metal_b12138.json`](https://github.com/aditi-p31/cache-divergence-study/blob/main/cachecheck/examples/qwen2.5-3b-q4km_metal_b12138.json).
+It took about 15 minutes on a laptop.
+
+| Exit status | Meaning |
+|---|---|
+| 0 | No valid probe differed, and at least 80% of probes were valid |
+| 1 | At least one valid probe differed |
+| 2 | Inconclusive: too few valid probes, or a request failed |
+
+The JSON report keeps every answer, the finish reasons, token counts, cached
+token counts and the server build.
+
+The rate is specific to these short probes. It is not comparable to the
+episode rates below, where one agent episode chains about ten requests.
+
+## What to do about a difference
+
+For runs that must repeat, turn prefix reuse off and record that setting with
+your results:
+
+- llama.cpp: `"cache_prompt": false` per request, or `--no-cache-prompt` for
+  the server
+- vLLM: `--no-enable-prefix-caching`
+- SGLang: `--disable-radix-cache`
+
+A difference is not a bug in your model. It shows that the answer depends on
+the server's cache state as well as on the request.
+
+## Background and citation
+
+`cachecheck` adapts the paired cache-on and cache-off design of the study in
+[cache-divergence-study repository](https://github.com/aditi-p31/cache-divergence-study).
+The study used Qwen2.5-7B-Instruct on llama.cpp with greedy
+decoding, 80 BFCL multi-turn agent episodes and the host-memory prompt cache
+off. The share of episodes whose trajectory changed between a cache hit and a
+recomputation was 37.4% with F16 weights and 82.5% with Q4_K_M weights, as a
+mean over ten server histories.
+
+If `cachecheck` helps your work, please cite the paper:
+
+```bibtex
+@article{patodiya2026samerequest,
+  title   = {Same Request, Different Answer: Quantization Amplifies
+             Cache-Induced Divergence in LLM Serving},
+  author  = {Patodiya, Aditi},
+  journal = {IEEE Access},
+  year    = {2026},
+  doi     = {10.1109/ACCESS.2026.3749922},
+  note    = {Early Access},
+  url     = {https://doi.org/10.1109/ACCESS.2026.3749922}
+}
+```
+
+MIT License. See `LICENSE`.

llm_cachecheck-0.2.1/README.md

@@ -0,0 +1,156 @@
+# llm-cachecheck
+
+On PyPI this tool is `llm-cachecheck`. The PyPI package named `cachecheck` is
+a different project.
+
+How often does the prompt cache change your server's answers?
+
+Serving stacks reuse the cached key-value state of a shared prompt prefix. A
+cache hit and a recomputation run different operations, so with greedy
+decoding a small numeric change can flip a token. After that the answer goes
+its own way. This is known behavior: the llama.cpp server README says that
+`cache_prompt` "can cause nondeterministic results". What varies is how often
+it happens on your model, weights and build. `cachecheck` measures that.
+
+It sends short two-turn conversations with greedy decoding and compares the
+second answer on two paths:
+
+- **cache path:** turn 2 reuses the state the server wrote while it generated
+  turn 1
+- **recompute path:** turn 2 is processed from scratch
+
+Each path runs twice. A probe counts as valid only when all of these hold:
+
+- both paths repeat on their own
+- the server's own counts show that the cache hit covered the turn-1 answer
+- the server's own counts show that the recompute path reused nothing
+
+## Run it
+
+Python 3.8 or newer, no dependencies. Install it from PyPI:
+
+```bash
+pip install llm-cachecheck
+llm-cachecheck --help
+```
+
+Or download the single file
+[`cachecheck.py`](https://github.com/aditi-p31/cache-divergence-study/blob/main/cachecheck/cachecheck.py)
+and run `python cachecheck.py` in place of `llm-cachecheck`. Both take the
+same options. A full run
+sends 120 requests.
+
+**llama.cpp.** The cache is switched per request. Start the server with one
+slot, because the default is several slots that share a cache. Turn the
+host-memory prompt cache off to match the study's controlled runs:
+
+```bash
+llama-server -m model.gguf -np 1 --cache-ram 0 --port 8080
+llm-cachecheck --base-url http://127.0.0.1:8080
+```
+
+**vLLM, SGLang or another OpenAI-compatible server.** The cache is a server
+setting, so serve the same model twice, once with prefix caching on and once
+with it off. Both servers must report cached tokens. For vLLM on one GPU:
+
+```bash
+vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 --gpu-memory-utilization 0.45 \
+    --generation-config vllm --enable-prompt-tokens-details
+vllm serve Qwen/Qwen2.5-7B-Instruct --port 8001 --gpu-memory-utilization 0.45 \
+    --generation-config vllm --enable-prompt-tokens-details --no-enable-prefix-caching
+llm-cachecheck --mode two-servers --base-url http://127.0.0.1:8000 \
+    --nocache-url http://127.0.0.1:8001 --model Qwen/Qwen2.5-7B-Instruct
+```
+
+For SGLang, add `--enable-cache-report` to both servers and
+`--disable-radix-cache` to the second one.
+
+The two-server mode is covered by the unit tests against a simulated server.
+It has not yet been run against a real vLLM or SGLang server. If you run it,
+an [issue](https://github.com/aditi-p31/cache-divergence-study/issues) with what you saw is welcome.
+
+Two server processes can also differ for other reasons. To check that, start
+both with prefix caching off and run with `--skip-hit-check`. Every probe
+should come out the same.
+
+Options: `--n` sets the number of probes (default 20), `--max-tokens` the
+answer length (default 256), `--out` the JSON report path.
+
+Models that think before answering need thinking turned off for this check.
+Their templates often drop past thinking from the history, which shortens the
+cache hit, and the tool marks those probes as inconclusive.
+
+## Reading the result
+
+A real run on Apple Silicon (Metal) with Qwen2.5-3B-Instruct at Q4_K_M,
+llama.cpp b12138, `-np 1 --cache-ram 0`, default options:
+
+```
+probe  1/20: same
+...
+probe 10/20: DIFFERENT (from character 578)
+...
+probe 13/20: DIFFERENT (from character 235)
+...
+probe 15/20: DIFFERENT (from character 376)
+...
+3 of 20 valid probes gave a different turn-2 answer on the cache path.
+```
+
+All 20 probes were valid. On every probe the server reported that the cache
+hit covered all of turn 1 and its answer, and that the recompute path reused
+nothing. The full report (made with a pre-release build of 0.2.0) is in
+[`examples/qwen2.5-3b-q4km_metal_b12138.json`](https://github.com/aditi-p31/cache-divergence-study/blob/main/cachecheck/examples/qwen2.5-3b-q4km_metal_b12138.json).
+It took about 15 minutes on a laptop.
+
+| Exit status | Meaning |
+|---|---|
+| 0 | No valid probe differed, and at least 80% of probes were valid |
+| 1 | At least one valid probe differed |
+| 2 | Inconclusive: too few valid probes, or a request failed |
+
+The JSON report keeps every answer, the finish reasons, token counts, cached
+token counts and the server build.
+
+The rate is specific to these short probes. It is not comparable to the
+episode rates below, where one agent episode chains about ten requests.
+
+## What to do about a difference
+
+For runs that must repeat, turn prefix reuse off and record that setting with
+your results:
+
+- llama.cpp: `"cache_prompt": false` per request, or `--no-cache-prompt` for
+  the server
+- vLLM: `--no-enable-prefix-caching`
+- SGLang: `--disable-radix-cache`
+
+A difference is not a bug in your model. It shows that the answer depends on
+the server's cache state as well as on the request.
+
+## Background and citation
+
+`cachecheck` adapts the paired cache-on and cache-off design of the study in
+[cache-divergence-study repository](https://github.com/aditi-p31/cache-divergence-study).
+The study used Qwen2.5-7B-Instruct on llama.cpp with greedy
+decoding, 80 BFCL multi-turn agent episodes and the host-memory prompt cache
+off. The share of episodes whose trajectory changed between a cache hit and a
+recomputation was 37.4% with F16 weights and 82.5% with Q4_K_M weights, as a
+mean over ten server histories.
+
+If `cachecheck` helps your work, please cite the paper:
+
+```bibtex
+@article{patodiya2026samerequest,
+  title   = {Same Request, Different Answer: Quantization Amplifies
+             Cache-Induced Divergence in LLM Serving},
+  author  = {Patodiya, Aditi},
+  journal = {IEEE Access},
+  year    = {2026},
+  doi     = {10.1109/ACCESS.2026.3749922},
+  note    = {Early Access},
+  url     = {https://doi.org/10.1109/ACCESS.2026.3749922}
+}
+```
+
+MIT License. See `LICENSE`.