kvcache-bench 0.1.0__tar.gz

kvcache_bench-0.1.0/LICENSE

@@ -0,0 +1,15 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

kvcache_bench-0.1.0/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: kvcache-bench
+Version: 0.1.0
+Summary: Benchmark every KV cache compression method on your GPU. One command, real numbers.
+Author: back2matching
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/back2matching/kvcache-bench
+Keywords: llm,kv-cache,benchmark,vram,gpu,ollama,llama-cpp,quantization
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28.0
+Requires-Dist: pynvml>=11.5.0
+Provides-Extra: charts
+Requires-Dist: matplotlib; extra == "charts"
+Provides-Extra: hf
+Requires-Dist: torch; extra == "hf"
+Requires-Dist: transformers; extra == "hf"
+Dynamic: license-file
+
+# kvcache-bench
+
+Benchmark every KV cache compression method on your GPU. One command, real numbers.
+
+```
+kvcache-bench --model qwen3.5:9b
+```
+
+```
+| KV Type | Context | Prompt    | Gen tok/s | Prefill tok/s | VRAM +MB | Quality |
+|---------|---------|-----------|-----------|---------------|----------|---------|
+| f16     | 4096    | short     | 80.1      | 712.3         | +142     | PASS    |
+| q8_0    | 4096    | short     | 79.5      | 723.5         | +71      | PASS    |
+| q4_0    | 4096    | short     | 78.2      | 698.1         | +36      | PASS    |
+```
+
+## Why
+
+When you run a local LLM, the KV cache eats your VRAM. Ollama and llama.cpp support different KV cache quantization types (f16, q8_0, q4_0), but nobody tells you what the actual tradeoff is on YOUR hardware.
+
+Current state of the world:
+- You Google "ollama kv cache quantization" and find forum posts with conflicting advice
+- You manually test each config, eyeball nvidia-smi, and guess
+- No tool compares them systematically
+
+kvcache-bench fixes this. It tests every KV cache type on your GPU and gives you a comparison table with speed, VRAM, and quality.
+
+## Install
+
+```bash
+pip install kvcache-bench
+```
+
+## Usage
+
+```bash
+# Auto-detect your first model, test all KV types
+kvcache-bench
+
+# Specific model
+kvcache-bench --model qwen3.5:9b
+
+# Test at multiple context lengths (where KV savings matter most)
+kvcache-bench --model llama3.1:8b --context 4096,8192,16384
+
+# Include tool calling test
+kvcache-bench --model qwen3.5:9b --prompts short,code,reasoning,tool_call
+
+# Save results as JSON
+kvcache-bench --model qwen3.5:9b --json results.json
+
+# Just show GPU info
+kvcache-bench --gpu
+
+# List available models
+kvcache-bench --list-models
+```
+
+## What It Tests
+
+For each KV cache type (f16, q8_0, q4_0), it measures:
+
+| Metric | How |
+|--------|-----|
+| **Generation speed** | Tokens per second during generation |
+| **Prefill speed** | Tokens per second processing the prompt |
+| **VRAM delta** | Extra VRAM used beyond model weights (measured via nvidia-smi) |
+| **Quality** | Auto-checked against expected answers (Paris, code structure, reasoning) |
+
+## How It Works
+
+1. Detects your GPU and Ollama installation
+2. For each KV cache type: restarts Ollama with `OLLAMA_KV_CACHE_TYPE=<type>`, warms up the model, runs benchmark prompts
+3. Measures VRAM before and during inference via nvidia-smi
+4. Extracts timing from Ollama's API response (prompt_eval_duration, eval_duration)
+5. Checks response quality with simple auto-graders
+6. Produces a markdown table (and optional JSON)
+
+## What the Research Says
+
+Based on llama.cpp community benchmarks and our testing:
+
+| KV Type | VRAM Savings | Perplexity Impact | Best For |
+|---------|-------------|-------------------|----------|
+| f16 | Baseline | None | When you have VRAM to spare |
+| q8_0 | 2x | +0.004 (negligible) | **Default recommendation.** Free VRAM, zero quality cost. |
+| q4_0 | 4x | +0.2 (noticeable) | When you need max context length or are VRAM-constrained |
+
+The sweet spot for most users: **q8_0**. Halves your KV cache VRAM with essentially zero quality loss.
+
+## Requirements
+
+- Python 3.10+
+- NVIDIA GPU with nvidia-smi
+- Ollama installed and running
+
+## Roadmap
+
+- [ ] Mixed K/V types (q8 keys + q4 values)
+- [ ] Context length sweep charts
+- [ ] HuggingFace backend (vLLM, TGI)
+- [ ] TurboQuant integration
+- [ ] Multi-model matrix
+- [ ] HuggingFace Spaces leaderboard
+- [ ] Community result submissions
+
+## License
+
+Apache 2.0
+
+## Related
+
+- [turboquant](https://github.com/back2matching/turboquant) -- TurboQuant KV cache compression (sub-4-bit)
+- [NVIDIA kvpress](https://github.com/NVIDIA/kvpress) -- KV cache eviction/pruning methods
+- [llama.cpp](https://github.com/ggml-org/llama.cpp) -- Where KV cache quantization lives

kvcache_bench-0.1.0/README.md

@@ -0,0 +1,115 @@
+# kvcache-bench
+
+Benchmark every KV cache compression method on your GPU. One command, real numbers.
+
+```
+kvcache-bench --model qwen3.5:9b
+```
+
+```
+| KV Type | Context | Prompt    | Gen tok/s | Prefill tok/s | VRAM +MB | Quality |
+|---------|---------|-----------|-----------|---------------|----------|---------|
+| f16     | 4096    | short     | 80.1      | 712.3         | +142     | PASS    |
+| q8_0    | 4096    | short     | 79.5      | 723.5         | +71      | PASS    |
+| q4_0    | 4096    | short     | 78.2      | 698.1         | +36      | PASS    |
+```
+
+## Why
+
+When you run a local LLM, the KV cache eats your VRAM. Ollama and llama.cpp support different KV cache quantization types (f16, q8_0, q4_0), but nobody tells you what the actual tradeoff is on YOUR hardware.
+
+Current state of the world:
+- You Google "ollama kv cache quantization" and find forum posts with conflicting advice
+- You manually test each config, eyeball nvidia-smi, and guess
+- No tool compares them systematically
+
+kvcache-bench fixes this. It tests every KV cache type on your GPU and gives you a comparison table with speed, VRAM, and quality.
+
+## Install
+
+```bash
+pip install kvcache-bench
+```
+
+## Usage
+
+```bash
+# Auto-detect your first model, test all KV types
+kvcache-bench
+
+# Specific model
+kvcache-bench --model qwen3.5:9b
+
+# Test at multiple context lengths (where KV savings matter most)
+kvcache-bench --model llama3.1:8b --context 4096,8192,16384
+
+# Include tool calling test
+kvcache-bench --model qwen3.5:9b --prompts short,code,reasoning,tool_call
+
+# Save results as JSON
+kvcache-bench --model qwen3.5:9b --json results.json
+
+# Just show GPU info
+kvcache-bench --gpu
+
+# List available models
+kvcache-bench --list-models
+```
+
+## What It Tests
+
+For each KV cache type (f16, q8_0, q4_0), it measures:
+
+| Metric | How |
+|--------|-----|
+| **Generation speed** | Tokens per second during generation |
+| **Prefill speed** | Tokens per second processing the prompt |
+| **VRAM delta** | Extra VRAM used beyond model weights (measured via nvidia-smi) |
+| **Quality** | Auto-checked against expected answers (Paris, code structure, reasoning) |
+
+## How It Works
+
+1. Detects your GPU and Ollama installation
+2. For each KV cache type: restarts Ollama with `OLLAMA_KV_CACHE_TYPE=<type>`, warms up the model, runs benchmark prompts
+3. Measures VRAM before and during inference via nvidia-smi
+4. Extracts timing from Ollama's API response (prompt_eval_duration, eval_duration)
+5. Checks response quality with simple auto-graders
+6. Produces a markdown table (and optional JSON)
+
+## What the Research Says
+
+Based on llama.cpp community benchmarks and our testing:
+
+| KV Type | VRAM Savings | Perplexity Impact | Best For |
+|---------|-------------|-------------------|----------|
+| f16 | Baseline | None | When you have VRAM to spare |
+| q8_0 | 2x | +0.004 (negligible) | **Default recommendation.** Free VRAM, zero quality cost. |
+| q4_0 | 4x | +0.2 (noticeable) | When you need max context length or are VRAM-constrained |
+
+The sweet spot for most users: **q8_0**. Halves your KV cache VRAM with essentially zero quality loss.
+
+## Requirements
+
+- Python 3.10+
+- NVIDIA GPU with nvidia-smi
+- Ollama installed and running
+
+## Roadmap
+
+- [ ] Mixed K/V types (q8 keys + q4 values)
+- [ ] Context length sweep charts
+- [ ] HuggingFace backend (vLLM, TGI)
+- [ ] TurboQuant integration
+- [ ] Multi-model matrix
+- [ ] HuggingFace Spaces leaderboard
+- [ ] Community result submissions
+
+## License
+
+Apache 2.0
+
+## Related
+
+- [turboquant](https://github.com/back2matching/turboquant) -- TurboQuant KV cache compression (sub-4-bit)
+- [NVIDIA kvpress](https://github.com/NVIDIA/kvpress) -- KV cache eviction/pruning methods
+- [llama.cpp](https://github.com/ggml-org/llama.cpp) -- Where KV cache quantization lives

kvcache_bench-0.1.0/kvcache_bench/__init__.py

@@ -0,0 +1,10 @@
+"""
+kvcache-bench: Benchmark every KV cache compression method on your GPU.
+
+Usage:
+    kvcache-bench --model qwen3.5:9b
+    kvcache-bench --model llama3.1:8b --context 8192,16384,32768
+    kvcache-bench --all-types --json results.json
+"""
+
+__version__ = "0.1.0"

kvcache_bench-0.1.0/kvcache_bench/bench.py

@@ -0,0 +1,256 @@
+"""Core benchmark engine: runs comparison across KV cache types."""
+
+import time
+import json
+import os
+import subprocess
+from dataclasses import dataclass, asdict, field
+from typing import Optional
+from kvcache_bench.gpu import detect_gpu, measure_vram, VramTracker
+from kvcache_bench.ollama import (
+    check_ollama, list_models, run_inference, run_chat,
+    BENCH_PROMPTS, BENCH_TOOL, KV_TYPES, MIXED_KV,
+)
+
+
+@dataclass
+class BenchResult:
+    model: str
+    kv_type: str  # "f16", "q8_0", "q4_0", or "q8_0/q4_0" for mixed
+    context_length: int
+    prompt_type: str
+    prompt_tokens: int
+    generated_tokens: int
+    prompt_eval_rate: float  # tok/s
+    eval_rate: float  # tok/s
+    vram_baseline_mb: int
+    vram_peak_mb: int
+    vram_delta_mb: int
+    total_time_s: float
+    response_preview: str
+    correct: Optional[bool] = None  # For quality checks
+    error: Optional[str] = None
+
+
+def set_kv_cache_env(kv_type_k: str, kv_type_v: str):
+    """Set Ollama KV cache environment variables. Requires Ollama restart."""
+    os.environ["OLLAMA_KV_CACHE_TYPE"] = kv_type_k
+    # Note: Ollama doesn't support separate K/V types via env vars.
+    # For mixed K/V testing, we'd need llama.cpp server directly.
+    # For now, both K and V use the same type.
+
+
+def restart_ollama_with_kv(kv_type: str) -> bool:
+    """
+    Restart Ollama with a new KV cache type.
+
+    This is the key insight: Ollama's KV cache type is server-level,
+    so we must restart between tests. We kill, set env, restart.
+    """
+    # On Windows, Ollama runs as a tray app or service
+    try:
+        # Kill existing Ollama
+        if os.name == 'nt':
+            subprocess.run(["taskkill", "/F", "/IM", "ollama.exe"], capture_output=True, timeout=5)
+            subprocess.run(["taskkill", "/F", "/IM", "ollama app.exe"], capture_output=True, timeout=5)
+        else:
+            subprocess.run(["pkill", "-f", "ollama"], capture_output=True, timeout=5)
+
+        time.sleep(3)
+
+        # Set env vars
+        os.environ["OLLAMA_FLASH_ATTENTION"] = "1"
+        os.environ["OLLAMA_KV_CACHE_TYPE"] = kv_type
+        os.environ["OLLAMA_NUM_PARALLEL"] = "1"  # Consistent for benchmarking
+
+        # Start Ollama
+        if os.name == 'nt':
+            ollama_path = os.path.expandvars(r"%LOCALAPPDATA%\Programs\Ollama\ollama.exe")
+            if not os.path.exists(ollama_path):
+                ollama_path = "ollama"  # Fall back to PATH
+            subprocess.Popen(
+                [ollama_path, "serve"],
+                env={**os.environ},
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL,
+            )
+        else:
+            subprocess.Popen(
+                ["ollama", "serve"],
+                env={**os.environ},
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL,
+            )
+
+        # Wait for it to come up
+        for _ in range(30):
+            time.sleep(1)
+            if check_ollama():
+                return True
+        return False
+
+    except Exception as e:
+        print(f"  Failed to restart Ollama: {e}")
+        return False
+
+
+def check_quality(prompt_type: str, response: str) -> Optional[bool]:
+    """Simple quality checks for standard prompts."""
+    # Strip thinking tags (Qwen3.5 uses <think>...</think>)
+    import re
+    clean = re.sub(r'<think>.*?</think>', '', response, flags=re.DOTALL).strip()
+    if not clean:
+        clean = response  # Fallback to full response
+    clean_lower = clean.lower().strip()
+
+    if prompt_type == "short":
+        return "paris" in clean_lower
+    elif prompt_type == "reasoning":
+        return "9" in clean
+    elif prompt_type == "code":
+        return "def " in clean and "return" in clean
+    return None  # Can't auto-check
+
+
+def run_single_bench(
+    model: str,
+    kv_type: str,
+    prompt_type: str,
+    context_length: int = 4096,
+) -> BenchResult:
+    """Run a single benchmark measurement."""
+    prompt = BENCH_PROMPTS.get(prompt_type, prompt_type)
+
+    tracker = VramTracker()
+    tracker.start()
+
+    t0 = time.perf_counter()
+
+    if prompt_type == "tool_call":
+        result = run_chat(
+            model,
+            [{"role": "user", "content": prompt}],
+            num_ctx=context_length,
+            max_tokens=100,
+            tools=[BENCH_TOOL],
+        )
+    else:
+        result = run_inference(model, prompt, num_ctx=context_length, max_tokens=150)
+
+    elapsed = time.perf_counter() - t0
+    vram = tracker.stop()
+
+    if not result or "error" in result:
+        return BenchResult(
+            model=model, kv_type=kv_type, context_length=context_length,
+            prompt_type=prompt_type, prompt_tokens=0, generated_tokens=0,
+            prompt_eval_rate=0, eval_rate=0,
+            vram_baseline_mb=vram["baseline_mb"], vram_peak_mb=vram["peak_mb"],
+            vram_delta_mb=vram["delta_mb"], total_time_s=elapsed,
+            response_preview="", error=result.get("error", "Unknown error"),
+        )
+
+    response_text = result.get("response", "")
+    if not response_text and "message" in result:
+        msg = result["message"]
+        if isinstance(msg, dict):
+            response_text = msg.get("content", "")
+            if msg.get("tool_calls"):
+                response_text = json.dumps(msg["tool_calls"][0]["function"])
+
+    prompt_eval_count = result.get("prompt_eval_count", 0)
+    eval_count = result.get("eval_count", 0)
+    prompt_eval_dur = result.get("prompt_eval_duration", 1) / 1e9  # ns to s
+    eval_dur = result.get("eval_duration", 1) / 1e9
+
+    return BenchResult(
+        model=model,
+        kv_type=kv_type,
+        context_length=context_length,
+        prompt_type=prompt_type,
+        prompt_tokens=prompt_eval_count,
+        generated_tokens=eval_count,
+        prompt_eval_rate=round(prompt_eval_count / prompt_eval_dur, 1) if prompt_eval_dur > 0 else 0,
+        eval_rate=round(eval_count / eval_dur, 1) if eval_dur > 0 else 0,
+        vram_baseline_mb=vram["baseline_mb"],
+        vram_peak_mb=vram["peak_mb"],
+        vram_delta_mb=vram["delta_mb"],
+        total_time_s=round(elapsed, 2),
+        response_preview=response_text[:200],
+        correct=check_quality(prompt_type, response_text),
+    )
+
+
+def run_full_benchmark(
+    model: str,
+    kv_types: list[str] = None,
+    context_lengths: list[int] = None,
+    prompt_types: list[str] = None,
+    auto_restart: bool = True,
+) -> list[BenchResult]:
+    """Run full benchmark matrix."""
+    if kv_types is None:
+        kv_types = KV_TYPES
+    if context_lengths is None:
+        context_lengths = [4096]
+    if prompt_types is None:
+        prompt_types = ["short", "code", "reasoning"]
+
+    results = []
+    gpu = detect_gpu()
+
+    print(f"\n{'='*70}")
+    print(f"KVCache-Bench v0.1.0")
+    print(f"{'='*70}")
+    if gpu:
+        print(f"GPU: {gpu.name} ({gpu.vram_total_mb} MB VRAM)")
+    print(f"Model: {model}")
+    print(f"KV types: {', '.join(kv_types)}")
+    print(f"Context lengths: {', '.join(str(c) for c in context_lengths)}")
+    print(f"Prompts: {', '.join(prompt_types)}")
+    print(f"{'='*70}\n")
+
+    for kv_type in kv_types:
+        print(f"\n--- KV type: {kv_type} ---")
+
+        if auto_restart:
+            print(f"  Restarting Ollama with OLLAMA_KV_CACHE_TYPE={kv_type}...")
+            if not restart_ollama_with_kv(kv_type):
+                print(f"  FAILED to restart Ollama. Skipping.")
+                continue
+            # Warm up: load model
+            print(f"  Warming up model...")
+            run_inference(model, "Hi", num_ctx=512, max_tokens=1)
+            time.sleep(2)
+
+        for ctx in context_lengths:
+            for pt in prompt_types:
+                print(f"  [{kv_type}] ctx={ctx}, prompt={pt}...", end=" ", flush=True)
+                result = run_single_bench(model, kv_type, pt, ctx)
+                results.append(result)
+
+                if result.error:
+                    print(f"ERROR: {result.error}")
+                else:
+                    quality = "?" if result.correct is None else ("PASS" if result.correct else "FAIL")
+                    print(f"{result.eval_rate} tok/s, VRAM +{result.vram_delta_mb}MB, quality={quality}")
+
+    return results
+
+
+def format_results_table(results: list[BenchResult]) -> str:
+    """Format results as a markdown table."""
+    lines = []
+    lines.append("")
+    lines.append(f"| KV Type | Context | Prompt | Gen tok/s | Prefill tok/s | VRAM +MB | Quality |")
+    lines.append(f"|---------|---------|--------|-----------|---------------|----------|---------|")
+
+    for r in results:
+        if r.error:
+            lines.append(f"| {r.kv_type} | {r.context_length} | {r.prompt_type} | ERROR | - | - | {r.error[:30]} |")
+        else:
+            q = "?" if r.correct is None else ("PASS" if r.correct else "FAIL")
+            lines.append(f"| {r.kv_type} | {r.context_length} | {r.prompt_type} | {r.eval_rate} | {r.prompt_eval_rate} | +{r.vram_delta_mb} | {q} |")
+
+    lines.append("")
+    return "\n".join(lines)

kvcache_bench-0.1.0/kvcache_bench/cli.py

@@ -0,0 +1,124 @@
+"""CLI entry point: kvcache-bench command."""
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from kvcache_bench.gpu import detect_gpu
+from kvcache_bench.ollama import check_ollama, list_models, KV_TYPES
+from kvcache_bench.bench import run_full_benchmark, format_results_table
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="kvcache-bench",
+        description="Benchmark every KV cache compression method on your GPU.",
+    )
+    parser.add_argument("--model", "-m", help="Ollama model name (e.g., qwen3.5:9b)")
+    parser.add_argument("--context", "-c", default="4096",
+                        help="Context lengths, comma-separated (default: 4096)")
+    parser.add_argument("--types", "-t", default=",".join(KV_TYPES),
+                        help=f"KV cache types to test (default: {','.join(KV_TYPES)})")
+    parser.add_argument("--prompts", "-p", default="short,code,reasoning",
+                        help="Prompt types: short,code,reasoning,long,tool_call")
+    parser.add_argument("--json", "-j", help="Save results to JSON file")
+    parser.add_argument("--no-restart", action="store_true",
+                        help="Don't restart Ollama between tests (use current KV type)")
+    parser.add_argument("--list-models", action="store_true", help="List available Ollama models")
+    parser.add_argument("--gpu", action="store_true", help="Show GPU info and exit")
+    args = parser.parse_args()
+
+    # GPU info
+    if args.gpu:
+        gpu = detect_gpu()
+        if gpu:
+            print(f"GPU: {gpu.name}")
+            print(f"VRAM: {gpu.vram_used_mb}/{gpu.vram_total_mb} MB ({gpu.vram_free_mb} MB free)")
+            print(f"Driver: {gpu.driver_version}")
+        else:
+            print("No NVIDIA GPU detected.")
+        return
+
+    # Check Ollama
+    if not check_ollama():
+        print("Ollama is not running. Start it with: ollama serve")
+        sys.exit(1)
+
+    # List models
+    if args.list_models:
+        models = list_models()
+        if models:
+            print("Available models:")
+            for m in models:
+                print(f"  - {m}")
+        else:
+            print("No models found. Pull one with: ollama pull qwen3.5:9b")
+        return
+
+    # Need a model
+    if not args.model:
+        models = list_models()
+        if models:
+            args.model = models[0]
+            print(f"No model specified, using: {args.model}")
+        else:
+            print("No model specified and no models available.")
+            print("Usage: kvcache-bench --model qwen3.5:9b")
+            sys.exit(1)
+
+    # Parse args
+    context_lengths = [int(c) for c in args.context.split(",")]
+    kv_types = args.types.split(",")
+    prompt_types = args.prompts.split(",")
+
+    # Run
+    from dataclasses import asdict
+    results = run_full_benchmark(
+        model=args.model,
+        kv_types=kv_types,
+        context_lengths=context_lengths,
+        prompt_types=prompt_types,
+        auto_restart=not args.no_restart,
+    )
+
+    # Output
+    table = format_results_table(results)
+    print(table)
+
+    # Save JSON
+    if args.json:
+        out_path = Path(args.json)
+        with open(out_path, "w") as f:
+            json.dump([asdict(r) for r in results], f, indent=2)
+        print(f"\nResults saved to {out_path}")
+
+    # Summary
+    if results:
+        gpu = detect_gpu()
+        print(f"\nGPU: {gpu.name if gpu else 'Unknown'}")
+        print(f"Model: {args.model}")
+        print(f"Tests: {len(results)} ({len(kv_types)} types x {len(context_lengths)} contexts x {len(prompt_types)} prompts)")
+
+        # Find best config
+        valid = [r for r in results if not r.error and r.eval_rate > 0]
+        if valid:
+            fastest = max(valid, key=lambda r: r.eval_rate)
+            lowest_vram = min(valid, key=lambda r: r.vram_delta_mb)
+            print(f"\nFastest: {fastest.kv_type} ({fastest.eval_rate} tok/s)")
+            print(f"Lowest VRAM: {lowest_vram.kv_type} (+{lowest_vram.vram_delta_mb} MB)")
+
+            # Recommendation
+            # q8_0 is usually the sweet spot (negligible quality loss, good VRAM savings)
+            q8_results = [r for r in valid if r.kv_type == "q8_0"]
+            if q8_results:
+                q8_avg_rate = sum(r.eval_rate for r in q8_results) / len(q8_results)
+                f16_results = [r for r in valid if r.kv_type == "f16"]
+                if f16_results:
+                    f16_avg_rate = sum(r.eval_rate for r in f16_results) / len(f16_results)
+                    speed_diff = abs(q8_avg_rate - f16_avg_rate) / f16_avg_rate * 100
+                    if speed_diff < 5:
+                        print(f"\nRecommendation: Use q8_0. Near-zero speed difference ({speed_diff:.1f}%) with 2x VRAM savings.")
+
+
+if __name__ == "__main__":
+    main()

kvcache_bench-0.1.0/kvcache_bench/gpu.py

@@ -0,0 +1,79 @@
+"""GPU detection and VRAM monitoring."""
+
+import subprocess
+import time
+from dataclasses import dataclass
+from typing import Optional
+
+
+@dataclass
+class GpuInfo:
+    name: str
+    vram_total_mb: int
+    vram_used_mb: int
+    vram_free_mb: int
+    driver_version: str
+
+
+def detect_gpu() -> Optional[GpuInfo]:
+    """Detect NVIDIA GPU via nvidia-smi."""
+    try:
+        result = subprocess.run(
+            ["nvidia-smi", "--query-gpu=name,memory.total,memory.used,memory.free,driver_version",
+             "--format=csv,noheader,nounits"],
+            capture_output=True, text=True, timeout=5,
+        )
+        if result.returncode != 0:
+            return None
+        parts = result.stdout.strip().split(",")
+        if len(parts) < 5:
+            return None
+        return GpuInfo(
+            name=parts[0].strip(),
+            vram_total_mb=int(float(parts[1].strip())),
+            vram_used_mb=int(float(parts[2].strip())),
+            vram_free_mb=int(float(parts[3].strip())),
+            driver_version=parts[4].strip(),
+        )
+    except (FileNotFoundError, subprocess.TimeoutExpired):
+        return None
+
+
+def measure_vram() -> int:
+    """Get current VRAM usage in MB."""
+    try:
+        result = subprocess.run(
+            ["nvidia-smi", "--query-gpu=memory.used", "--format=csv,noheader,nounits"],
+            capture_output=True, text=True, timeout=5,
+        )
+        return int(float(result.stdout.strip()))
+    except Exception:
+        return 0
+
+
+class VramTracker:
+    """Track VRAM usage over a time period."""
+
+    def __init__(self):
+        self.baseline = 0
+        self.peak = 0
+        self._samples = []
+
+    def start(self):
+        self.baseline = measure_vram()
+        self.peak = self.baseline
+        self._samples = [self.baseline]
+
+    def sample(self):
+        v = measure_vram()
+        self._samples.append(v)
+        self.peak = max(self.peak, v)
+
+    def stop(self) -> dict:
+        self.sample()
+        return {
+            "baseline_mb": self.baseline,
+            "peak_mb": self.peak,
+            "delta_mb": self.peak - self.baseline,
+            "samples": len(self._samples),
+        }

kvcache_bench-0.1.0/kvcache_bench/ollama.py

@@ -0,0 +1,133 @@
+"""Ollama backend: run inference with different KV cache types."""
+
+import json
+import time
+import requests
+from dataclasses import dataclass, field
+from typing import Optional
+
+
+OLLAMA_BASE = "http://localhost:11434"
+
+# KV cache types supported by Ollama/llama.cpp
+KV_TYPES = ["f16", "q8_0", "q4_0"]
+
+# Mixed K/V configurations worth testing
+MIXED_KV = [
+    ("f16", "f16"),      # Baseline
+    ("q8_0", "q8_0"),    # 8-bit both
+    ("q4_0", "q4_0"),    # 4-bit both
+    ("q8_0", "q4_0"),    # 8-bit keys, 4-bit values (sweet spot per research)
+    ("q4_0", "q8_0"),    # 4-bit keys, 8-bit values (keys are more sensitive)
+]
+
+
+@dataclass
+class OllamaResult:
+    model: str
+    kv_type_k: str
+    kv_type_v: str
+    context_length: int
+    prompt_tokens: int
+    generated_tokens: int
+    prompt_eval_rate: float  # tok/s
+    eval_rate: float  # tok/s
+    total_duration_ms: float
+    response_text: str = ""
+
+
+def check_ollama() -> bool:
+    """Check if Ollama is running."""
+    try:
+        r = requests.get(f"{OLLAMA_BASE}/", timeout=3)
+        return r.status_code == 200
+    except Exception:
+        return False
+
+
+def list_models() -> list[str]:
+    """List available Ollama models."""
+    try:
+        r = requests.get(f"{OLLAMA_BASE}/api/tags", timeout=5)
+        data = r.json()
+        return [m["name"] for m in data.get("models", [])]
+    except Exception:
+        return []
+
+
+def run_inference(
+    model: str,
+    prompt: str,
+    num_ctx: int = 4096,
+    max_tokens: int = 100,
+    temperature: float = 0.0,
+) -> Optional[dict]:
+    """Run a single inference call via Ollama API."""
+    try:
+        r = requests.post(
+            f"{OLLAMA_BASE}/api/generate",
+            json={
+                "model": model,
+                "prompt": prompt,
+                "stream": False,
+                "options": {
+                    "num_ctx": num_ctx,
+                    "num_predict": max_tokens,
+                    "temperature": temperature,
+                },
+            },
+            timeout=300,
+        )
+        return r.json()
+    except Exception as e:
+        return {"error": str(e)}
+
+
+def run_chat(
+    model: str,
+    messages: list[dict],
+    num_ctx: int = 4096,
+    max_tokens: int = 100,
+    tools: Optional[list] = None,
+) -> Optional[dict]:
+    """Run a chat completion via Ollama API."""
+    body = {
+        "model": model,
+        "messages": messages,
+        "stream": False,
+        "options": {
+            "num_ctx": num_ctx,
+            "num_predict": max_tokens,
+            "temperature": 0.0,
+        },
+    }
+    if tools:
+        body["tools"] = tools
+    try:
+        r = requests.post(f"{OLLAMA_BASE}/api/chat", json=body, timeout=300)
+        return r.json()
+    except Exception as e:
+        return {"error": str(e)}
+
+
+# Standard prompts for benchmarking
+BENCH_PROMPTS = {
+    "short": "What is the capital of France? Answer in one word.",
+    "code": "Write a Python function to check if a number is prime. Just the function, no explanation.",
+    "long": "You are an expert software engineer. " * 100 + "\n\nAnalyze the time complexity of binary search and explain why it's O(log n).",
+    "reasoning": "A farmer has 17 sheep. All but 9 die. How many sheep are left? Think step by step.",
+    "tool_call": "Get the current weather in Tokyo.",
+}
+
+BENCH_TOOL = {
+    "type": "function",
+    "function": {
+        "name": "get_weather",
+        "description": "Get current weather for a location",
+        "parameters": {
+            "type": "object",
+            "properties": {"location": {"type": "string", "description": "City name"}},
+            "required": ["location"],
+        },
+    },
+}

kvcache_bench-0.1.0/kvcache_bench.egg-info/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: kvcache-bench
+Version: 0.1.0
+Summary: Benchmark every KV cache compression method on your GPU. One command, real numbers.
+Author: back2matching
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/back2matching/kvcache-bench
+Keywords: llm,kv-cache,benchmark,vram,gpu,ollama,llama-cpp,quantization
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28.0
+Requires-Dist: pynvml>=11.5.0
+Provides-Extra: charts
+Requires-Dist: matplotlib; extra == "charts"
+Provides-Extra: hf
+Requires-Dist: torch; extra == "hf"
+Requires-Dist: transformers; extra == "hf"
+Dynamic: license-file
+
+# kvcache-bench
+
+Benchmark every KV cache compression method on your GPU. One command, real numbers.
+
+```
+kvcache-bench --model qwen3.5:9b
+```
+
+```
+| KV Type | Context | Prompt    | Gen tok/s | Prefill tok/s | VRAM +MB | Quality |
+|---------|---------|-----------|-----------|---------------|----------|---------|
+| f16     | 4096    | short     | 80.1      | 712.3         | +142     | PASS    |
+| q8_0    | 4096    | short     | 79.5      | 723.5         | +71      | PASS    |
+| q4_0    | 4096    | short     | 78.2      | 698.1         | +36      | PASS    |
+```
+
+## Why
+
+When you run a local LLM, the KV cache eats your VRAM. Ollama and llama.cpp support different KV cache quantization types (f16, q8_0, q4_0), but nobody tells you what the actual tradeoff is on YOUR hardware.
+
+Current state of the world:
+- You Google "ollama kv cache quantization" and find forum posts with conflicting advice
+- You manually test each config, eyeball nvidia-smi, and guess
+- No tool compares them systematically
+
+kvcache-bench fixes this. It tests every KV cache type on your GPU and gives you a comparison table with speed, VRAM, and quality.
+
+## Install
+
+```bash
+pip install kvcache-bench
+```
+
+## Usage
+
+```bash
+# Auto-detect your first model, test all KV types
+kvcache-bench
+
+# Specific model
+kvcache-bench --model qwen3.5:9b
+
+# Test at multiple context lengths (where KV savings matter most)
+kvcache-bench --model llama3.1:8b --context 4096,8192,16384
+
+# Include tool calling test
+kvcache-bench --model qwen3.5:9b --prompts short,code,reasoning,tool_call
+
+# Save results as JSON
+kvcache-bench --model qwen3.5:9b --json results.json
+
+# Just show GPU info
+kvcache-bench --gpu
+
+# List available models
+kvcache-bench --list-models
+```
+
+## What It Tests
+
+For each KV cache type (f16, q8_0, q4_0), it measures:
+
+| Metric | How |
+|--------|-----|
+| **Generation speed** | Tokens per second during generation |
+| **Prefill speed** | Tokens per second processing the prompt |
+| **VRAM delta** | Extra VRAM used beyond model weights (measured via nvidia-smi) |
+| **Quality** | Auto-checked against expected answers (Paris, code structure, reasoning) |
+
+## How It Works
+
+1. Detects your GPU and Ollama installation
+2. For each KV cache type: restarts Ollama with `OLLAMA_KV_CACHE_TYPE=<type>`, warms up the model, runs benchmark prompts
+3. Measures VRAM before and during inference via nvidia-smi
+4. Extracts timing from Ollama's API response (prompt_eval_duration, eval_duration)
+5. Checks response quality with simple auto-graders
+6. Produces a markdown table (and optional JSON)
+
+## What the Research Says
+
+Based on llama.cpp community benchmarks and our testing:
+
+| KV Type | VRAM Savings | Perplexity Impact | Best For |
+|---------|-------------|-------------------|----------|
+| f16 | Baseline | None | When you have VRAM to spare |
+| q8_0 | 2x | +0.004 (negligible) | **Default recommendation.** Free VRAM, zero quality cost. |
+| q4_0 | 4x | +0.2 (noticeable) | When you need max context length or are VRAM-constrained |
+
+The sweet spot for most users: **q8_0**. Halves your KV cache VRAM with essentially zero quality loss.
+
+## Requirements
+
+- Python 3.10+
+- NVIDIA GPU with nvidia-smi
+- Ollama installed and running
+
+## Roadmap
+
+- [ ] Mixed K/V types (q8 keys + q4 values)
+- [ ] Context length sweep charts
+- [ ] HuggingFace backend (vLLM, TGI)
+- [ ] TurboQuant integration
+- [ ] Multi-model matrix
+- [ ] HuggingFace Spaces leaderboard
+- [ ] Community result submissions
+
+## License
+
+Apache 2.0
+
+## Related
+
+- [turboquant](https://github.com/back2matching/turboquant) -- TurboQuant KV cache compression (sub-4-bit)
+- [NVIDIA kvpress](https://github.com/NVIDIA/kvpress) -- KV cache eviction/pruning methods
+- [llama.cpp](https://github.com/ggml-org/llama.cpp) -- Where KV cache quantization lives

kvcache_bench-0.1.0/kvcache_bench.egg-info/SOURCES.txt

@@ -0,0 +1,14 @@
+LICENSE
+README.md
+pyproject.toml
+kvcache_bench/__init__.py
+kvcache_bench/bench.py
+kvcache_bench/cli.py
+kvcache_bench/gpu.py
+kvcache_bench/ollama.py
+kvcache_bench.egg-info/PKG-INFO
+kvcache_bench.egg-info/SOURCES.txt
+kvcache_bench.egg-info/dependency_links.txt
+kvcache_bench.egg-info/entry_points.txt
+kvcache_bench.egg-info/requires.txt
+kvcache_bench.egg-info/top_level.txt

kvcache_bench-0.1.0/kvcache_bench.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

kvcache_bench-0.1.0/kvcache_bench.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+kvcache-bench = kvcache_bench.cli:main

kvcache_bench-0.1.0/kvcache_bench.egg-info/requires.txt

@@ -0,0 +1,9 @@
+requests>=2.28.0
+pynvml>=11.5.0
+
+[charts]
+matplotlib
+
+[hf]
+torch
+transformers

kvcache_bench-0.1.0/kvcache_bench.egg-info/top_level.txt

@@ -0,0 +1 @@
+kvcache_bench

kvcache_bench-0.1.0/pyproject.toml

@@ -0,0 +1,33 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "kvcache-bench"
+version = "0.1.0"
+description = "Benchmark every KV cache compression method on your GPU. One command, real numbers."
+readme = "README.md"
+license = {text = "Apache-2.0"}
+requires-python = ">=3.10"
+authors = [{name = "back2matching"}]
+keywords = ["llm", "kv-cache", "benchmark", "vram", "gpu", "ollama", "llama-cpp", "quantization"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "License :: OSI Approved :: Apache Software License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["requests>=2.28.0", "pynvml>=11.5.0"]
+
+[project.optional-dependencies]
+charts = ["matplotlib"]
+hf = ["torch", "transformers"]
+
+[project.urls]
+Homepage = "https://github.com/back2matching/kvcache-bench"
+
+[project.scripts]
+kvcache-bench = "kvcache_bench.cli:main"
+
+[tool.setuptools.packages.find]
+include = ["kvcache_bench*"]

kvcache_bench-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+