kvquant-hf 0.2.0__tar.gz

kvquant_hf-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sardor Razikov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

kvquant_hf-0.2.0/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.4
+Name: kvquant-hf
+Version: 0.2.0
+Summary: Tested KV-cache quantization for Hugging Face transformers >= 5 (with a silent-no-op stress test).
+License: MIT
+Keywords: llm,kv-cache,quantization,transformers,inference,efficiency
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.1
+Provides-Extra: bench
+Requires-Dist: transformers>=5.0; extra == "bench"
+Requires-Dist: numpy; extra == "bench"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: numpy; extra == "dev"
+Dynamic: license-file
+
+# kvquant
+
+**Tested KV-cache quantization for Hugging Face `transformers >= 5` — with a stress test so a silent no-op can't fool you.**
+
+Most "quantize the KV cache" snippets you'll find mutate `cache.key_cache[i]`. On
+`transformers 5.x` that is a **non-persisting compatibility accessor**: your edit is
+thrown away and the model keeps using the original fp tensors. Everything *looks* like
+it worked — same code, no error — but perplexity doesn't move because nothing changed.
+The real tensors live at `cache.layers[i].keys` / `.values`. kvquant mutates the right
+ones and ships a guard that **proves** the quantization actually bit.
+
+## Install
+
+```bash
+pip install kvquant-hf            # core
+pip install "kvquant-hf[bench]"   # + transformers/numpy for the perplexity benchmark
+```
+
+## Quickstart
+
+```python
+from kvquant import quantize_cache          # in-place int-N quant of a transformers>=5 Cache
+change = quantize_cache(past_key_values, bits=8)   # returns mean |change|; 0.0 == a no-op bug
+```
+
+Measure the quality cost yourself:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from kvquant import kv_ppl_sweep
+m = AutoModelForCausalLM.from_pretrained("google/gemma-3-270m-it")
+t = AutoTokenizer.from_pretrained("google/gemma-3-270m-it")
+print(kv_ppl_sweep(m, t))            # {None/8/4/2: {ppl, delta_pct, change}}
+```
+
+or from the shell:
+
+```bash
+kvquant-bench --model google/gemma-3-270m-it
+```
+
+## What int8 KV actually costs (measured)
+
+`google/gemma-3-270m-it`, 300 tokens, per-channel symmetric int quant of K and V:
+
+| KV cache | perplexity | Δ vs fp | note |
+|---|---|---|---|
+| fp (baseline) | 4.5479 | — | |
+| **int8** | 4.5437 | **−0.09%** | within noise — effectively lossless |
+| int4 | 4.7528 | +4.5% | visible degradation |
+| int2 | 982.76 | +21509% | breaks (this is the stress check) |
+
+int8 K/V carries **half** the bytes of fp16 (a quarter of fp32), so an int8 cache is
+the obvious lever for longer context / larger batch at near-zero quality cost. Run the
+sweep on *your* model — numbers vary by architecture.
+
+### Real memory, measured (not just quality)
+
+`kvquant.compress_cache` stores a populated cache as int8 + a per-token fp16 scale and
+reports the actual byte reduction:
+
+```python
+from kvquant import compress_cache
+stores, report = compress_cache(past_key_values)
+print(report)   # {'fp_bytes': 5_824_512, 'int8_bytes': 1_467_504, 'ratio': 3.97}
+```
+
+On the gemma cache above (158 tokens, 18 layers): **5,824,512 → 1,467,504 bytes = 3.97×
+smaller vs fp32 (~2× vs fp16)**, ~2% per-tensor reconstruction error. Use it to compress
+a cache for offload / serialization / holding an idle long context.
+
+### Live int8 cache (holds int8 *during* generation)
+
+`make_live_int8` converts a cache so every layer **stores** int8 + a per-token fp16 scale
+and dequantizes on read inside attention — the memory held is really int8, no extra deps:
+
+```python
+from kvquant import make_live_int8, held_bytes
+
+past = None
+for step in decode_loop:
+    out = model(next_token, past_key_values=past, use_cache=True)
+    past = make_live_int8(out.past_key_values)   # cache now holds int8
+```
+
+Measured on `gemma-3-270m-it` (300 tokens, autoregressive): perplexity **+0.74%**, cache
+held as `torch.int8` and **3.97× smaller than fp32 (~2× vs fp16)**. It subclasses each
+layer's own class, so sliding-window and other behaviours are preserved.
+
+## The silent-no-op guard
+
+`kv_ppl_sweep` asserts that int2 both changes the tensors (`change > 0`) **and** wrecks
+perplexity. If it doesn't, you're mutating the wrong thing and the whole measurement is
+meaningless — so it raises instead of printing a comfortable lie. The same idea is
+packaged as a standalone decorator:
+
+```python
+from kvquant.doesitbite import bites
+
+@bites()                      # raises AssertionError if the transform doesn't change its input
+def my_quantizer(x): ...
+```
+
+## Also included
+
+- **`kvquant.doesitbite`** — `@bites` / `assert_bites` to catch transforms that silently
+  do nothing (mis-targeted in-place edits, dead paths, clamps that never fire).
+- **`kvquant.rankconf`** — confidence utilities for rank metrics (AURC/risk-coverage):
+  `aurc`, `is_rank_noop` (any strictly-monotonic rescale is a no-op for a rank metric),
+  and `agreement_confidence` (a stronger signal than a model's self-reported confidence).
+
+## Honest limitations (v0)
+
+- Three tools, clear jobs: `kv_ppl_sweep` / `quantize_cache` **fake-quantize** (quant→dequant)
+  to measure the *quality* cost with the correct non-no-op hook; `compress_cache` /
+  `Int8KVStore` do **real int8 storage** for a finished cache; `make_live_int8` holds int8
+  **during** generation. All measured above.
+- Symmetric quantization, no outlier handling or post-RoPE split yet (those shrink the
+  int8 quality gap further).
+- Benchmarked on small models on a laptop (gemma-3-270m). Try it on your model and
+  architecture and report back — especially GQA/MQA and sliding-window variants.
+
+## Support
+
+kvquant is free and MIT-licensed. If it saved you GPU memory or an afternoon of
+debugging, a tip keeps the work going — no obligation.
+
+- **GitHub Sponsors:** the ❤ Sponsor button above (once enabled)
+- **BTC:** `bc1q9hm4upksmvkzhl8z3zvxgz94da358qfe3tred0`
+- **ETH / Polygon / USDT (ERC-20):** `0xa9e3b63799f5E5cc6258042D45869a5a1d39F021`
+- **USDT (TRC-20) / TRON:** `TUyNkMXme1CYAc1wZ8SZ64X9upBc9nwTZ5`
+- **Solana:** `j8zZCsWwM8WmUZwMSDHikNuz6E49Vn8nwo9FMQnHSoW`
+
+## License
+
+MIT.

kvquant_hf-0.2.0/README.md

@@ -0,0 +1,136 @@
+# kvquant
+
+**Tested KV-cache quantization for Hugging Face `transformers >= 5` — with a stress test so a silent no-op can't fool you.**
+
+Most "quantize the KV cache" snippets you'll find mutate `cache.key_cache[i]`. On
+`transformers 5.x` that is a **non-persisting compatibility accessor**: your edit is
+thrown away and the model keeps using the original fp tensors. Everything *looks* like
+it worked — same code, no error — but perplexity doesn't move because nothing changed.
+The real tensors live at `cache.layers[i].keys` / `.values`. kvquant mutates the right
+ones and ships a guard that **proves** the quantization actually bit.
+
+## Install
+
+```bash
+pip install kvquant-hf            # core
+pip install "kvquant-hf[bench]"   # + transformers/numpy for the perplexity benchmark
+```
+
+## Quickstart
+
+```python
+from kvquant import quantize_cache          # in-place int-N quant of a transformers>=5 Cache
+change = quantize_cache(past_key_values, bits=8)   # returns mean |change|; 0.0 == a no-op bug
+```
+
+Measure the quality cost yourself:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from kvquant import kv_ppl_sweep
+m = AutoModelForCausalLM.from_pretrained("google/gemma-3-270m-it")
+t = AutoTokenizer.from_pretrained("google/gemma-3-270m-it")
+print(kv_ppl_sweep(m, t))            # {None/8/4/2: {ppl, delta_pct, change}}
+```
+
+or from the shell:
+
+```bash
+kvquant-bench --model google/gemma-3-270m-it
+```
+
+## What int8 KV actually costs (measured)
+
+`google/gemma-3-270m-it`, 300 tokens, per-channel symmetric int quant of K and V:
+
+| KV cache | perplexity | Δ vs fp | note |
+|---|---|---|---|
+| fp (baseline) | 4.5479 | — | |
+| **int8** | 4.5437 | **−0.09%** | within noise — effectively lossless |
+| int4 | 4.7528 | +4.5% | visible degradation |
+| int2 | 982.76 | +21509% | breaks (this is the stress check) |
+
+int8 K/V carries **half** the bytes of fp16 (a quarter of fp32), so an int8 cache is
+the obvious lever for longer context / larger batch at near-zero quality cost. Run the
+sweep on *your* model — numbers vary by architecture.
+
+### Real memory, measured (not just quality)
+
+`kvquant.compress_cache` stores a populated cache as int8 + a per-token fp16 scale and
+reports the actual byte reduction:
+
+```python
+from kvquant import compress_cache
+stores, report = compress_cache(past_key_values)
+print(report)   # {'fp_bytes': 5_824_512, 'int8_bytes': 1_467_504, 'ratio': 3.97}
+```
+
+On the gemma cache above (158 tokens, 18 layers): **5,824,512 → 1,467,504 bytes = 3.97×
+smaller vs fp32 (~2× vs fp16)**, ~2% per-tensor reconstruction error. Use it to compress
+a cache for offload / serialization / holding an idle long context.
+
+### Live int8 cache (holds int8 *during* generation)
+
+`make_live_int8` converts a cache so every layer **stores** int8 + a per-token fp16 scale
+and dequantizes on read inside attention — the memory held is really int8, no extra deps:
+
+```python
+from kvquant import make_live_int8, held_bytes
+
+past = None
+for step in decode_loop:
+    out = model(next_token, past_key_values=past, use_cache=True)
+    past = make_live_int8(out.past_key_values)   # cache now holds int8
+```
+
+Measured on `gemma-3-270m-it` (300 tokens, autoregressive): perplexity **+0.74%**, cache
+held as `torch.int8` and **3.97× smaller than fp32 (~2× vs fp16)**. It subclasses each
+layer's own class, so sliding-window and other behaviours are preserved.
+
+## The silent-no-op guard
+
+`kv_ppl_sweep` asserts that int2 both changes the tensors (`change > 0`) **and** wrecks
+perplexity. If it doesn't, you're mutating the wrong thing and the whole measurement is
+meaningless — so it raises instead of printing a comfortable lie. The same idea is
+packaged as a standalone decorator:
+
+```python
+from kvquant.doesitbite import bites
+
+@bites()                      # raises AssertionError if the transform doesn't change its input
+def my_quantizer(x): ...
+```
+
+## Also included
+
+- **`kvquant.doesitbite`** — `@bites` / `assert_bites` to catch transforms that silently
+  do nothing (mis-targeted in-place edits, dead paths, clamps that never fire).
+- **`kvquant.rankconf`** — confidence utilities for rank metrics (AURC/risk-coverage):
+  `aurc`, `is_rank_noop` (any strictly-monotonic rescale is a no-op for a rank metric),
+  and `agreement_confidence` (a stronger signal than a model's self-reported confidence).
+
+## Honest limitations (v0)
+
+- Three tools, clear jobs: `kv_ppl_sweep` / `quantize_cache` **fake-quantize** (quant→dequant)
+  to measure the *quality* cost with the correct non-no-op hook; `compress_cache` /
+  `Int8KVStore` do **real int8 storage** for a finished cache; `make_live_int8` holds int8
+  **during** generation. All measured above.
+- Symmetric quantization, no outlier handling or post-RoPE split yet (those shrink the
+  int8 quality gap further).
+- Benchmarked on small models on a laptop (gemma-3-270m). Try it on your model and
+  architecture and report back — especially GQA/MQA and sliding-window variants.
+
+## Support
+
+kvquant is free and MIT-licensed. If it saved you GPU memory or an afternoon of
+debugging, a tip keeps the work going — no obligation.
+
+- **GitHub Sponsors:** the ❤ Sponsor button above (once enabled)
+- **BTC:** `bc1q9hm4upksmvkzhl8z3zvxgz94da358qfe3tred0`
+- **ETH / Polygon / USDT (ERC-20):** `0xa9e3b63799f5E5cc6258042D45869a5a1d39F021`
+- **USDT (TRC-20) / TRON:** `TUyNkMXme1CYAc1wZ8SZ64X9upBc9nwTZ5`
+- **Solana:** `j8zZCsWwM8WmUZwMSDHikNuz6E49Vn8nwo9FMQnHSoW`
+
+## License
+
+MIT.

kvquant_hf-0.2.0/kvquant/__init__.py

@@ -0,0 +1,18 @@
+"""kvquant — tested KV-cache quantization for Hugging Face transformers >= 5.
+
+Most KV-quant snippets online silently no-op on transformers 5.x because they mutate
+``cache.key_cache[i]`` (a non-persisting compat accessor) instead of the real
+``cache.layers[i].keys`` / ``.values``. kvquant mutates the right tensors and ships a
+stress test so you can never be fooled.
+"""
+from .core import quantize_cache, quantize_dequantize
+from .bench import kv_ppl_sweep
+from .storage import Int8KVStore, cache_bytes, compress_cache
+from .live import make_live_int8, held_bytes
+
+__all__ = [
+    "quantize_cache", "quantize_dequantize", "kv_ppl_sweep",
+    "Int8KVStore", "cache_bytes", "compress_cache",
+    "make_live_int8", "held_bytes",
+]
+__version__ = "0.2.0"

kvquant_hf-0.2.0/kvquant/bench.py

@@ -0,0 +1,91 @@
+"""Perplexity benchmark for KV-cache quantization, with a built-in silent-no-op guard.
+
+Scores tokens autoregressively (so the *cache* is actually exercised) and quantizes
+the KV cache after every step. Reports perplexity at fp / int8 / int4 / int2. The int2
+row MUST blow up and show a non-zero quant-change — if it does not, the quantization
+is a silent no-op and the numbers are meaningless, so :func:`kv_ppl_sweep` asserts it.
+"""
+from __future__ import annotations
+import math
+import torch
+from .core import quantize_cache
+
+DEFAULT_TEXT = (
+    "Perplexity measures how well a probability model predicts a sample; lower is "
+    "better. Quantizing the key-value cache to eight-bit integers halves its memory "
+    "footprint, which lets a server hold a longer context or a larger batch on the "
+    "same hardware. We test whether that compression degrades predictions. "
+) * 4
+
+
+@torch.no_grad()
+def _score(model, input_ids, bits):
+    past = None
+    nll = 0.0
+    n = 0
+    change = 0.0
+    T = input_ids.shape[1]
+    for t in range(T - 1):
+        out = model(input_ids[:, t : t + 1], past_key_values=past, use_cache=True)
+        past = out.past_key_values
+        if bits:
+            change += quantize_cache(past, bits)
+        logp = torch.log_softmax(out.logits[:, -1, :].float(), dim=-1)
+        nll += -logp[0, input_ids[0, t + 1]].item()
+        n += 1
+    return math.exp(nll / n), change
+
+
+def kv_ppl_sweep(model, tokenizer, text: str = DEFAULT_TEXT, bits_list=(None, 8, 4, 2),
+                 max_tokens: int = 400, device: str | None = None):
+    """Return ``{bits: {"ppl", "delta_pct", "change"}}`` and assert it isn't a no-op."""
+    if device is None:
+        device = "mps" if torch.backends.mps.is_available() else (
+            "cuda" if torch.cuda.is_available() else "cpu")
+    model = model.to(device).eval()
+    ids = tokenizer(text, return_tensors="pt").input_ids.to(device)[:, :max_tokens]
+    out, base = {}, None
+    for bits in bits_list:
+        ppl, change = _score(model, ids, bits)
+        if bits is None:
+            base = ppl
+        out[bits] = {"ppl": ppl, "delta_pct": (ppl / base - 1) * 100, "change": change}
+    # silent-no-op guard: 2-bit must both bite (change>0) and wreck ppl
+    two = out.get(2)
+    if two is not None and (two["change"] == 0.0 or two["delta_pct"] < 50):
+        raise AssertionError(
+            "int2 KV did not degrade perplexity (change=%.3f, delta=%.1f%%) — the "
+            "quantization is a SILENT NO-OP; you are probably mutating the wrong cache "
+            "attribute." % (two["change"], two["delta_pct"]))
+    return out
+
+
+def _main():
+    import argparse
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+    ap = argparse.ArgumentParser(description="KV-cache quantization perplexity sweep")
+    ap.add_argument("--model", default="google/gemma-3-270m-it")
+    ap.add_argument("--tokens", type=int, default=400)
+    args = ap.parse_args()
+    tok = AutoTokenizer.from_pretrained(args.model)
+    model = AutoModelForCausalLM.from_pretrained(args.model, dtype=torch.float32)
+    res = kv_ppl_sweep(model, tok, max_tokens=args.tokens)
+    print(f"model: {args.model}")
+    for bits, r in res.items():
+        tag = "fp (baseline)" if bits is None else f"int{bits}"
+        print(f"  {tag:14s} ppl={r['ppl']:.4f}  dppl={r['delta_pct']:+.3f}%  "
+              f"change={r['change']:.3f}")
+    # real, measured memory saving from int8 storage on a populated cache
+    import torch as _t
+    from .storage import compress_cache
+    with _t.no_grad():
+        ids = tok(DEFAULT_TEXT, return_tensors="pt").input_ids[:, : args.tokens]
+        dev = next(model.parameters()).device
+        cache = model(ids.to(dev), use_cache=True).past_key_values
+    _, mem = compress_cache(cache)
+    print(f"  int8 storage: {mem['fp_bytes']:,} -> {mem['int8_bytes']:,} bytes "
+          f"= {mem['ratio']:.2f}x smaller vs fp32 (~{mem['ratio']/2:.2f}x vs fp16).")
+
+
+if __name__ == "__main__":
+    _main()

kvquant_hf-0.2.0/kvquant/core.py

@@ -0,0 +1,55 @@
+"""Core KV-cache quantization for Hugging Face transformers >= 5.
+
+The one thing people get wrong on transformers 5.x: the real K/V tensors live at
+``cache.layers[i].keys`` / ``.values``. The old ``cache.key_cache[i]`` is a
+non-persisting compatibility accessor — mutating it is a **silent no-op** (the model
+keeps using the original fp tensors). :func:`quantize_cache` mutates the real tensors
+and returns the mean absolute change so you can *prove* it actually bit.
+"""
+from __future__ import annotations
+import torch
+
+
+def quantize_dequantize(x: torch.Tensor, bits: int = 8, axis: int = -2) -> torch.Tensor:
+    """Symmetric per-channel fake-quantization of ``x`` to ``bits`` integers.
+
+    Scale is computed per slice along ``axis`` (default: the sequence axis of a
+    ``[batch, heads, seq, head_dim]`` KV tensor). Returns a dequantized float tensor
+    with the same shape and dtype — this measures the *quality* cost of an int cache.
+    (For real memory savings the cache must *store* int8; see the README roadmap.)
+    """
+    if bits < 2 or bits > 8:
+        raise ValueError("bits must be in [2, 8]")
+    lv = 2 ** (bits - 1) - 1
+    s = x.abs().amax(dim=axis, keepdim=True) / lv
+    s = torch.where(s == 0, torch.ones_like(s), s)
+    return torch.round(x / s).clamp(-lv, lv) * s
+
+
+def quantize_cache(cache, bits: int = 8) -> float:
+    """In-place int-``bits`` quantization of a transformers>=5 ``Cache``.
+
+    Mutates ``cache.layers[i].keys`` / ``.values`` (the tensors attention actually
+    reads). Returns the **mean absolute change** across layers. A return value of
+    ``0.0`` means the quantization did not bite — treat that as a bug, not a result.
+    """
+    layers = getattr(cache, "layers", None)
+    if layers is None:
+        raise TypeError(
+            "expected a transformers>=5 Cache exposing `.layers`; got "
+            f"{type(cache).__name__}. On transformers 5.x do NOT mutate `.key_cache` "
+            "(a non-persisting compat accessor) — that is a silent no-op."
+        )
+    total, n = 0.0, 0
+    for layer in layers:
+        keys = getattr(layer, "keys", None)
+        if keys is None:
+            continue
+        before = keys.clone()
+        layer.keys = quantize_dequantize(layer.keys, bits)
+        layer.values = quantize_dequantize(layer.values, bits)
+        total += (layer.keys - before).abs().mean().item()
+        n += 1
+    if n == 0:
+        raise RuntimeError("cache had no populated layers to quantize")
+    return total / n

kvquant_hf-0.2.0/kvquant/doesitbite.py

@@ -0,0 +1,72 @@
+"""doesitbite — catch silent no-ops in ML pipeline transforms.
+
+A transform that doesn't actually change its input (a mis-targeted in-place edit, a
+dead code path, a clamp that never fires) silently degrades your results while every
+test stays green. ``@bites`` asserts a transform MEASURABLY changes its input; use it
+in your tests or wrap a suspicious step.
+
+Born 2026-10-28/29: a KV-cache quantizer that mutated the wrong attribute and a
+packing optimizer stuck in a local minimum both passed unnoticed until stress-tested.
+"""
+from __future__ import annotations
+import functools
+
+
+def _meanabs(x):
+    try:
+        import torch
+        if torch.is_tensor(x):
+            return (x.float()).abs().mean().item()
+    except Exception:
+        pass
+    try:
+        import numpy as np
+        a = np.asarray(x, dtype="float64")
+        return float(np.abs(a).mean())
+    except Exception:
+        return None
+
+
+def changed(before, after) -> float | None:
+    """Mean absolute difference between two array/tensor-like objects, or None."""
+    try:
+        import torch
+        if torch.is_tensor(before) and torch.is_tensor(after):
+            return (after.float() - before.float()).abs().mean().item()
+    except Exception:
+        pass
+    try:
+        import numpy as np
+        return float(np.abs(np.asarray(after, "float64") - np.asarray(before, "float64")).mean())
+    except Exception:
+        return None
+
+
+def bites(min_change: float = 1e-9):
+    """Decorator: assert ``fn(x, ...)`` returns something that differs from ``x``.
+
+    Raises ``AssertionError`` (a silent no-op) when the output equals the input within
+    ``min_change``. The first positional arg is treated as the input.
+    """
+    def deco(fn):
+        @functools.wraps(fn)
+        def wrapper(x, *a, **k):
+            try:
+                import torch
+                before = x.clone() if hasattr(x, "clone") else x
+            except Exception:
+                before = x
+            out = fn(x, *a, **k)
+            d = changed(before, out)
+            if d is not None and d <= min_change:
+                raise AssertionError(
+                    f"{fn.__name__} is a SILENT NO-OP: output differs from input by "
+                    f"{d:.3g} (<= {min_change:g}). It did not bite.")
+            return out
+        return wrapper
+    return deco
+
+
+def assert_bites(fn, x, *args, min_change: float = 1e-9, **kwargs):
+    """Imperative form: run ``fn(x, ...)`` once and assert it changed ``x``."""
+    return bites(min_change)(fn)(x, *args, **kwargs)

kvquant_hf-0.2.0/kvquant/live.py

@@ -0,0 +1,103 @@
+"""Live int8 KV cache — actually HOLDS int8 in memory during generation.
+
+Unlike :func:`kvquant.quantize_cache` (fake-quant, measures quality) and
+:mod:`kvquant.storage` (compress a finished cache), this converts a transformers>=5
+cache so each layer *stores* its keys/values as int8 + a per-token fp16 scale and
+dequantizes them on read inside attention. Memory held is ~1/4 of fp32 (~1/2 of fp16);
+the quality cost is the int8 cost (sub-1% perplexity on the models tested).
+
+Usage: call :func:`make_live_int8` on the cache after each forward in your decode loop
+(or once on a prefilled cache). It is idempotent and keeps the layer's own class
+behaviour (e.g. sliding-window) by subclassing it, so attention still works.
+"""
+from __future__ import annotations
+import torch
+
+_variants: dict = {}
+
+
+def _q(fp):
+    s = fp.abs().amax(dim=-1, keepdim=True) / 127.0
+    s = torch.where(s == 0, torch.ones_like(s), s)
+    return torch.round(fp / s).clamp(-127, 127).to(torch.int8), s.to(torch.float16)
+
+
+def _dq(q, s, dtype):
+    return (q.float() * s.float()).to(dtype) if q is not None else None
+
+
+def _variant(base):
+    """A subclass of ``base`` whose keys/values are int8-backed properties."""
+    if base in _variants:
+        return _variants[base]
+
+    class Int8Layer(base):
+        _kvquant_int8 = True
+
+        @property
+        def keys(self):
+            return _dq(getattr(self, "_qk", None), getattr(self, "_sk", None),
+                       getattr(self, "_dt", torch.float32))
+
+        @keys.setter
+        def keys(self, v):
+            if v is None:
+                self._qk = self._sk = None
+            else:
+                self._dt = v.dtype
+                self._qk, self._sk = _q(v)
+
+        @property
+        def values(self):
+            return _dq(getattr(self, "_qv", None), getattr(self, "_sv", None),
+                       getattr(self, "_dt", torch.float32))
+
+        @values.setter
+        def values(self, v):
+            if v is None:
+                self._qv = self._sv = None
+            else:
+                self._dt = v.dtype
+                self._qv, self._sv = _q(v)
+
+        def int8_bytes(self) -> int:
+            b = 0
+            for a in ("_qk", "_sk", "_qv", "_sv"):
+                t = getattr(self, a, None)
+                if t is not None:
+                    b += t.element_size() * t.nelement()
+            return b
+
+    _variants[base] = Int8Layer
+    return Int8Layer
+
+
+def make_live_int8(cache):
+    """Convert a transformers>=5 cache in place so its layers hold int8. Idempotent."""
+    layers = getattr(cache, "layers", None)
+    if layers is None:
+        raise TypeError("expected a transformers>=5 Cache exposing `.layers`")
+    for layer in layers:
+        if getattr(type(layer), "_kvquant_int8", False):
+            continue  # already converted
+        k = getattr(layer, "keys", None)
+        v = getattr(layer, "values", None)
+        layer.__class__ = _variant(type(layer))
+        layer._dt = k.dtype if k is not None else torch.float32
+        layer.keys = k
+        layer.values = v
+    return cache
+
+
+def held_bytes(cache) -> int:
+    """Live byte footprint of the cache's K/V storage (int8 if converted, else fp)."""
+    total = 0
+    for layer in getattr(cache, "layers", []):
+        if hasattr(layer, "int8_bytes"):
+            total += layer.int8_bytes()
+        else:
+            for name in ("keys", "values"):
+                t = getattr(layer, name, None)
+                if t is not None:
+                    total += t.element_size() * t.nelement()
+    return total

kvquant_hf-0.2.0/kvquant/rankconf.py

@@ -0,0 +1,58 @@
+"""rankconf — confidence utilities for rank-based selective-prediction metrics.
+
+Two facts people miss on metrics that depend only on the *order* of a confidence
+score (AURC, risk-coverage, AUROC):
+
+1. Any **strictly monotonic** rescale of confidence (isotonic in the no-tie case,
+   Platt, temperature, sigmoid) leaves the metric unchanged — it is a provable no-op.
+   So calibrating to improve AURC is wasted effort; spend it on a *better signal*.
+2. **Agreement-based** confidence (how much an independent second pass agrees) is a
+   much stronger ranking signal than a model's self-reported confidence.
+
+Verified 2026-10-29: on synthetic-but-faithful data, strictly-monotonic rescales gave
+identical AURC (delta 0), and agreement confidence cut AURC by ~34% vs self-reported.
+"""
+from __future__ import annotations
+import numpy as np
+
+
+def aurc(confidence, error) -> float:
+    """Area under the risk-coverage curve (lower is better).
+
+    Sorts items by ``confidence`` descending and averages the running mean ``error``
+    over all coverage levels. Depends only on the *order* of ``confidence``.
+    """
+    confidence = np.asarray(confidence, dtype="float64")
+    error = np.asarray(error, dtype="float64")
+    order = np.argsort(-confidence)
+    e = error[order]
+    risk = np.cumsum(e) / np.arange(1, len(e) + 1)
+    return float(risk.mean())
+
+
+def is_rank_noop(confidence, transform, error, tol: float = 1e-9) -> bool:
+    """True if applying ``transform`` to ``confidence`` leaves AURC unchanged."""
+    a = aurc(confidence, error)
+    b = aurc(transform(np.asarray(confidence, dtype="float64")), error)
+    return abs(a - b) <= tol
+
+
+def agreement_confidence(primary, secondary, continuous: bool = True):
+    """Confidence from agreement between two independent predictions.
+
+    ``continuous=True`` returns negative normalized distance (works for numbers or
+    equal-length strings via character distance); otherwise exact-match (1.0/0.0).
+    Higher = more agreement = more confident.
+    """
+    out = []
+    for p, s in zip(primary, secondary):
+        if not continuous:
+            out.append(1.0 if p == s else 0.0)
+        elif isinstance(p, (int, float)) and isinstance(s, (int, float)):
+            out.append(-abs(float(p) - float(s)))
+        else:
+            p, s = str(p), str(s)
+            m = max(len(p), len(s)) or 1
+            dist = sum(a != b for a, b in zip(p, s)) + abs(len(p) - len(s))
+            out.append(-dist / m)
+    return np.asarray(out, dtype="float64")

kvquant_hf-0.2.0/kvquant/storage.py

@@ -0,0 +1,74 @@
+"""Real int8 storage for a KV cache — actual bytes saved, not just a quality probe.
+
+``Int8KVStore`` holds a ``[batch, heads, seq, head_dim]`` K or V tensor as int8 plus a
+per-token fp16 scale, so its footprint is ~1/4 of fp32 (~1/2 of fp16) with <1% scale
+overhead. Per-token scaling (over the ``head_dim`` axis) is append-friendly: every token
+is quantized independently, so a growing cache never needs re-quantizing.
+
+Use it to compress a populated cache for offload / serialization / holding an idle
+long context. For int8 *during* attention (read-from-int8 every step) pair this scheme
+with transformers' ``QuantizedCache``; kvquant's value-add is the tested scheme, the
+stress guard, and the measured numbers.
+"""
+from __future__ import annotations
+import torch
+
+
+def _bytes(t: torch.Tensor) -> int:
+    return t.element_size() * t.nelement()
+
+
+class Int8KVStore:
+    """int8 + per-token fp16 scale storage for one K or V tensor."""
+
+    def __init__(self, fp: torch.Tensor):
+        if fp.ndim < 2:
+            raise ValueError("expected a [..., seq, head_dim] tensor")
+        scale = fp.abs().amax(dim=-1, keepdim=True) / 127.0
+        scale = torch.where(scale == 0, torch.ones_like(scale), scale)
+        self.q = torch.round(fp / scale).clamp(-127, 127).to(torch.int8)
+        self.scale = scale.to(torch.float16)
+        self.dtype = fp.dtype
+        self.shape = tuple(fp.shape)
+
+    def dequantize(self) -> torch.Tensor:
+        return (self.q.float() * self.scale.float()).to(self.dtype)
+
+    @property
+    def nbytes(self) -> int:
+        return _bytes(self.q) + _bytes(self.scale)
+
+
+def cache_bytes(cache) -> int:
+    """Live byte footprint of a transformers>=5 cache's K/V tensors."""
+    total = 0
+    for layer in getattr(cache, "layers", []):
+        for name in ("keys", "values"):
+            t = getattr(layer, name, None)
+            if t is not None:
+                total += _bytes(t)
+    return total
+
+
+def compress_cache(cache):
+    """Return ``[(Int8KVStore, Int8KVStore), ...]`` per layer and the measured savings.
+
+    Returns ``(stores, report)`` where ``report`` has ``fp_bytes``, ``int8_bytes`` and
+    ``ratio`` (fp / int8). The original cache is left untouched.
+    """
+    stores = []
+    fp_bytes = int8_bytes = 0
+    for layer in getattr(cache, "layers", []):
+        k, v = getattr(layer, "keys", None), getattr(layer, "values", None)
+        if k is None:
+            continue
+        sk, sv = Int8KVStore(k), Int8KVStore(v)
+        stores.append((sk, sv))
+        fp_bytes += _bytes(k) + _bytes(v)
+        int8_bytes += sk.nbytes + sv.nbytes
+    report = {
+        "fp_bytes": fp_bytes,
+        "int8_bytes": int8_bytes,
+        "ratio": (fp_bytes / int8_bytes) if int8_bytes else 0.0,
+    }
+    return stores, report

kvquant_hf-0.2.0/kvquant_hf.egg-info/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.4
+Name: kvquant-hf
+Version: 0.2.0
+Summary: Tested KV-cache quantization for Hugging Face transformers >= 5 (with a silent-no-op stress test).
+License: MIT
+Keywords: llm,kv-cache,quantization,transformers,inference,efficiency
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.1
+Provides-Extra: bench
+Requires-Dist: transformers>=5.0; extra == "bench"
+Requires-Dist: numpy; extra == "bench"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: numpy; extra == "dev"
+Dynamic: license-file
+
+# kvquant
+
+**Tested KV-cache quantization for Hugging Face `transformers >= 5` — with a stress test so a silent no-op can't fool you.**
+
+Most "quantize the KV cache" snippets you'll find mutate `cache.key_cache[i]`. On
+`transformers 5.x` that is a **non-persisting compatibility accessor**: your edit is
+thrown away and the model keeps using the original fp tensors. Everything *looks* like
+it worked — same code, no error — but perplexity doesn't move because nothing changed.
+The real tensors live at `cache.layers[i].keys` / `.values`. kvquant mutates the right
+ones and ships a guard that **proves** the quantization actually bit.
+
+## Install
+
+```bash
+pip install kvquant-hf            # core
+pip install "kvquant-hf[bench]"   # + transformers/numpy for the perplexity benchmark
+```
+
+## Quickstart
+
+```python
+from kvquant import quantize_cache          # in-place int-N quant of a transformers>=5 Cache
+change = quantize_cache(past_key_values, bits=8)   # returns mean |change|; 0.0 == a no-op bug
+```
+
+Measure the quality cost yourself:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from kvquant import kv_ppl_sweep
+m = AutoModelForCausalLM.from_pretrained("google/gemma-3-270m-it")
+t = AutoTokenizer.from_pretrained("google/gemma-3-270m-it")
+print(kv_ppl_sweep(m, t))            # {None/8/4/2: {ppl, delta_pct, change}}
+```
+
+or from the shell:
+
+```bash
+kvquant-bench --model google/gemma-3-270m-it
+```
+
+## What int8 KV actually costs (measured)
+
+`google/gemma-3-270m-it`, 300 tokens, per-channel symmetric int quant of K and V:
+
+| KV cache | perplexity | Δ vs fp | note |
+|---|---|---|---|
+| fp (baseline) | 4.5479 | — | |
+| **int8** | 4.5437 | **−0.09%** | within noise — effectively lossless |
+| int4 | 4.7528 | +4.5% | visible degradation |
+| int2 | 982.76 | +21509% | breaks (this is the stress check) |
+
+int8 K/V carries **half** the bytes of fp16 (a quarter of fp32), so an int8 cache is
+the obvious lever for longer context / larger batch at near-zero quality cost. Run the
+sweep on *your* model — numbers vary by architecture.
+
+### Real memory, measured (not just quality)
+
+`kvquant.compress_cache` stores a populated cache as int8 + a per-token fp16 scale and
+reports the actual byte reduction:
+
+```python
+from kvquant import compress_cache
+stores, report = compress_cache(past_key_values)
+print(report)   # {'fp_bytes': 5_824_512, 'int8_bytes': 1_467_504, 'ratio': 3.97}
+```
+
+On the gemma cache above (158 tokens, 18 layers): **5,824,512 → 1,467,504 bytes = 3.97×
+smaller vs fp32 (~2× vs fp16)**, ~2% per-tensor reconstruction error. Use it to compress
+a cache for offload / serialization / holding an idle long context.
+
+### Live int8 cache (holds int8 *during* generation)
+
+`make_live_int8` converts a cache so every layer **stores** int8 + a per-token fp16 scale
+and dequantizes on read inside attention — the memory held is really int8, no extra deps:
+
+```python
+from kvquant import make_live_int8, held_bytes
+
+past = None
+for step in decode_loop:
+    out = model(next_token, past_key_values=past, use_cache=True)
+    past = make_live_int8(out.past_key_values)   # cache now holds int8
+```
+
+Measured on `gemma-3-270m-it` (300 tokens, autoregressive): perplexity **+0.74%**, cache
+held as `torch.int8` and **3.97× smaller than fp32 (~2× vs fp16)**. It subclasses each
+layer's own class, so sliding-window and other behaviours are preserved.
+
+## The silent-no-op guard
+
+`kv_ppl_sweep` asserts that int2 both changes the tensors (`change > 0`) **and** wrecks
+perplexity. If it doesn't, you're mutating the wrong thing and the whole measurement is
+meaningless — so it raises instead of printing a comfortable lie. The same idea is
+packaged as a standalone decorator:
+
+```python
+from kvquant.doesitbite import bites
+
+@bites()                      # raises AssertionError if the transform doesn't change its input
+def my_quantizer(x): ...
+```
+
+## Also included
+
+- **`kvquant.doesitbite`** — `@bites` / `assert_bites` to catch transforms that silently
+  do nothing (mis-targeted in-place edits, dead paths, clamps that never fire).
+- **`kvquant.rankconf`** — confidence utilities for rank metrics (AURC/risk-coverage):
+  `aurc`, `is_rank_noop` (any strictly-monotonic rescale is a no-op for a rank metric),
+  and `agreement_confidence` (a stronger signal than a model's self-reported confidence).
+
+## Honest limitations (v0)
+
+- Three tools, clear jobs: `kv_ppl_sweep` / `quantize_cache` **fake-quantize** (quant→dequant)
+  to measure the *quality* cost with the correct non-no-op hook; `compress_cache` /
+  `Int8KVStore` do **real int8 storage** for a finished cache; `make_live_int8` holds int8
+  **during** generation. All measured above.
+- Symmetric quantization, no outlier handling or post-RoPE split yet (those shrink the
+  int8 quality gap further).
+- Benchmarked on small models on a laptop (gemma-3-270m). Try it on your model and
+  architecture and report back — especially GQA/MQA and sliding-window variants.
+
+## Support
+
+kvquant is free and MIT-licensed. If it saved you GPU memory or an afternoon of
+debugging, a tip keeps the work going — no obligation.
+
+- **GitHub Sponsors:** the ❤ Sponsor button above (once enabled)
+- **BTC:** `bc1q9hm4upksmvkzhl8z3zvxgz94da358qfe3tred0`
+- **ETH / Polygon / USDT (ERC-20):** `0xa9e3b63799f5E5cc6258042D45869a5a1d39F021`
+- **USDT (TRC-20) / TRON:** `TUyNkMXme1CYAc1wZ8SZ64X9upBc9nwTZ5`
+- **Solana:** `j8zZCsWwM8WmUZwMSDHikNuz6E49Vn8nwo9FMQnHSoW`
+
+## License
+
+MIT.

kvquant_hf-0.2.0/kvquant_hf.egg-info/SOURCES.txt

@@ -0,0 +1,17 @@
+LICENSE
+README.md
+pyproject.toml
+kvquant/__init__.py
+kvquant/bench.py
+kvquant/core.py
+kvquant/doesitbite.py
+kvquant/live.py
+kvquant/rankconf.py
+kvquant/storage.py
+kvquant_hf.egg-info/PKG-INFO
+kvquant_hf.egg-info/SOURCES.txt
+kvquant_hf.egg-info/dependency_links.txt
+kvquant_hf.egg-info/entry_points.txt
+kvquant_hf.egg-info/requires.txt
+kvquant_hf.egg-info/top_level.txt
+tests/test_all.py

kvquant_hf-0.2.0/kvquant_hf.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

kvquant_hf-0.2.0/kvquant_hf.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+kvquant-bench = kvquant.bench:_main

kvquant_hf-0.2.0/kvquant_hf.egg-info/requires.txt

@@ -0,0 +1,9 @@
+torch>=2.1
+
+[bench]
+transformers>=5.0
+numpy
+
+[dev]
+pytest
+numpy

kvquant_hf-0.2.0/kvquant_hf.egg-info/top_level.txt

@@ -0,0 +1 @@
+kvquant

kvquant_hf-0.2.0/pyproject.toml

@@ -0,0 +1,23 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "kvquant-hf"
+version = "0.2.0"
+description = "Tested KV-cache quantization for Hugging Face transformers >= 5 (with a silent-no-op stress test)."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+keywords = ["llm", "kv-cache", "quantization", "transformers", "inference", "efficiency"]
+dependencies = ["torch>=2.1"]
+
+[project.optional-dependencies]
+bench = ["transformers>=5.0", "numpy"]
+dev = ["pytest", "numpy"]
+
+[project.scripts]
+kvquant-bench = "kvquant.bench:_main"
+
+[tool.setuptools.packages.find]
+include = ["kvquant*"]

kvquant_hf-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

kvquant_hf-0.2.0/tests/test_all.py

@@ -0,0 +1,76 @@
+import numpy as np
+import pytest
+import torch
+
+from kvquant.core import quantize_dequantize, quantize_cache
+from kvquant import doesitbite as dib
+from kvquant import rankconf as rc
+
+
+def test_quantize_dequantize_bits():
+    x = torch.randn(1, 2, 64, 16)
+    q8 = quantize_dequantize(x, 8)
+    q2 = quantize_dequantize(x, 2)
+    assert q8.shape == x.shape
+    # coarser quantization must distort more
+    assert (q2 - x).abs().mean() > (q8 - x).abs().mean() > 0
+
+
+def test_quantize_cache_rejects_wrong_object():
+    with pytest.raises(TypeError):
+        quantize_cache(object())  # no .layers -> the transformers-5.x gotcha message
+
+
+def test_doesitbite_catches_noop():
+    @dib.bites()
+    def identity(x):
+        return x.clone()
+    with pytest.raises(AssertionError):
+        identity(torch.ones(10))
+
+    @dib.bites()
+    def real(x):
+        return x + 1
+    assert real(torch.zeros(5)).sum().item() == 5
+
+
+def test_rankconf_monotonic_noop():
+    rng = np.random.default_rng(0)
+    conf = rng.random(2000)
+    err = rng.random(2000) ** 2
+    for f in (lambda x: 1 / (1 + np.exp(-x)), lambda x: x ** 3, np.exp):
+        assert rc.is_rank_noop(conf, f, err)
+
+
+def test_rankconf_agreement_beats_self():
+    rng = np.random.default_rng(1)
+    n = 3000
+    err = rng.random(n) ** 2
+    self_conf = -0.3 * err + rng.normal(0, 0.3, n)
+    agree = -err + rng.normal(0, 0.25, n)
+    assert rc.aurc(agree, err) < rc.aurc(self_conf, err)
+
+
+def test_int8_store_saves_bytes_and_reconstructs():
+    from kvquant.storage import Int8KVStore
+    x = torch.randn(1, 4, 128, 64)  # [B, H, seq, head_dim]
+    s = Int8KVStore(x)
+    # ~4x smaller than fp32 (int8 + tiny fp16 per-token scale)
+    assert s.nbytes < x.element_size() * x.nelement() / 3.5
+    rel = (s.dequantize() - x).abs().mean() / x.abs().mean()
+    assert rel < 0.05  # int8 per-token reconstruction stays within a few percent
+
+
+def test_live_int8_cache_holds_int8():
+    import torch
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+    from kvquant.live import make_live_int8, held_bytes
+    tok = AutoTokenizer.from_pretrained("google/gemma-3-270m-it")
+    m = AutoModelForCausalLM.from_pretrained("google/gemma-3-270m-it", dtype=torch.float32).eval()
+    ids = tok("live int8 cache test " * 8, return_tensors="pt").input_ids[:, :40]
+    with torch.no_grad():
+        cache = m(ids, use_cache=True).past_key_values
+    fp = held_bytes(cache)
+    make_live_int8(cache)
+    assert cache.layers[0]._qk.dtype == torch.int8
+    assert held_bytes(cache) < fp / 3          # real int8 storage, ~4x vs fp32