aither-kvcache 2.2.0__tar.gz → 2.4.0__tar.gz

{aither_kvcache-2.2.0/aither_kvcache.egg-info → aither_kvcache-2.4.0}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: aither-kvcache
-Version: 2.2.0
-Summary: Near-optimal KV cache compression for LLM inference — TurboQuant (vector quantization) + TriAttention (spectral compression)
+Version: 2.4.0
+Summary: KV cache tooling for LLM inference — TurboQuant (vector quantization), TriAttention (spectral compression), and cross-model KV transfer (skip prefill by converting one model's cache into another's)
 Author: Aitherium
 License-Expression: CC-BY-4.0
 Project-URL: Paper, https://arxiv.org/abs/2504.19874
 Project-URL: Repository, https://github.com/Aitherium/aitherkvcache
 Project-URL: Changelog, https://github.com/Aitherium/aitherkvcache/releases
-Keywords: kv-cache,quantization,llm,inference,compression,turboquant,triattention,spectral,vllm,attention
+Keywords: kv-cache,quantization,llm,inference,compression,turboquant,triattention,spectral,vllm,attention,kv-transfer,cross-model,prefill,rope,mla
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python :: 3
@@ -29,10 +29,13 @@ Provides-Extra: dev
 Requires-Dist: pytest; extra == "dev"
 Provides-Extra: vllm
 Requires-Dist: vllm>=0.15; extra == "vllm"
+Provides-Extra: transfer
+Requires-Dist: transformers>=4.40; extra == "transfer"
 Provides-Extra: all
 Requires-Dist: triton>=2.0; extra == "all"
 Requires-Dist: scipy; extra == "all"
 Requires-Dist: pytest; extra == "all"
+Requires-Dist: transformers>=4.40; extra == "all"
 Dynamic: license-file
 
 # aither-kvcache
@@ -46,12 +49,18 @@ Near-optimal KV cache compression for LLM inference. Two compression engines:
   Retains top RoPE frequency pairs, scores via trig series without materializing full K/V.
   ~10× compression with bounded approximation error. Calibrated for Qwen3.5 family.
 
+- **KVTransfer** *(NEW in v2.4)* — Cross-model KV transfer. Convert one model's KV cache
+  into another model's so the receiving model **skips prefill entirely**. Per-(layer, head)
+  linear maps fitted by ridge; source layers chosen per target layer by held-out R².
+  Same model family only — the two models must share a tokenizer.
+
 ## Installation
 
 ```bash
 pip install aither-kvcache            # core library
 pip install aither-kvcache[vllm]      # + vLLM plugin (v0.15+)
 pip install aither-kvcache[triton]    # + fused GPU kernels
+pip install aither-kvcache[transfer]  # + cross-model KV transfer
 pip install aither-kvcache[all]       # everything
 ```
 
@@ -126,6 +135,95 @@ instead of 256 — a ~10× reduction with bounded approximation error.
 | F=16, int4 | 4 | 34 | 7.53× |
 | F=8, int4 | 4 | 18 | **14.2×** |
 
+## Quick Start — KVTransfer (v2.4+)
+
+Prefill is the tax you pay before a model says anything, and a KV cache only works on the
+model that produced it. Route a conversation to a different model and the accumulated
+cache becomes dead weight. KVTransfer fits a map that converts one model's cache into
+another's, so the receiving model can skip prefill.
+
+```bash
+# 1. capture aligned KV from both models over one corpus
+python -m aither_kvcache.kvtransfer.capture \
+    --source Qwen/Qwen3-0.6B --target Qwen/Qwen3-4B \
+    --out ./cap --corpus ./my-corpus --seq-len 256 --train-seqs 64 --val-seqs 24
+
+# 2. fit the mapper pack
+python -m aither_kvcache.kvtransfer.fit --capture ./cap --out ./pack --top-k 8
+
+# 3. measure what the RECEIVING model does with a translated cache, and record it
+python -m aither_kvcache.kvtransfer.transfer \
+    --pack ./pack --source Qwen/Qwen3-0.6B --target Qwen/Qwen3-4B \
+    --corpus ./held-out --sequences 24 --cuts 32 64 96 128 160 192 224 --record
+```
+
+```python
+from aither_kvcache.kvtransfer import load_pack
+from aither_kvcache.kvtransfer.transfer import translate_cache
+
+pack = load_pack("./pack", live_source=src_geom, live_target=tgt_geom)
+layers = translate_cache(src_k, src_v, pack, positions)   # -> the target's KV cache
+```
+
+### Step 3 is not optional
+
+`load_pack` **refuses** any pack that carries no downstream acceptance evidence. That is
+the core design decision here, and it is not defensive programming — it is the only thing
+standing between an interesting R² and a broken deployment.
+
+A converted cache does not fail loudly. An approximate one makes the receiving model
+produce fluent, on-topic, **confidently wrong** text: no exception, no error status, no
+unhealthy process, nothing in a log. Every cheap signal is green.
+
+So a pack must carry a measurement, and the measurement has **four arms**:
+
+| arm | what it is |
+|---|---|
+| `reference` | the target prefills the context itself — the ceiling |
+| `translated` | the target is handed the converted cache — the candidate |
+| `control` | the map is run on a **different document** — catches a mapper ignoring its input |
+| `nocontext` | no cache at all — the floor |
+
+The control arm is the one that matters. A mapper that has quietly learned the target's
+*average* key/value statistics scores respectably against `reference` alone. Without a
+floor, "68% agreement" is unreadable — it could be excellent, or exactly what a dead
+mapper gets.
+
+A minimum sample size is enforced too. Top-1 agreement is a proportion, so a mean over a
+few dozen positions has a standard error near ten percentage points — and once recorded it
+is indistinguishable from a mean over ten thousand.
+
+### Preconditions that are definitional, not quality knobs
+
+- **The two models must tokenize identically.** The map sends position *i* to position
+  *i*; different tokenizers mean row *i* is not the same token and the fit is regressing
+  misaligned data. Nothing downstream can see this — it reads as "this pair transfers
+  poorly" rather than "these rows do not correspond". Enforced by vocabulary digest, and
+  by comparing token ids per document at capture time.
+- **The RoPE schedule must be exactly reproducible.** Keys carry a position rotation, so
+  the map is fitted in position-free space: stripped with the *source* schedule, re-applied
+  with the *target's* (not a no-op — the two models often disagree on `rope_theta`).
+  `default`, `linear` and `yarn` are implemented; anything else is refused rather than
+  approximated.
+
+`check_pair()` separates definitional refusals from merely-unproven regimes — mismatched
+KV-head counts, differing head dimension, cross-family pairs. The latter are recorded as
+flags and decided by measurement, because a gate keyed on "heads must match" would refuse
+the very experiment that tests whether heads need to match.
+
+### Latent attention (MLA)
+
+Models such as DeepSeek V2/V3/V4 cache a compressed latent plus a decoupled RoPE key
+rather than per-head K and V. That is a different cache **layout**, not an obstacle — the
+latent is dense, and often smaller than a conventional cache for the same context.
+`KVGeometry.roles()` returns `("c", "kr")` for those models, and only `kr` carries a
+rotation because the latent is position-free by construction.
+
+The latent is captured at the compression projection rather than from `past_key_values`:
+HuggingFace materialises per-head K/V before caching, so reading the cache would yield
+tensors a real serving engine never stores, and a mapper fitted to them could not drive a
+production deployment while looking entirely correct end to end.
+
 ## vLLM Integration
 
 ### Native kv-cache-dtype (recommended)

aither_kvcache-2.2.0/PKG-INFO → aither_kvcache-2.4.0/README.md

@@ -1,40 +1,3 @@
-Metadata-Version: 2.4
-Name: aither-kvcache
-Version: 2.2.0
-Summary: Near-optimal KV cache compression for LLM inference — TurboQuant (vector quantization) + TriAttention (spectral compression)
-Author: Aitherium
-License-Expression: CC-BY-4.0
-Project-URL: Paper, https://arxiv.org/abs/2504.19874
-Project-URL: Repository, https://github.com/Aitherium/aitherkvcache
-Project-URL: Changelog, https://github.com/Aitherium/aitherkvcache/releases
-Keywords: kv-cache,quantization,llm,inference,compression,turboquant,triattention,spectral,vllm,attention
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Science/Research
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: torch>=2.0
-Requires-Dist: numpy
-Provides-Extra: triton
-Requires-Dist: triton>=2.0; extra == "triton"
-Provides-Extra: scipy
-Requires-Dist: scipy; extra == "scipy"
-Provides-Extra: dev
-Requires-Dist: pytest; extra == "dev"
-Provides-Extra: vllm
-Requires-Dist: vllm>=0.15; extra == "vllm"
-Provides-Extra: all
-Requires-Dist: triton>=2.0; extra == "all"
-Requires-Dist: scipy; extra == "all"
-Requires-Dist: pytest; extra == "all"
-Dynamic: license-file
-
 # aither-kvcache
 
 Near-optimal KV cache compression for LLM inference. Two compression engines:
@@ -46,12 +9,18 @@ Near-optimal KV cache compression for LLM inference. Two compression engines:
   Retains top RoPE frequency pairs, scores via trig series without materializing full K/V.
   ~10× compression with bounded approximation error. Calibrated for Qwen3.5 family.
 
+- **KVTransfer** *(NEW in v2.4)* — Cross-model KV transfer. Convert one model's KV cache
+  into another model's so the receiving model **skips prefill entirely**. Per-(layer, head)
+  linear maps fitted by ridge; source layers chosen per target layer by held-out R².
+  Same model family only — the two models must share a tokenizer.
+
 ## Installation
 
 ```bash
 pip install aither-kvcache            # core library
 pip install aither-kvcache[vllm]      # + vLLM plugin (v0.15+)
 pip install aither-kvcache[triton]    # + fused GPU kernels
+pip install aither-kvcache[transfer]  # + cross-model KV transfer
 pip install aither-kvcache[all]       # everything
 ```
 
@@ -126,6 +95,95 @@ instead of 256 — a ~10× reduction with bounded approximation error.
 | F=16, int4 | 4 | 34 | 7.53× |
 | F=8, int4 | 4 | 18 | **14.2×** |
 
+## Quick Start — KVTransfer (v2.4+)
+
+Prefill is the tax you pay before a model says anything, and a KV cache only works on the
+model that produced it. Route a conversation to a different model and the accumulated
+cache becomes dead weight. KVTransfer fits a map that converts one model's cache into
+another's, so the receiving model can skip prefill.
+
+```bash
+# 1. capture aligned KV from both models over one corpus
+python -m aither_kvcache.kvtransfer.capture \
+    --source Qwen/Qwen3-0.6B --target Qwen/Qwen3-4B \
+    --out ./cap --corpus ./my-corpus --seq-len 256 --train-seqs 64 --val-seqs 24
+
+# 2. fit the mapper pack
+python -m aither_kvcache.kvtransfer.fit --capture ./cap --out ./pack --top-k 8
+
+# 3. measure what the RECEIVING model does with a translated cache, and record it
+python -m aither_kvcache.kvtransfer.transfer \
+    --pack ./pack --source Qwen/Qwen3-0.6B --target Qwen/Qwen3-4B \
+    --corpus ./held-out --sequences 24 --cuts 32 64 96 128 160 192 224 --record
+```
+
+```python
+from aither_kvcache.kvtransfer import load_pack
+from aither_kvcache.kvtransfer.transfer import translate_cache
+
+pack = load_pack("./pack", live_source=src_geom, live_target=tgt_geom)
+layers = translate_cache(src_k, src_v, pack, positions)   # -> the target's KV cache
+```
+
+### Step 3 is not optional
+
+`load_pack` **refuses** any pack that carries no downstream acceptance evidence. That is
+the core design decision here, and it is not defensive programming — it is the only thing
+standing between an interesting R² and a broken deployment.
+
+A converted cache does not fail loudly. An approximate one makes the receiving model
+produce fluent, on-topic, **confidently wrong** text: no exception, no error status, no
+unhealthy process, nothing in a log. Every cheap signal is green.
+
+So a pack must carry a measurement, and the measurement has **four arms**:
+
+| arm | what it is |
+|---|---|
+| `reference` | the target prefills the context itself — the ceiling |
+| `translated` | the target is handed the converted cache — the candidate |
+| `control` | the map is run on a **different document** — catches a mapper ignoring its input |
+| `nocontext` | no cache at all — the floor |
+
+The control arm is the one that matters. A mapper that has quietly learned the target's
+*average* key/value statistics scores respectably against `reference` alone. Without a
+floor, "68% agreement" is unreadable — it could be excellent, or exactly what a dead
+mapper gets.
+
+A minimum sample size is enforced too. Top-1 agreement is a proportion, so a mean over a
+few dozen positions has a standard error near ten percentage points — and once recorded it
+is indistinguishable from a mean over ten thousand.
+
+### Preconditions that are definitional, not quality knobs
+
+- **The two models must tokenize identically.** The map sends position *i* to position
+  *i*; different tokenizers mean row *i* is not the same token and the fit is regressing
+  misaligned data. Nothing downstream can see this — it reads as "this pair transfers
+  poorly" rather than "these rows do not correspond". Enforced by vocabulary digest, and
+  by comparing token ids per document at capture time.
+- **The RoPE schedule must be exactly reproducible.** Keys carry a position rotation, so
+  the map is fitted in position-free space: stripped with the *source* schedule, re-applied
+  with the *target's* (not a no-op — the two models often disagree on `rope_theta`).
+  `default`, `linear` and `yarn` are implemented; anything else is refused rather than
+  approximated.
+
+`check_pair()` separates definitional refusals from merely-unproven regimes — mismatched
+KV-head counts, differing head dimension, cross-family pairs. The latter are recorded as
+flags and decided by measurement, because a gate keyed on "heads must match" would refuse
+the very experiment that tests whether heads need to match.
+
+### Latent attention (MLA)
+
+Models such as DeepSeek V2/V3/V4 cache a compressed latent plus a decoupled RoPE key
+rather than per-head K and V. That is a different cache **layout**, not an obstacle — the
+latent is dense, and often smaller than a conventional cache for the same context.
+`KVGeometry.roles()` returns `("c", "kr")` for those models, and only `kr` carries a
+rotation because the latent is position-free by construction.
+
+The latent is captured at the compression projection rather than from `past_key_values`:
+HuggingFace materialises per-head K/V before caching, so reading the cache would yield
+tensors a real serving engine never stores, and a mapper fitted to them could not drive a
+production deployment while looking entirely correct end to end.
+
 ## vLLM Integration
 
 ### Native kv-cache-dtype (recommended)

{aither_kvcache-2.2.0 → aither_kvcache-2.4.0}/aither_kvcache/__init__.py

@@ -40,7 +40,7 @@ Usage:
     candidates = advisor.get_eviction_candidates(n=16)
 """
 
-from .quantizer import TurboQuant, TurboQuantConfig
+from turboquant import TurboQuant, TurboQuantConfig
 from .hybrid_quantizer import HybridTurboQuant, HybridLayout, GroupLayout
 from .kvcache_graph import KVCacheGraph, KVBlockNode, KVEdge, EdgeType, get_kvcache_graph
 from .eviction_advisor import GraphEvictionAdvisor, reorder_by_ranking
@@ -56,4 +56,4 @@ __all__ = [
     # TriAttention — lazy-loaded via subpackage
     # from aither_kvcache.triattention import TriAttention, TriAttentionConfig
 ]
-__version__ = "2.0.1"
+__version__ = "2.1.0"

{aither_kvcache-2.2.0 → aither_kvcache-2.4.0}/aither_kvcache/block_selector.py

@@ -20,7 +20,6 @@ Integration:
 """
 
 import math
-import os
 import torch
 from typing import Optional, Tuple
 
@@ -107,7 +106,6 @@ class BlockRepresentativeCache:
         If block_indices is None, finalizes all blocks with count > 0.
         """
         if block_indices is not None:
-            mask = self._counts[block_indices] > 0
             safe_count = self._counts[block_indices].clamp(min=1).unsqueeze(-1).float()
             self.rep_even[block_indices] = self._acc_even[block_indices] / safe_count
             self.rep_odd[block_indices] = self._acc_odd[block_indices] / safe_count

{aither_kvcache-2.2.0 → aither_kvcache-2.4.0}/aither_kvcache/fused_kv_update.py

@@ -114,7 +114,6 @@ if HAS_TRITON:
         # --- Step 3: compute L2 norm and normalize ---
         norm_sq = tl.sum(x_vec * x_vec)
         norm = tl.sqrt(norm_sq + 1e-20)
-        x_unit = x_vec / norm
 
         # --- Step 4: tiled rotation y = rotation @ x_unit ---
         # rotation is [HEAD_DIM, HEAD_DIM] stored row-major.

{aither_kvcache-2.2.0 → aither_kvcache-2.4.0}/aither_kvcache/hybrid_quantizer.py

@@ -36,7 +36,6 @@ from __future__ import annotations
 
 import functools
 import math
-import time
 from dataclasses import dataclass
 from typing import Dict, Optional, Tuple
 
@@ -405,7 +404,6 @@ def _pack_sign_bits(x: torch.Tensor) -> torch.Tensor:
     Returns:
         [..., ceil(dim / 8)] uint8 tensor.
     """
-    dim = x.shape[-1]
     signs = (x >= 0).to(torch.uint8)     # 1 for positive, 0 for negative
     return pack_multibit(signs, bits=1)
 
@@ -679,8 +677,6 @@ class HybridTurboQuant:
         Returns:
             [N, packed_dim] uint8
         """
-        N = x.shape[0]
-        device = x.device
 
         # Gather group dimensions
         x_g0 = x[:, g0_idx]  # [N, g0_dim]
@@ -733,9 +729,6 @@ class HybridTurboQuant:
         Returns:
             [N, layout.total_bytes] uint8 packed tensor.
         """
-        N = x_group.shape[0]
-        G = x_group.shape[1]
-        device = x_group.device
 
         # 1. Compute L2 norm
         vec_norm = x_group.norm(dim=-1)                       # [N]

aither_kvcache-2.4.0/aither_kvcache/kvtransfer/__init__.py

@@ -0,0 +1,70 @@
+"""Cross-model KV cache transfer.
+
+Convert one model's KV cache into another model's, so the receiving model can
+skip prefill entirely. Per-(target layer, head) affine maps fitted by ridge
+regression, with source layers chosen per target layer by held-out R².
+
+    capture  ->  fit  ->  measure  ->  translate
+
+Each stage is a CLI as well as an API::
+
+    python -m aither_kvcache.kvtransfer.capture  --source A --target B --out CAP
+    python -m aither_kvcache.kvtransfer.fit      --capture CAP --out PACK
+    python -m aither_kvcache.kvtransfer.transfer --pack PACK --source A --target B --record
+
+**The measure step is not optional.** ``load_pack`` refuses any pack that does
+not carry downstream acceptance evidence over a large enough sample, because a
+converted cache fails silently: an approximate one does not raise, it makes the
+receiving model produce fluent, on-topic, confidently wrong text with a green
+healthcheck and nothing in a log.
+
+Two preconditions are definitional rather than quality knobs. The two models
+must **tokenize identically** — the map sends position *i* to position *i*, so
+different tokenizers mean the fit is regressing misaligned rows — and the RoPE
+schedule must be exactly reproducible. Both are enforced.
+
+Only ``numpy`` is needed for the mapper, the pack format and the rotation math.
+``capture`` and ``transfer``'s evaluation additionally need ``torch`` and
+``transformers``; they are imported lazily so the rest of the package works
+without them.
+"""
+
+from __future__ import annotations
+
+from .geometry import KVGeometry, PairVerdict, check_pair, source_roles_for
+from .mapper import GramStats, LayerMap, apply_layer_map, fit_layer_map, r2_from_gram
+from .pack import (
+    AcceptanceMetrics,
+    LoadedPack,
+    PackManifest,
+    PackRefusedError,
+    load_pack,
+    record_acceptance,
+    write_pack,
+)
+from .rope import RopeSpec, apply_rope, rotate_to_target, strip_rope
+
+__all__ = [
+    "AcceptanceMetrics",
+    "GramStats",
+    "KVGeometry",
+    "LayerMap",
+    "LoadedPack",
+    "PackManifest",
+    "PackRefusedError",
+    "PairVerdict",
+    "RopeSpec",
+    "apply_layer_map",
+    "apply_rope",
+    "check_pair",
+    "fit_layer_map",
+    "load_pack",
+    "r2_from_gram",
+    "record_acceptance",
+    "rotate_to_target",
+    "source_roles_for",
+    "strip_rope",
+    "write_pack",
+]
+
+__version__ = "2.4.0"