dag-kv 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. dag_kv-0.2.0/LICENSE +17 -0
  2. dag_kv-0.2.0/PKG-INFO +351 -0
  3. dag_kv-0.2.0/README.md +329 -0
  4. dag_kv-0.2.0/dag_kv/__init__.py +49 -0
  5. dag_kv-0.2.0/dag_kv/cli.py +204 -0
  6. dag_kv-0.2.0/dag_kv/core/__init__.py +45 -0
  7. dag_kv-0.2.0/dag_kv/core/block_manager.py +312 -0
  8. dag_kv-0.2.0/dag_kv/core/lse_probe.py +147 -0
  9. dag_kv-0.2.0/dag_kv/core/quantizer.py +89 -0
  10. dag_kv-0.2.0/dag_kv/core/rope_adjoint.py +159 -0
  11. dag_kv-0.2.0/dag_kv/core/sink_canonicalizer.py +156 -0
  12. dag_kv-0.2.0/dag_kv/devices/__init__.py +13 -0
  13. dag_kv-0.2.0/dag_kv/devices/device_manager.py +268 -0
  14. dag_kv-0.2.0/dag_kv/devices/shard_config.py +199 -0
  15. dag_kv-0.2.0/dag_kv/integrations/__init__.py +11 -0
  16. dag_kv-0.2.0/dag_kv/integrations/huggingface.py +145 -0
  17. dag_kv-0.2.0/dag_kv/integrations/vllm.py +90 -0
  18. dag_kv-0.2.0/dag_kv/kernels/__init__.py +35 -0
  19. dag_kv-0.2.0/dag_kv/kernels/dispatch.py +152 -0
  20. dag_kv-0.2.0/dag_kv/kernels/pytorch_fallbacks.py +282 -0
  21. dag_kv-0.2.0/dag_kv/kernels/triton_adjoint_paged_attn.py +268 -0
  22. dag_kv-0.2.0/dag_kv/kernels/triton_sparse_stitch.py +259 -0
  23. dag_kv-0.2.0/dag_kv/models/__init__.py +5 -0
  24. dag_kv-0.2.0/dag_kv/models/qwen2_dag_patch.py +571 -0
  25. dag_kv-0.2.0/dag_kv/server.py +175 -0
  26. dag_kv-0.2.0/dag_kv/worker/__init__.py +10 -0
  27. dag_kv-0.2.0/dag_kv/worker/dual_gpu_pipeline.py +287 -0
  28. dag_kv-0.2.0/dag_kv/worker/universal_pipeline.py +232 -0
  29. dag_kv-0.2.0/dag_kv.egg-info/PKG-INFO +351 -0
  30. dag_kv-0.2.0/dag_kv.egg-info/SOURCES.txt +44 -0
  31. dag_kv-0.2.0/dag_kv.egg-info/dependency_links.txt +1 -0
  32. dag_kv-0.2.0/dag_kv.egg-info/entry_points.txt +2 -0
  33. dag_kv-0.2.0/dag_kv.egg-info/requires.txt +15 -0
  34. dag_kv-0.2.0/dag_kv.egg-info/top_level.txt +1 -0
  35. dag_kv-0.2.0/pyproject.toml +40 -0
  36. dag_kv-0.2.0/setup.cfg +4 -0
  37. dag_kv-0.2.0/tests/test_core_components.py +108 -0
  38. dag_kv-0.2.0/tests/test_device_manager.py +62 -0
  39. dag_kv-0.2.0/tests/test_dual_gpu_pipeline.py +30 -0
  40. dag_kv-0.2.0/tests/test_dynamic_blocks.py +80 -0
  41. dag_kv-0.2.0/tests/test_hf_integration.py +90 -0
  42. dag_kv-0.2.0/tests/test_pytorch_fallback.py +155 -0
  43. dag_kv-0.2.0/tests/test_quantization.py +94 -0
  44. dag_kv-0.2.0/tests/test_rope_adjoint.py +91 -0
  45. dag_kv-0.2.0/tests/test_sparse_bridge_e2e.py +97 -0
  46. dag_kv-0.2.0/tests/test_triton_sm75.py +128 -0
dag_kv-0.2.0/LICENSE ADDED
@@ -0,0 +1,17 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ Copyright 2026 Kaushal Nandaniya
6
+
7
+ Licensed under the Apache License, Version 2.0 (the "License");
8
+ you may not use this file except in compliance with the License.
9
+ You may obtain a copy of the License at
10
+
11
+ http://www.apache.org/licenses/LICENSE-2.0
12
+
13
+ Unless required by applicable law or agreed to in writing, software
14
+ distributed under the License is distributed on an "AS IS" BASIS,
15
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16
+ See the License for the specific language governing permissions and
17
+ limitations under the License.
dag_kv-0.2.0/PKG-INFO ADDED
@@ -0,0 +1,351 @@
1
+ Metadata-Version: 2.4
2
+ Name: dag-kv
3
+ Version: 0.2.0
4
+ Summary: Universal Position-Decoupled Graph KV Cache with Query-Side Adjoint RoPE & LSE-Bounded Sparse Bridge Repair
5
+ Author: Kaushal Nandaniya
6
+ License-Expression: Apache-2.0
7
+ Requires-Python: >=3.10
8
+ Description-Content-Type: text/markdown
9
+ License-File: LICENSE
10
+ Requires-Dist: torch>=2.2.0
11
+ Requires-Dist: transformers>=4.44.0
12
+ Requires-Dist: accelerate>=0.33.0
13
+ Requires-Dist: numpy>=1.26.0
14
+ Requires-Dist: pyyaml>=5.3.0
15
+ Requires-Dist: triton>=3.0.0; sys_platform == "linux"
16
+ Provides-Extra: dev
17
+ Requires-Dist: pytest>=8.0.0; extra == "dev"
18
+ Requires-Dist: pytest-cov; extra == "dev"
19
+ Provides-Extra: vllm
20
+ Requires-Dist: vllm>=0.6.0; extra == "vllm"
21
+ Dynamic: license-file
22
+
23
+ # DAG-KV: Universal Position-Decoupled Graph KV Cache with Query-Side Adjoint RoPE & LSE-Bounded Sparse Bridge Repair
24
+
25
+ [![Hardware: NVIDIA | AMD ROCm | Apple Silicon MPS | CPU](https://img.shields.io/badge/Hardware-NVIDIA%20%7C%20AMD%20ROCm%20%7C%20Apple%20Silicon%20%7C%20CPU-76B900)](https://pytorch.org)
26
+ [![Execution: Triton JIT + PyTorch Native Fallback](https://img.shields.io/badge/Kernels-Triton%20%2B%20PyTorch%20Fallback-orange)](dag_kv/kernels/)
27
+ [![Tests: 43/43 Passed](https://img.shields.io/badge/Tests-43%2F43%20Passed-brightgreen)](tests/)
28
+ [![Paper: PDF](https://img.shields.io/badge/Paper-PDF-red?logo=adobeacrobatreader)](paper/dag_kv_paper.pdf)
29
+ [![CI/CD: Multi-Arch Matrix](https://img.shields.io/badge/CI%2FCD-Ubuntu%20%7C%20macOS-blue?logo=githubactions)](.github/workflows/ci.yml)
30
+ [![Docker: Multi-Arch amd64/arm64](https://img.shields.io/badge/Docker-Multi--Arch%20(amd64%2Farm64)-2496ED?logo=docker)](Dockerfile)
31
+ [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-green.svg)](LICENSE)
32
+
33
+ ---
34
+
35
+ ## Executive Summary
36
+
37
+ Existing prefix-caching architectures (e.g., **vLLM Automatic Prefix Caching**, **SGLang RadixAttention**) suffer a catastrophic **0% cache hit rate** when context chunks are dynamically permuted, reordered, or multi-hop composed in Retrieval-Augmented Generation (RAG) and multi-engineer codebase assistant workflows (e.g., `[Sys, Doc_A, Doc_B]` vs `[Sys, Doc_B, Doc_A]`). This failure stems from two fundamental blockers:
38
+ 1. **RoPE Position Coupling:** Rotary Position Embeddings bake absolute sequence indices directly into cached Key vectors. Mutating cached keys in HBM to shift position offsets destroys Copy-on-Write (CoW) sharing across concurrent requests.
39
+ 2. **Missing Cross-Chunk Causal Attention:** Chunks prefilled in isolation miss attention interactions with predecessor chunks.
40
+
41
+ **DAG-KV** turns graph-structured context caching into a **universal, production-ready library** across all hardware architectures (NVIDIA, AMD ROCm, Apple Silicon MPS, and Host CPU):
42
+ - **Query-Side Adjoint RoPE:** Exploits the orthogonal adjoint identity in $SO(2)$ to counter-rotate query vectors in GPU/CPU registers ($q_{\text{adj}} = \mathcal{R}(-\Delta) \tilde{q}$), leaving cached Key blocks **100% read-only and zero-copy in memory**.
43
+ - **Canonical Sink Prefix:** Neutralizes the position-zero attention sink pathology by prepending a 4-token system sink during isolated prefill and slicing it off before registration.
44
+ - **LSE Perturbation Bound & Submodular Bridge Repair:** Uses Layer-1 Online Softmax perturbation scalars $\alpha_i = \sigma(\ell_{i \to \text{prior}} - \ell_i^{\text{local}})$ with quality-diversity non-maximum suppression to select sparse Bridge Tokens for ragged micro-prefill, storing repaired keys into request-private **Copy-on-Write (CoW) Shadow Blocks** while reusing **92%–96% of the canonical KV cache untouched**.
45
+ - **Hardware Abstraction Layer (HAL):** Dynamic device manager with auto-discovery, topology auto-configuration (Single-GPU, Multi-GPU Pipeline, Apple Silicon MPS, CPU, and CPU-offloaded KV pooling), adaptive INT8/INT4 quantization, and seamless drop-in integrations for **Hugging Face** and **vLLM**.
46
+
47
+ > **Research Paper:** Read the complete 3-page manuscript in [`paper/dag_kv_paper.pdf`](paper/dag_kv_paper.pdf).
48
+
49
+ ---
50
+
51
+ ## Universal Hardware Abstraction & Multi-Backend Architecture
52
+
53
+ DAG-KV abstracts away low-level GPU dependencies, providing automated backend routing based on the host environment:
54
+
55
+ ```
56
+ ┌──────────────────────────────────┐
57
+ │ User API / HTTP Endpoint / │
58
+ │ HuggingFace / vLLM Integrations│
59
+ └─────────────────┬────────────────┘
60
+ │
61
+ ▼
62
+ ┌──────────────────────────────────┐
63
+ │ Hardware Abstraction Layer │
64
+ │ DeviceManager & ShardConfig │
65
+ └─────────────────┬────────────────┘
66
+ │
67
+ ┌───────────────────────────────┴───────────────────────────────┐
68
+ ▼ ▼
69
+ ┌───────────────────────┐ ┌───────────────────────┐
70
+ │ High-Performance │ │ Universal Native │
71
+ │ Triton JIT Path │ │ PyTorch Fallback Path │
72
+ └───────────┬───────────┘ └───────────┬───────────┘
73
+ │ │
74
+ ┌────────┴────────┐ ┌────────┴────────┐
75
+ ▼ ▼ ▼ ▼
76
+ NVIDIA GPUs AMD ROCm Apple Silicon Host CPU
77
+ (Hopper, Ada, (MI200, MI300) (M1/M2/M3/M4 MPS) (x86_64, ARM64)
78
+ Ampere, Turing)
79
+ ```
80
+
81
+ ### 1. Triton JIT with Vectorized PyTorch Fallback
82
+ * **Triton Execution Path:** Maximum throughput on NVIDIA and AMD ROCm GPUs using fused FP16/BF16 kernels (`triton_adjoint_paged_attn.py` and `triton_sparse_stitch.py`).
83
+ * **PyTorch Native Fallback Path:** Pure PyTorch tensor implementation (`pytorch_fallbacks.py`) supporting Apple Silicon (Metal/MPS), CPU-only servers, and quantized pools with zero external kernel compilation dependencies.
84
+ * **Unified Dispatcher:** Transparently selects the fastest available backend at runtime via `dag_kv.kernels.dispatch`.
85
+
86
+ ### 2. Dynamic Auto-Sharding Topologies
87
+ * **Single-Device Mode:** Runs prefill and decode on a single accelerator (`cuda:0`, `mps`) or CPU.
88
+ * **Multi-GPU Disaggregated Pipeline:** Pins asynchronous canonical chunk prefill to one GPU (e.g. `cuda:1`) while serving stitched multi-file decode on another (e.g. `cuda:0`), synchronizing via page-locked (pinned) Host RAM or P2P DMA.
89
+ * **CPU-Offloaded KV Pooling:** Keeps model weights on high-speed accelerator HBM while offloading massive shared canonical KV blocks to system RAM with pinned memory staging.
90
+
91
+ ### 3. Adaptive Quantization & Memory Compression
92
+ * **Runtime Quantization Selector:** Toggle on the fly between native floating point (`FP16`/`BF16`/`FP32`), **INT8 symmetric quantization** (cutting KV VRAM by 50%), and **INT4 packed quantization** (cutting KV VRAM by 75%).
93
+ * **Configurable Block Sizes:** Dynamic block sizes (`BLOCK_SIZE = 8, 16, 32, 64`) to balance fragmentation vs. memory efficiency for small contexts or massive codebases.
94
+
95
+ ---
96
+
97
+ ## 3 Core Mathematical Pillars
98
+
99
+ ### Pillar 1: Query-Side Adjoint RoPE in $\mathrm{SO}(2)$
100
+
101
+ Let document chunk $c$ be prefilled once at local positions $p \in [0, L_c - 1]$ yielding canonical key vectors $\tilde{\mathbf{k}}_p = \mathcal{R}(p)\mathbf{k}_p$. When chunk $c$ is dynamically placed at global offset $\Delta_c$ in a prompt DAG, its tokens sit at global positions $m = p + \Delta_c$.
102
+
103
+ Because 2D RoPE rotation matrices form a one-parameter subgroup of $SO(2)$:
104
+ $$\mathcal{R}(p + \Delta_c) = \mathcal{R}(\Delta_c)\mathcal{R}(p) \quad \text{and} \quad \mathcal{R}(\Delta_c)^T = \mathcal{R}(-\Delta_c)$$
105
+
106
+ The attention score between query $\mathbf{q}_n$ at global position $n$ and cached key $\tilde{\mathbf{k}}_p$ satisfies the exact adjoint identity:
107
+ $$\text{Score}(n, p; \Delta_c) = \big\langle \mathcal{R}(n)\mathbf{q}_n, \, \mathcal{R}(p + \Delta_c)\mathbf{k}_p \big\rangle = \big\langle \mathcal{R}(-\Delta_c)\tilde{\mathbf{q}}_n, \, \tilde{\mathbf{k}}_p \big\rangle$$
108
+
109
+ ```text
110
+ Standard System (Mutates Keys in HBM - Destroys Sharing):
111
+ HBM Memory: [ R(0)k0, R(1)k1, ... ] ---> In-place mutate to [ R(1024)k0, R(1025)k1, ... ] (Corrupts concurrent batch!)
112
+
113
+ DAG-KV (Query-Side Adjoint - 100% Zero-Copy HBM):
114
+ Query in SRAM: q_adj = q * cos(Δ) - rotate_half(q) * sin(Δ)
115
+ HBM Memory: [ R(0)k0, R(1)k1, ... ] (Never modified, 100% read-only shared pool!)
116
+ ```
117
+
118
+ ### Pillar 2: Canonical Sink Prefix (Solving Norm Pathology)
119
+
120
+ Autoregressive language models concentrate massive attention softmax mass ("attention sinks") onto the first 1–4 tokens. If a chunk is prefilled at raw token index 0, its first token absorbs abnormal sink activation norms that corrupt cross-attention representations when shifted to position $\Delta_c > 0$.
121
+
122
+ **Mechanism:** Every chunk is prefilled with a standardized 4-token system sink prefix (`<|im_start|>system\n`) at indices `[0, 1, 2, 3]`, placing the chunk's payload at canonical positions `[4, ..., L_c + 3]`. After prefill, the 4 sink KV slots are sliced away, storing only the payload KV alongside its local causal Log-Sum-Exp vector $\ell_i^{\text{local}}$.
123
+
124
+ ### Pillar 3: Layer-1 LSE Perturbation Bound & Submodular Bridge Selection
125
+
126
+ When concatenating reordered chunks `[Chunk_A, Chunk_B]`, tokens in `Chunk_B` missed attending to `Chunk_A`. Via Online Softmax decomposition:
127
+ $$\alpha_i = \frac{\exp(\ell_{i \to \text{prior}})}{\exp(\ell_i^{\text{local}}) + \exp(\ell_{i \to \text{prior}})} = \sigma\big(\ell_{i \to \text{prior}} - \ell_i^{\text{local}}\big)$$
128
+
129
+ By the Cauchy-Schwarz bound on attention outputs:
130
+ $$\|\mathbf{o}_i^{\text{exact}} - \mathbf{o}_i^{\text{cached}}\|_2 \le 2\alpha_i \max_j \|\mathbf{v}_j\|_2$$
131
+
132
+ Rather than greedy top-$k$ selection, DAG-KV maximizes a spatial quality-diversity objective via 1D neighborhood non-maximum suppression ($r=2$), selecting **4.0% bridge tokens on Stage 0** (`cuda:0`, Layers 0–13) and pruning to **1.5% before crossing PCIe to Stage 1** (`cuda:1`, Layers 14–27). Repaired KVs are written into request-private **Copy-on-Write (CoW) Shadow Blocks**, achieving **4.66x faster stitching** while reusing **96% of the canonical KV cache untouched**.
133
+
134
+ ---
135
+
136
+ ## Ecosystem Integrations (Hugging Face & vLLM)
137
+
138
+ ### 1. Hugging Face Drop-In Wrapper (`patch_hf_model`)
139
+ Use DAG-KV with standard Hugging Face causal language models (Qwen, LLaMA, Mistral) in 3 lines of code:
140
+
141
+ ```python
142
+ from transformers import AutoModelForCausalLM, AutoTokenizer
143
+ from dag_kv import patch_hf_model
144
+
145
+ model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
146
+ tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
147
+
148
+ # Wrap model with DAG-KV position-decoupled caching (auto-detects hardware)
149
+ dag_model = patch_hf_model(model, tokenizer, quantization="none", block_size=16)
150
+
151
+ # Prefill documents once in isolation
152
+ chunk_a = dag_model.prefill_chunk("File A: Authentication service implementation...")
153
+ chunk_b = dag_model.prefill_chunk("File B: Database schema definitions...")
154
+
155
+ # Query in any permutation with 100% KV reuse: [Chunk_B, Chunk_A]
156
+ response = dag_model.generate([chunk_b, chunk_a], prompt_suffix="Question: What DB is used?")
157
+ print(response)
158
+ ```
159
+
160
+ ### 2. vLLM BlockManager Plugin (`DAGKVvLLMBlockManagerPlugin`)
161
+ Integrate DAG-KV's Content-SHA256 registry and offset block table into vLLM's continuous batching engine:
162
+
163
+ ```python
164
+ from dag_kv.integrations.vllm import DAGKVvLLMBlockManagerPlugin
165
+
166
+ plugin = DAGKVvLLMBlockManagerPlugin(block_size=16)
167
+
168
+ # Hash and register pre-computed physical blocks
169
+ hash_a = plugin.hash_tokens(chunk_a_tokens)
170
+ plugin.register_chunk(hash_a, physical_block_ids=[12, 13, 14])
171
+
172
+ # Dynamically assemble sequence mappings with RoPE delta offsets
173
+ seq_blocks = plugin.assemble_sequence_mapping(
174
+ seq_id="request_101",
175
+ chunk_hashes_and_lengths=[(hash_b, 128), (hash_a, 256)],
176
+ )
177
+ ```
178
+
179
+ ---
180
+
181
+ ## CLI & Production Serving
182
+
183
+ Install the CLI directly via pip:
184
+ ```bash
185
+ pip install -e .
186
+ ```
187
+
188
+ ### Inspect Discovered Hardware
189
+ ```bash
190
+ dag-kv info
191
+ ```
192
+ Output:
193
+ ```text
194
+ ============================================================
195
+ DAG-KV Hardware Abstraction Layer (HAL) Report
196
+ ============================================================
197
+ CUDA Available: True
198
+ CUDA Device Count: 2
199
+ Apple Silicon MPS: False
200
+ Triton JIT Available: True
201
+ Primary Compute Dev: cuda:0
202
+ ------------------------------------------------------------
203
+ Discovered Devices:
204
+ [0] CUDA | NVIDIA GeForce RTX 2080 Ti | Total: 10.75 GB | Free: 10.30 GB | CC: sm_75
205
+ [1] CUDA | NVIDIA GeForce RTX 2080 Ti | Total: 10.75 GB | Free: 10.59 GB | CC: sm_75
206
+ [2] CPU | Host CPU (48 cores, x86_64) | Total: 187.54 GB | Free: 93.77 GB | CC: N/A
207
+ ------------------------------------------------------------
208
+ Recommended Topology:
209
+ Mode: pipeline
210
+ Compute Dev: cuda:0
211
+ KV Device: cuda:0
212
+ Dtype: torch.float16
213
+ Block Size: 16
214
+ ============================================================
215
+ ```
216
+
217
+ ### Launch OpenAI-Compatible HTTP Server
218
+ ```bash
219
+ dag-kv serve --host 0.0.0.0 --port 8000 --model Qwen/Qwen2.5-7B-Instruct --device auto
220
+ ```
221
+
222
+ Query the endpoint:
223
+ ```bash
224
+ curl http://localhost:8000/v1/chat/completions \
225
+ -H "Content-Type: application/json" \
226
+ -d '{
227
+ "model": "Qwen/Qwen2.5-7B-Instruct",
228
+ "messages": [
229
+ {"role": "user", "content": "### FILE: auth.py\n...\n### FILE: db.py\n...\nQuestion: What DB is used?"}
230
+ ],
231
+ "max_tokens": 16
232
+ }'
233
+ ```
234
+
235
+ ---
236
+
237
+ ## Empirical Benchmark Results
238
+
239
+ ![DAG-KV Architecture and Empirical Benchmarks](assets/dag_kv_benchmarks.png)
240
+
241
+ ### Summary of Performance Gains on $2\times$ NVIDIA RTX 2080 Ti (`Qwen2.5-7B-Instruct`)
242
+
243
+ | Dimension | Standard Serving Baseline (vLLM / SGLang) | DAG-KV Stitched Engine (Ours) | Measured Gain |
244
+ | :--- | :--- | :--- | :---: |
245
+ | **Context Stitch Latency (4K Toks)** | 1,441.8 ms (Dense 7B Prefill) | **309.5 ms** (Submodular $4\% \to 1.5\%$) | **4.66x Faster** |
246
+ | **Multi-User KV VRAM (12 Sessions)** | 2,522.1 MB (Private copies) | **258.2 MB** (Shared Pool + INT8) | **9.8x Less VRAM** |
247
+ | **Decode Throughput ($B=4\times 2$)** | 23.1 tok/s (Sequential $B=1$) | **100.4 tok/s** (Ping-Pong Overlap) | **4.35x Throughput** |
248
+ | **Live IDE Save Re-Indexing** | 994.4 ms (Full re-prefill) | **79.3 ms** (AST Function-Delta) | **12.5x Faster** |
249
+ | **Reordered File Cache Hit Rate** | **0.0%** (Prefix cache miss) | **100.0%** (Zero-copy canonical hit) | **$\infty$** |
250
+ | **Codebase QA Accuracy** | N/A | **100.0% (12/12 Exact Answers)** | **1.000 F1** |
251
+
252
+ ---
253
+
254
+ ## Repository Architecture
255
+
256
+ ```text
257
+ dag-kv/
258
+ ├── dag_kv/
259
+ │ ├── core/
260
+ │ │ ├── rope_adjoint.py # Adjoint RoPE R(-Delta) implementation and mathematical proofs
261
+ │ │ ├── sink_canonicalizer.py # 4-token sink prefix canonicalization & SHA256 hashing
262
+ │ │ ├── lse_probe.py # Layer-1 LSE perturbation bound alpha_i & top-k% selector
263
+ │ │ ├── block_manager.py # Paged Block Pool, OffsetBlockTable & CoW Shadow Blocks
264
+ │ │ └── quantizer.py # Vectorized symmetric INT8 / INT4 KV cache quantizer
265
+ │ ├── devices/
266
+ │ │ ├── shard_config.py # Pluggable topologies: Single-GPU, Pipeline, MPS, CPU Offload
267
+ │ │ └── device_manager.py # Hardware Abstraction Layer auto-discovery & VRAM probing
268
+ │ ├── kernels/
269
+ │ │ ├── dispatch.py # Dynamic dispatch between Triton and PyTorch fallbacks
270
+ │ │ ├── pytorch_fallbacks.py # Universal pure PyTorch kernels for Apple Silicon (MPS) & CPU
271
+ │ │ ├── triton_adjoint_paged_attn.py # Turing SM75-tuned PagedAttention decode with R(-Delta_b)q
272
+ │ │ └── triton_sparse_stitch.py # Fused ragged micro-prefill kernel for Bridge Tokens
273
+ │ ├── integrations/
274
+ │ │ ├── huggingface.py # Drop-in DAGKVModelWrapper & patch_hf_model adapter
275
+ │ │ └── vllm.py # DAGKVvLLMBlockManagerPlugin for vLLM continuous batching
276
+ │ ├── models/
277
+ │ │ └── qwen2_dag_patch.py # End-to-end Qwen2 inference engine with universal dispatch
278
+ │ ├── worker/
279
+ │ │ ├── universal_pipeline.py # Universal single/dual device & CPU-offloaded pipeline
280
+ │ │ └── dual_gpu_pipeline.py # Dedicated Dual-GPU async prefill-decode pipeline
281
+ │ ├── cli.py # Command line interface (dag-kv info, serve)
282
+ │ └── server.py # OpenAI-compatible /v1/chat/completions HTTP server
283
+ ├── configs/
284
+ │ └── hw_rtx2080ti.yaml # Turing SM75 tile configs, memory constraints, dual-GPU pipeline
285
+ ├── paper/
286
+ │ ├── dag_kv_paper.pdf # Compiled 3-page research paper PDF (ready for publication)
287
+ │ └── main.tex # LaTeX manuscript source
288
+ ├── assets/
289
+ │ └── dag_kv_benchmarks.png # High-resolution benchmark overview chart
290
+ ├── .github/workflows/
291
+ │ └── ci.yml # GitHub Actions Multi-Platform CI workflow
292
+ ├── Dockerfile # Multi-Arch Production Dockerfile (linux/amd64, linux/arm64)
293
+ ├── tests/ # 43/43 Passing Tests (CPU fallbacks, GPU Triton, HAL, Quant)
294
+ └── benchmarks/ # Production evaluation & benchmark suite
295
+ ```
296
+
297
+ ---
298
+
299
+ ## Test Suite (43/43 Tests Passing)
300
+
301
+ Run the entire test suite across hardware backends:
302
+ ```bash
303
+ pytest -v tests/
304
+ ```
305
+ Output:
306
+ ```text
307
+ tests/test_core_components.py::test_sink_canonicalizer PASSED [ 2%]
308
+ tests/test_core_components.py::test_lse_probe_and_bridge_selection PASSED [ 4%]
309
+ tests/test_core_components.py::test_block_manager_cow_and_assembly PASSED [ 6%]
310
+ tests/test_device_manager.py::test_device_manager_singleton_and_probing PASSED [ 9%]
311
+ tests/test_device_manager.py::test_shard_config_presets PASSED [ 11%]
312
+ tests/test_device_manager.py::test_auto_detect_topology PASSED [ 13%]
313
+ tests/test_dual_gpu_pipeline.py::test_dual_gpu_disaggregated_prefill_and_stitch PASSED [ 16%]
314
+ tests/test_dynamic_blocks.py::test_dynamic_block_size_allocation[8] PASSED [ 18%]
315
+ tests/test_dynamic_blocks.py::test_dynamic_block_size_allocation[16] PASSED [ 20%]
316
+ tests/test_dynamic_blocks.py::test_dynamic_block_size_allocation[32] PASSED [ 23%]
317
+ tests/test_dynamic_blocks.py::test_attention_with_variable_block_sizes[8] PASSED [ 25%]
318
+ tests/test_dynamic_blocks.py::test_attention_with_variable_block_sizes[32] PASSED [ 27%]
319
+ tests/test_hf_integration.py::test_vllm_block_manager_plugin PASSED [ 30%]
320
+ tests/test_hf_integration.py::test_hf_wrapper_cpu_offline PASSED [ 32%]
321
+ tests/test_pytorch_fallback.py::test_pytorch_adjoint_paged_attention_cpu PASSED [ 34%]
322
+ tests/test_pytorch_fallback.py::test_pytorch_vs_triton_adjoint_paged_attention_parity PASSED [ 37%]
323
+ tests/test_pytorch_fallback.py::test_pytorch_vs_triton_sparse_bridge_parity PASSED [ 39%]
324
+ tests/test_quantization.py::test_kv_quantizer_int8_accuracy PASSED [ 41%]
325
+ tests/test_quantization.py::test_kv_quantizer_int4_accuracy PASSED [ 44%]
326
+ tests/test_quantization.py::test_block_manager_int8_quantization_lifecycle PASSED [ 46%]
327
+ tests/test_rope_adjoint.py::test_rope_adjoint_identity_scalar ... PASSED [ 65%]
328
+ tests/test_rope_adjoint.py::test_rope_adjoint_batched_multihead ... PASSED [ 74%]
329
+ tests/test_rope_adjoint.py::test_rotate_half_orthogonality PASSED [ 76%]
330
+ tests/test_sparse_bridge_e2e.py::test_qwen2_dag_kv_e2e_stitching_and_repair PASSED [ 79%]
331
+ tests/test_triton_sm75.py::test_triton_adjoint_paged_attention_accuracy ... PASSED [ 93%]
332
+ tests/test_triton_sm75.py::test_triton_sparse_bridge_attention_accuracy ... PASSED [ 97%]
333
+ tests/test_triton_sm75.py::test_turing_sm75_shared_memory_limits PASSED [100%]
334
+
335
+ ============================= 43 passed in 45.73s ==============================
336
+ ```
337
+
338
+ ---
339
+
340
+ ## Citation & License
341
+
342
+ ```bibtex
343
+ @software{dag_kv_2026,
344
+ title={{DAG-KV: Universal Position-Decoupled Graph KV Cache with Query-Side Adjoint RoPE & LSE-Bounded Sparse Bridge Repair}},
345
+ author={Nandaniya, Kaushal and Collaborators},
346
+ year={2026},
347
+ url={https://github.com/kaushalnandaniya/dag-kv}
348
+ }
349
+ ```
350
+
351
+ Licensed under the [Apache 2.0 License](LICENSE).