dag-kv 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dag_kv-0.2.0/LICENSE +17 -0
- dag_kv-0.2.0/PKG-INFO +351 -0
- dag_kv-0.2.0/README.md +329 -0
- dag_kv-0.2.0/dag_kv/__init__.py +49 -0
- dag_kv-0.2.0/dag_kv/cli.py +204 -0
- dag_kv-0.2.0/dag_kv/core/__init__.py +45 -0
- dag_kv-0.2.0/dag_kv/core/block_manager.py +312 -0
- dag_kv-0.2.0/dag_kv/core/lse_probe.py +147 -0
- dag_kv-0.2.0/dag_kv/core/quantizer.py +89 -0
- dag_kv-0.2.0/dag_kv/core/rope_adjoint.py +159 -0
- dag_kv-0.2.0/dag_kv/core/sink_canonicalizer.py +156 -0
- dag_kv-0.2.0/dag_kv/devices/__init__.py +13 -0
- dag_kv-0.2.0/dag_kv/devices/device_manager.py +268 -0
- dag_kv-0.2.0/dag_kv/devices/shard_config.py +199 -0
- dag_kv-0.2.0/dag_kv/integrations/__init__.py +11 -0
- dag_kv-0.2.0/dag_kv/integrations/huggingface.py +145 -0
- dag_kv-0.2.0/dag_kv/integrations/vllm.py +90 -0
- dag_kv-0.2.0/dag_kv/kernels/__init__.py +35 -0
- dag_kv-0.2.0/dag_kv/kernels/dispatch.py +152 -0
- dag_kv-0.2.0/dag_kv/kernels/pytorch_fallbacks.py +282 -0
- dag_kv-0.2.0/dag_kv/kernels/triton_adjoint_paged_attn.py +268 -0
- dag_kv-0.2.0/dag_kv/kernels/triton_sparse_stitch.py +259 -0
- dag_kv-0.2.0/dag_kv/models/__init__.py +5 -0
- dag_kv-0.2.0/dag_kv/models/qwen2_dag_patch.py +571 -0
- dag_kv-0.2.0/dag_kv/server.py +175 -0
- dag_kv-0.2.0/dag_kv/worker/__init__.py +10 -0
- dag_kv-0.2.0/dag_kv/worker/dual_gpu_pipeline.py +287 -0
- dag_kv-0.2.0/dag_kv/worker/universal_pipeline.py +232 -0
- dag_kv-0.2.0/dag_kv.egg-info/PKG-INFO +351 -0
- dag_kv-0.2.0/dag_kv.egg-info/SOURCES.txt +44 -0
- dag_kv-0.2.0/dag_kv.egg-info/dependency_links.txt +1 -0
- dag_kv-0.2.0/dag_kv.egg-info/entry_points.txt +2 -0
- dag_kv-0.2.0/dag_kv.egg-info/requires.txt +15 -0
- dag_kv-0.2.0/dag_kv.egg-info/top_level.txt +1 -0
- dag_kv-0.2.0/pyproject.toml +40 -0
- dag_kv-0.2.0/setup.cfg +4 -0
- dag_kv-0.2.0/tests/test_core_components.py +108 -0
- dag_kv-0.2.0/tests/test_device_manager.py +62 -0
- dag_kv-0.2.0/tests/test_dual_gpu_pipeline.py +30 -0
- dag_kv-0.2.0/tests/test_dynamic_blocks.py +80 -0
- dag_kv-0.2.0/tests/test_hf_integration.py +90 -0
- dag_kv-0.2.0/tests/test_pytorch_fallback.py +155 -0
- dag_kv-0.2.0/tests/test_quantization.py +94 -0
- dag_kv-0.2.0/tests/test_rope_adjoint.py +91 -0
- dag_kv-0.2.0/tests/test_sparse_bridge_e2e.py +97 -0
- dag_kv-0.2.0/tests/test_triton_sm75.py +128 -0
dag_kv-0.2.0/LICENSE
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
Apache License
|
|
2
|
+
Version 2.0, January 2004
|
|
3
|
+
http://www.apache.org/licenses/
|
|
4
|
+
|
|
5
|
+
Copyright 2026 Kaushal Nandaniya
|
|
6
|
+
|
|
7
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
8
|
+
you may not use this file except in compliance with the License.
|
|
9
|
+
You may obtain a copy of the License at
|
|
10
|
+
|
|
11
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
12
|
+
|
|
13
|
+
Unless required by applicable law or agreed to in writing, software
|
|
14
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
15
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
16
|
+
See the License for the specific language governing permissions and
|
|
17
|
+
limitations under the License.
|
dag_kv-0.2.0/PKG-INFO
ADDED
|
@@ -0,0 +1,351 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: dag-kv
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Universal Position-Decoupled Graph KV Cache with Query-Side Adjoint RoPE & LSE-Bounded Sparse Bridge Repair
|
|
5
|
+
Author: Kaushal Nandaniya
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Requires-Python: >=3.10
|
|
8
|
+
Description-Content-Type: text/markdown
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Requires-Dist: torch>=2.2.0
|
|
11
|
+
Requires-Dist: transformers>=4.44.0
|
|
12
|
+
Requires-Dist: accelerate>=0.33.0
|
|
13
|
+
Requires-Dist: numpy>=1.26.0
|
|
14
|
+
Requires-Dist: pyyaml>=5.3.0
|
|
15
|
+
Requires-Dist: triton>=3.0.0; sys_platform == "linux"
|
|
16
|
+
Provides-Extra: dev
|
|
17
|
+
Requires-Dist: pytest>=8.0.0; extra == "dev"
|
|
18
|
+
Requires-Dist: pytest-cov; extra == "dev"
|
|
19
|
+
Provides-Extra: vllm
|
|
20
|
+
Requires-Dist: vllm>=0.6.0; extra == "vllm"
|
|
21
|
+
Dynamic: license-file
|
|
22
|
+
|
|
23
|
+
# DAG-KV: Universal Position-Decoupled Graph KV Cache with Query-Side Adjoint RoPE & LSE-Bounded Sparse Bridge Repair
|
|
24
|
+
|
|
25
|
+
[](https://pytorch.org)
|
|
26
|
+
[](dag_kv/kernels/)
|
|
27
|
+
[](tests/)
|
|
28
|
+
[](paper/dag_kv_paper.pdf)
|
|
29
|
+
[](.github/workflows/ci.yml)
|
|
30
|
+
[-2496ED?logo=docker)](Dockerfile)
|
|
31
|
+
[](LICENSE)
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## Executive Summary
|
|
36
|
+
|
|
37
|
+
Existing prefix-caching architectures (e.g., **vLLM Automatic Prefix Caching**, **SGLang RadixAttention**) suffer a catastrophic **0% cache hit rate** when context chunks are dynamically permuted, reordered, or multi-hop composed in Retrieval-Augmented Generation (RAG) and multi-engineer codebase assistant workflows (e.g., `[Sys, Doc_A, Doc_B]` vs `[Sys, Doc_B, Doc_A]`). This failure stems from two fundamental blockers:
|
|
38
|
+
1. **RoPE Position Coupling:** Rotary Position Embeddings bake absolute sequence indices directly into cached Key vectors. Mutating cached keys in HBM to shift position offsets destroys Copy-on-Write (CoW) sharing across concurrent requests.
|
|
39
|
+
2. **Missing Cross-Chunk Causal Attention:** Chunks prefilled in isolation miss attention interactions with predecessor chunks.
|
|
40
|
+
|
|
41
|
+
**DAG-KV** turns graph-structured context caching into a **universal, production-ready library** across all hardware architectures (NVIDIA, AMD ROCm, Apple Silicon MPS, and Host CPU):
|
|
42
|
+
- **Query-Side Adjoint RoPE:** Exploits the orthogonal adjoint identity in $SO(2)$ to counter-rotate query vectors in GPU/CPU registers ($q_{\text{adj}} = \mathcal{R}(-\Delta) \tilde{q}$), leaving cached Key blocks **100% read-only and zero-copy in memory**.
|
|
43
|
+
- **Canonical Sink Prefix:** Neutralizes the position-zero attention sink pathology by prepending a 4-token system sink during isolated prefill and slicing it off before registration.
|
|
44
|
+
- **LSE Perturbation Bound & Submodular Bridge Repair:** Uses Layer-1 Online Softmax perturbation scalars $\alpha_i = \sigma(\ell_{i \to \text{prior}} - \ell_i^{\text{local}})$ with quality-diversity non-maximum suppression to select sparse Bridge Tokens for ragged micro-prefill, storing repaired keys into request-private **Copy-on-Write (CoW) Shadow Blocks** while reusing **92%–96% of the canonical KV cache untouched**.
|
|
45
|
+
- **Hardware Abstraction Layer (HAL):** Dynamic device manager with auto-discovery, topology auto-configuration (Single-GPU, Multi-GPU Pipeline, Apple Silicon MPS, CPU, and CPU-offloaded KV pooling), adaptive INT8/INT4 quantization, and seamless drop-in integrations for **Hugging Face** and **vLLM**.
|
|
46
|
+
|
|
47
|
+
> **Research Paper:** Read the complete 3-page manuscript in [`paper/dag_kv_paper.pdf`](paper/dag_kv_paper.pdf).
|
|
48
|
+
|
|
49
|
+
---
|
|
50
|
+
|
|
51
|
+
## Universal Hardware Abstraction & Multi-Backend Architecture
|
|
52
|
+
|
|
53
|
+
DAG-KV abstracts away low-level GPU dependencies, providing automated backend routing based on the host environment:
|
|
54
|
+
|
|
55
|
+
```
|
|
56
|
+
┌──────────────────────────────────┐
|
|
57
|
+
│ User API / HTTP Endpoint / │
|
|
58
|
+
│ HuggingFace / vLLM Integrations│
|
|
59
|
+
└─────────────────┬────────────────┘
|
|
60
|
+
│
|
|
61
|
+
▼
|
|
62
|
+
┌──────────────────────────────────┐
|
|
63
|
+
│ Hardware Abstraction Layer │
|
|
64
|
+
│ DeviceManager & ShardConfig │
|
|
65
|
+
└─────────────────┬────────────────┘
|
|
66
|
+
│
|
|
67
|
+
┌───────────────────────────────┴───────────────────────────────┐
|
|
68
|
+
▼ ▼
|
|
69
|
+
┌───────────────────────┐ ┌───────────────────────┐
|
|
70
|
+
│ High-Performance │ │ Universal Native │
|
|
71
|
+
│ Triton JIT Path │ │ PyTorch Fallback Path │
|
|
72
|
+
└───────────┬───────────┘ └───────────┬───────────┘
|
|
73
|
+
│ │
|
|
74
|
+
┌────────┴────────┐ ┌────────┴────────┐
|
|
75
|
+
▼ ▼ ▼ ▼
|
|
76
|
+
NVIDIA GPUs AMD ROCm Apple Silicon Host CPU
|
|
77
|
+
(Hopper, Ada, (MI200, MI300) (M1/M2/M3/M4 MPS) (x86_64, ARM64)
|
|
78
|
+
Ampere, Turing)
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### 1. Triton JIT with Vectorized PyTorch Fallback
|
|
82
|
+
* **Triton Execution Path:** Maximum throughput on NVIDIA and AMD ROCm GPUs using fused FP16/BF16 kernels (`triton_adjoint_paged_attn.py` and `triton_sparse_stitch.py`).
|
|
83
|
+
* **PyTorch Native Fallback Path:** Pure PyTorch tensor implementation (`pytorch_fallbacks.py`) supporting Apple Silicon (Metal/MPS), CPU-only servers, and quantized pools with zero external kernel compilation dependencies.
|
|
84
|
+
* **Unified Dispatcher:** Transparently selects the fastest available backend at runtime via `dag_kv.kernels.dispatch`.
|
|
85
|
+
|
|
86
|
+
### 2. Dynamic Auto-Sharding Topologies
|
|
87
|
+
* **Single-Device Mode:** Runs prefill and decode on a single accelerator (`cuda:0`, `mps`) or CPU.
|
|
88
|
+
* **Multi-GPU Disaggregated Pipeline:** Pins asynchronous canonical chunk prefill to one GPU (e.g. `cuda:1`) while serving stitched multi-file decode on another (e.g. `cuda:0`), synchronizing via page-locked (pinned) Host RAM or P2P DMA.
|
|
89
|
+
* **CPU-Offloaded KV Pooling:** Keeps model weights on high-speed accelerator HBM while offloading massive shared canonical KV blocks to system RAM with pinned memory staging.
|
|
90
|
+
|
|
91
|
+
### 3. Adaptive Quantization & Memory Compression
|
|
92
|
+
* **Runtime Quantization Selector:** Toggle on the fly between native floating point (`FP16`/`BF16`/`FP32`), **INT8 symmetric quantization** (cutting KV VRAM by 50%), and **INT4 packed quantization** (cutting KV VRAM by 75%).
|
|
93
|
+
* **Configurable Block Sizes:** Dynamic block sizes (`BLOCK_SIZE = 8, 16, 32, 64`) to balance fragmentation vs. memory efficiency for small contexts or massive codebases.
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
97
|
+
## 3 Core Mathematical Pillars
|
|
98
|
+
|
|
99
|
+
### Pillar 1: Query-Side Adjoint RoPE in $\mathrm{SO}(2)$
|
|
100
|
+
|
|
101
|
+
Let document chunk $c$ be prefilled once at local positions $p \in [0, L_c - 1]$ yielding canonical key vectors $\tilde{\mathbf{k}}_p = \mathcal{R}(p)\mathbf{k}_p$. When chunk $c$ is dynamically placed at global offset $\Delta_c$ in a prompt DAG, its tokens sit at global positions $m = p + \Delta_c$.
|
|
102
|
+
|
|
103
|
+
Because 2D RoPE rotation matrices form a one-parameter subgroup of $SO(2)$:
|
|
104
|
+
$$\mathcal{R}(p + \Delta_c) = \mathcal{R}(\Delta_c)\mathcal{R}(p) \quad \text{and} \quad \mathcal{R}(\Delta_c)^T = \mathcal{R}(-\Delta_c)$$
|
|
105
|
+
|
|
106
|
+
The attention score between query $\mathbf{q}_n$ at global position $n$ and cached key $\tilde{\mathbf{k}}_p$ satisfies the exact adjoint identity:
|
|
107
|
+
$$\text{Score}(n, p; \Delta_c) = \big\langle \mathcal{R}(n)\mathbf{q}_n, \, \mathcal{R}(p + \Delta_c)\mathbf{k}_p \big\rangle = \big\langle \mathcal{R}(-\Delta_c)\tilde{\mathbf{q}}_n, \, \tilde{\mathbf{k}}_p \big\rangle$$
|
|
108
|
+
|
|
109
|
+
```text
|
|
110
|
+
Standard System (Mutates Keys in HBM - Destroys Sharing):
|
|
111
|
+
HBM Memory: [ R(0)k0, R(1)k1, ... ] ---> In-place mutate to [ R(1024)k0, R(1025)k1, ... ] (Corrupts concurrent batch!)
|
|
112
|
+
|
|
113
|
+
DAG-KV (Query-Side Adjoint - 100% Zero-Copy HBM):
|
|
114
|
+
Query in SRAM: q_adj = q * cos(Δ) - rotate_half(q) * sin(Δ)
|
|
115
|
+
HBM Memory: [ R(0)k0, R(1)k1, ... ] (Never modified, 100% read-only shared pool!)
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
### Pillar 2: Canonical Sink Prefix (Solving Norm Pathology)
|
|
119
|
+
|
|
120
|
+
Autoregressive language models concentrate massive attention softmax mass ("attention sinks") onto the first 1–4 tokens. If a chunk is prefilled at raw token index 0, its first token absorbs abnormal sink activation norms that corrupt cross-attention representations when shifted to position $\Delta_c > 0$.
|
|
121
|
+
|
|
122
|
+
**Mechanism:** Every chunk is prefilled with a standardized 4-token system sink prefix (`<|im_start|>system\n`) at indices `[0, 1, 2, 3]`, placing the chunk's payload at canonical positions `[4, ..., L_c + 3]`. After prefill, the 4 sink KV slots are sliced away, storing only the payload KV alongside its local causal Log-Sum-Exp vector $\ell_i^{\text{local}}$.
|
|
123
|
+
|
|
124
|
+
### Pillar 3: Layer-1 LSE Perturbation Bound & Submodular Bridge Selection
|
|
125
|
+
|
|
126
|
+
When concatenating reordered chunks `[Chunk_A, Chunk_B]`, tokens in `Chunk_B` missed attending to `Chunk_A`. Via Online Softmax decomposition:
|
|
127
|
+
$$\alpha_i = \frac{\exp(\ell_{i \to \text{prior}})}{\exp(\ell_i^{\text{local}}) + \exp(\ell_{i \to \text{prior}})} = \sigma\big(\ell_{i \to \text{prior}} - \ell_i^{\text{local}}\big)$$
|
|
128
|
+
|
|
129
|
+
By the Cauchy-Schwarz bound on attention outputs:
|
|
130
|
+
$$\|\mathbf{o}_i^{\text{exact}} - \mathbf{o}_i^{\text{cached}}\|_2 \le 2\alpha_i \max_j \|\mathbf{v}_j\|_2$$
|
|
131
|
+
|
|
132
|
+
Rather than greedy top-$k$ selection, DAG-KV maximizes a spatial quality-diversity objective via 1D neighborhood non-maximum suppression ($r=2$), selecting **4.0% bridge tokens on Stage 0** (`cuda:0`, Layers 0–13) and pruning to **1.5% before crossing PCIe to Stage 1** (`cuda:1`, Layers 14–27). Repaired KVs are written into request-private **Copy-on-Write (CoW) Shadow Blocks**, achieving **4.66x faster stitching** while reusing **96% of the canonical KV cache untouched**.
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
136
|
+
## Ecosystem Integrations (Hugging Face & vLLM)
|
|
137
|
+
|
|
138
|
+
### 1. Hugging Face Drop-In Wrapper (`patch_hf_model`)
|
|
139
|
+
Use DAG-KV with standard Hugging Face causal language models (Qwen, LLaMA, Mistral) in 3 lines of code:
|
|
140
|
+
|
|
141
|
+
```python
|
|
142
|
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
143
|
+
from dag_kv import patch_hf_model
|
|
144
|
+
|
|
145
|
+
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
|
|
146
|
+
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
|
|
147
|
+
|
|
148
|
+
# Wrap model with DAG-KV position-decoupled caching (auto-detects hardware)
|
|
149
|
+
dag_model = patch_hf_model(model, tokenizer, quantization="none", block_size=16)
|
|
150
|
+
|
|
151
|
+
# Prefill documents once in isolation
|
|
152
|
+
chunk_a = dag_model.prefill_chunk("File A: Authentication service implementation...")
|
|
153
|
+
chunk_b = dag_model.prefill_chunk("File B: Database schema definitions...")
|
|
154
|
+
|
|
155
|
+
# Query in any permutation with 100% KV reuse: [Chunk_B, Chunk_A]
|
|
156
|
+
response = dag_model.generate([chunk_b, chunk_a], prompt_suffix="Question: What DB is used?")
|
|
157
|
+
print(response)
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
### 2. vLLM BlockManager Plugin (`DAGKVvLLMBlockManagerPlugin`)
|
|
161
|
+
Integrate DAG-KV's Content-SHA256 registry and offset block table into vLLM's continuous batching engine:
|
|
162
|
+
|
|
163
|
+
```python
|
|
164
|
+
from dag_kv.integrations.vllm import DAGKVvLLMBlockManagerPlugin
|
|
165
|
+
|
|
166
|
+
plugin = DAGKVvLLMBlockManagerPlugin(block_size=16)
|
|
167
|
+
|
|
168
|
+
# Hash and register pre-computed physical blocks
|
|
169
|
+
hash_a = plugin.hash_tokens(chunk_a_tokens)
|
|
170
|
+
plugin.register_chunk(hash_a, physical_block_ids=[12, 13, 14])
|
|
171
|
+
|
|
172
|
+
# Dynamically assemble sequence mappings with RoPE delta offsets
|
|
173
|
+
seq_blocks = plugin.assemble_sequence_mapping(
|
|
174
|
+
seq_id="request_101",
|
|
175
|
+
chunk_hashes_and_lengths=[(hash_b, 128), (hash_a, 256)],
|
|
176
|
+
)
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
---
|
|
180
|
+
|
|
181
|
+
## CLI & Production Serving
|
|
182
|
+
|
|
183
|
+
Install the CLI directly via pip:
|
|
184
|
+
```bash
|
|
185
|
+
pip install -e .
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
### Inspect Discovered Hardware
|
|
189
|
+
```bash
|
|
190
|
+
dag-kv info
|
|
191
|
+
```
|
|
192
|
+
Output:
|
|
193
|
+
```text
|
|
194
|
+
============================================================
|
|
195
|
+
DAG-KV Hardware Abstraction Layer (HAL) Report
|
|
196
|
+
============================================================
|
|
197
|
+
CUDA Available: True
|
|
198
|
+
CUDA Device Count: 2
|
|
199
|
+
Apple Silicon MPS: False
|
|
200
|
+
Triton JIT Available: True
|
|
201
|
+
Primary Compute Dev: cuda:0
|
|
202
|
+
------------------------------------------------------------
|
|
203
|
+
Discovered Devices:
|
|
204
|
+
[0] CUDA | NVIDIA GeForce RTX 2080 Ti | Total: 10.75 GB | Free: 10.30 GB | CC: sm_75
|
|
205
|
+
[1] CUDA | NVIDIA GeForce RTX 2080 Ti | Total: 10.75 GB | Free: 10.59 GB | CC: sm_75
|
|
206
|
+
[2] CPU | Host CPU (48 cores, x86_64) | Total: 187.54 GB | Free: 93.77 GB | CC: N/A
|
|
207
|
+
------------------------------------------------------------
|
|
208
|
+
Recommended Topology:
|
|
209
|
+
Mode: pipeline
|
|
210
|
+
Compute Dev: cuda:0
|
|
211
|
+
KV Device: cuda:0
|
|
212
|
+
Dtype: torch.float16
|
|
213
|
+
Block Size: 16
|
|
214
|
+
============================================================
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
### Launch OpenAI-Compatible HTTP Server
|
|
218
|
+
```bash
|
|
219
|
+
dag-kv serve --host 0.0.0.0 --port 8000 --model Qwen/Qwen2.5-7B-Instruct --device auto
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
Query the endpoint:
|
|
223
|
+
```bash
|
|
224
|
+
curl http://localhost:8000/v1/chat/completions \
|
|
225
|
+
-H "Content-Type: application/json" \
|
|
226
|
+
-d '{
|
|
227
|
+
"model": "Qwen/Qwen2.5-7B-Instruct",
|
|
228
|
+
"messages": [
|
|
229
|
+
{"role": "user", "content": "### FILE: auth.py\n...\n### FILE: db.py\n...\nQuestion: What DB is used?"}
|
|
230
|
+
],
|
|
231
|
+
"max_tokens": 16
|
|
232
|
+
}'
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
---
|
|
236
|
+
|
|
237
|
+
## Empirical Benchmark Results
|
|
238
|
+
|
|
239
|
+

|
|
240
|
+
|
|
241
|
+
### Summary of Performance Gains on $2\times$ NVIDIA RTX 2080 Ti (`Qwen2.5-7B-Instruct`)
|
|
242
|
+
|
|
243
|
+
| Dimension | Standard Serving Baseline (vLLM / SGLang) | DAG-KV Stitched Engine (Ours) | Measured Gain |
|
|
244
|
+
| :--- | :--- | :--- | :---: |
|
|
245
|
+
| **Context Stitch Latency (4K Toks)** | 1,441.8 ms (Dense 7B Prefill) | **309.5 ms** (Submodular $4\% \to 1.5\%$) | **4.66x Faster** |
|
|
246
|
+
| **Multi-User KV VRAM (12 Sessions)** | 2,522.1 MB (Private copies) | **258.2 MB** (Shared Pool + INT8) | **9.8x Less VRAM** |
|
|
247
|
+
| **Decode Throughput ($B=4\times 2$)** | 23.1 tok/s (Sequential $B=1$) | **100.4 tok/s** (Ping-Pong Overlap) | **4.35x Throughput** |
|
|
248
|
+
| **Live IDE Save Re-Indexing** | 994.4 ms (Full re-prefill) | **79.3 ms** (AST Function-Delta) | **12.5x Faster** |
|
|
249
|
+
| **Reordered File Cache Hit Rate** | **0.0%** (Prefix cache miss) | **100.0%** (Zero-copy canonical hit) | **$\infty$** |
|
|
250
|
+
| **Codebase QA Accuracy** | N/A | **100.0% (12/12 Exact Answers)** | **1.000 F1** |
|
|
251
|
+
|
|
252
|
+
---
|
|
253
|
+
|
|
254
|
+
## Repository Architecture
|
|
255
|
+
|
|
256
|
+
```text
|
|
257
|
+
dag-kv/
|
|
258
|
+
├── dag_kv/
|
|
259
|
+
│ ├── core/
|
|
260
|
+
│ │ ├── rope_adjoint.py # Adjoint RoPE R(-Delta) implementation and mathematical proofs
|
|
261
|
+
│ │ ├── sink_canonicalizer.py # 4-token sink prefix canonicalization & SHA256 hashing
|
|
262
|
+
│ │ ├── lse_probe.py # Layer-1 LSE perturbation bound alpha_i & top-k% selector
|
|
263
|
+
│ │ ├── block_manager.py # Paged Block Pool, OffsetBlockTable & CoW Shadow Blocks
|
|
264
|
+
│ │ └── quantizer.py # Vectorized symmetric INT8 / INT4 KV cache quantizer
|
|
265
|
+
│ ├── devices/
|
|
266
|
+
│ │ ├── shard_config.py # Pluggable topologies: Single-GPU, Pipeline, MPS, CPU Offload
|
|
267
|
+
│ │ └── device_manager.py # Hardware Abstraction Layer auto-discovery & VRAM probing
|
|
268
|
+
│ ├── kernels/
|
|
269
|
+
│ │ ├── dispatch.py # Dynamic dispatch between Triton and PyTorch fallbacks
|
|
270
|
+
│ │ ├── pytorch_fallbacks.py # Universal pure PyTorch kernels for Apple Silicon (MPS) & CPU
|
|
271
|
+
│ │ ├── triton_adjoint_paged_attn.py # Turing SM75-tuned PagedAttention decode with R(-Delta_b)q
|
|
272
|
+
│ │ └── triton_sparse_stitch.py # Fused ragged micro-prefill kernel for Bridge Tokens
|
|
273
|
+
│ ├── integrations/
|
|
274
|
+
│ │ ├── huggingface.py # Drop-in DAGKVModelWrapper & patch_hf_model adapter
|
|
275
|
+
│ │ └── vllm.py # DAGKVvLLMBlockManagerPlugin for vLLM continuous batching
|
|
276
|
+
│ ├── models/
|
|
277
|
+
│ │ └── qwen2_dag_patch.py # End-to-end Qwen2 inference engine with universal dispatch
|
|
278
|
+
│ ├── worker/
|
|
279
|
+
│ │ ├── universal_pipeline.py # Universal single/dual device & CPU-offloaded pipeline
|
|
280
|
+
│ │ └── dual_gpu_pipeline.py # Dedicated Dual-GPU async prefill-decode pipeline
|
|
281
|
+
│ ├── cli.py # Command line interface (dag-kv info, serve)
|
|
282
|
+
│ └── server.py # OpenAI-compatible /v1/chat/completions HTTP server
|
|
283
|
+
├── configs/
|
|
284
|
+
│ └── hw_rtx2080ti.yaml # Turing SM75 tile configs, memory constraints, dual-GPU pipeline
|
|
285
|
+
├── paper/
|
|
286
|
+
│ ├── dag_kv_paper.pdf # Compiled 3-page research paper PDF (ready for publication)
|
|
287
|
+
│ └── main.tex # LaTeX manuscript source
|
|
288
|
+
├── assets/
|
|
289
|
+
│ └── dag_kv_benchmarks.png # High-resolution benchmark overview chart
|
|
290
|
+
├── .github/workflows/
|
|
291
|
+
│ └── ci.yml # GitHub Actions Multi-Platform CI workflow
|
|
292
|
+
├── Dockerfile # Multi-Arch Production Dockerfile (linux/amd64, linux/arm64)
|
|
293
|
+
├── tests/ # 43/43 Passing Tests (CPU fallbacks, GPU Triton, HAL, Quant)
|
|
294
|
+
└── benchmarks/ # Production evaluation & benchmark suite
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
---
|
|
298
|
+
|
|
299
|
+
## Test Suite (43/43 Tests Passing)
|
|
300
|
+
|
|
301
|
+
Run the entire test suite across hardware backends:
|
|
302
|
+
```bash
|
|
303
|
+
pytest -v tests/
|
|
304
|
+
```
|
|
305
|
+
Output:
|
|
306
|
+
```text
|
|
307
|
+
tests/test_core_components.py::test_sink_canonicalizer PASSED [ 2%]
|
|
308
|
+
tests/test_core_components.py::test_lse_probe_and_bridge_selection PASSED [ 4%]
|
|
309
|
+
tests/test_core_components.py::test_block_manager_cow_and_assembly PASSED [ 6%]
|
|
310
|
+
tests/test_device_manager.py::test_device_manager_singleton_and_probing PASSED [ 9%]
|
|
311
|
+
tests/test_device_manager.py::test_shard_config_presets PASSED [ 11%]
|
|
312
|
+
tests/test_device_manager.py::test_auto_detect_topology PASSED [ 13%]
|
|
313
|
+
tests/test_dual_gpu_pipeline.py::test_dual_gpu_disaggregated_prefill_and_stitch PASSED [ 16%]
|
|
314
|
+
tests/test_dynamic_blocks.py::test_dynamic_block_size_allocation[8] PASSED [ 18%]
|
|
315
|
+
tests/test_dynamic_blocks.py::test_dynamic_block_size_allocation[16] PASSED [ 20%]
|
|
316
|
+
tests/test_dynamic_blocks.py::test_dynamic_block_size_allocation[32] PASSED [ 23%]
|
|
317
|
+
tests/test_dynamic_blocks.py::test_attention_with_variable_block_sizes[8] PASSED [ 25%]
|
|
318
|
+
tests/test_dynamic_blocks.py::test_attention_with_variable_block_sizes[32] PASSED [ 27%]
|
|
319
|
+
tests/test_hf_integration.py::test_vllm_block_manager_plugin PASSED [ 30%]
|
|
320
|
+
tests/test_hf_integration.py::test_hf_wrapper_cpu_offline PASSED [ 32%]
|
|
321
|
+
tests/test_pytorch_fallback.py::test_pytorch_adjoint_paged_attention_cpu PASSED [ 34%]
|
|
322
|
+
tests/test_pytorch_fallback.py::test_pytorch_vs_triton_adjoint_paged_attention_parity PASSED [ 37%]
|
|
323
|
+
tests/test_pytorch_fallback.py::test_pytorch_vs_triton_sparse_bridge_parity PASSED [ 39%]
|
|
324
|
+
tests/test_quantization.py::test_kv_quantizer_int8_accuracy PASSED [ 41%]
|
|
325
|
+
tests/test_quantization.py::test_kv_quantizer_int4_accuracy PASSED [ 44%]
|
|
326
|
+
tests/test_quantization.py::test_block_manager_int8_quantization_lifecycle PASSED [ 46%]
|
|
327
|
+
tests/test_rope_adjoint.py::test_rope_adjoint_identity_scalar ... PASSED [ 65%]
|
|
328
|
+
tests/test_rope_adjoint.py::test_rope_adjoint_batched_multihead ... PASSED [ 74%]
|
|
329
|
+
tests/test_rope_adjoint.py::test_rotate_half_orthogonality PASSED [ 76%]
|
|
330
|
+
tests/test_sparse_bridge_e2e.py::test_qwen2_dag_kv_e2e_stitching_and_repair PASSED [ 79%]
|
|
331
|
+
tests/test_triton_sm75.py::test_triton_adjoint_paged_attention_accuracy ... PASSED [ 93%]
|
|
332
|
+
tests/test_triton_sm75.py::test_triton_sparse_bridge_attention_accuracy ... PASSED [ 97%]
|
|
333
|
+
tests/test_triton_sm75.py::test_turing_sm75_shared_memory_limits PASSED [100%]
|
|
334
|
+
|
|
335
|
+
============================= 43 passed in 45.73s ==============================
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
---
|
|
339
|
+
|
|
340
|
+
## Citation & License
|
|
341
|
+
|
|
342
|
+
```bibtex
|
|
343
|
+
@software{dag_kv_2026,
|
|
344
|
+
title={{DAG-KV: Universal Position-Decoupled Graph KV Cache with Query-Side Adjoint RoPE & LSE-Bounded Sparse Bridge Repair}},
|
|
345
|
+
author={Nandaniya, Kaushal and Collaborators},
|
|
346
|
+
year={2026},
|
|
347
|
+
url={https://github.com/kaushalnandaniya/dag-kv}
|
|
348
|
+
}
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
Licensed under the [Apache 2.0 License](LICENSE).
|