phantom-kv 0.6.1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- phantom_kv-0.6.1.dist-info/METADATA +731 -0
- phantom_kv-0.6.1.dist-info/RECORD +24 -0
- phantom_kv-0.6.1.dist-info/WHEEL +5 -0
- phantom_kv-0.6.1.dist-info/entry_points.txt +5 -0
- phantom_kv-0.6.1.dist-info/licenses/LICENSE +21 -0
- phantom_kv-0.6.1.dist-info/top_level.txt +1 -0
- phantomkv/__init__.py +9 -0
- phantomkv/__main__.py +8 -0
- phantomkv/cli.py +456 -0
- phantomkv/engine.py +286 -0
- phantomkv/lmcache_server.py +118 -0
- phantomkv/offload_map.py +26 -0
- phantomkv/provenance.py +58 -0
- phantomkv/shrink.py +87 -0
- phantomkv/sim.py +172 -0
- phantomkv/tags.py +56 -0
- phantomkv/vllm_fault_connector.py +214 -0
- phantomkv/vllm_harness.py +709 -0
- phantomkv/vllm_model.py +227 -0
- phantomkv/vllm_offload.py +200 -0
- phantomkv/vllm_offload2.py +213 -0
- phantomkv/vllm_pd.py +440 -0
- phantomkv/vllm_plants.py +132 -0
- phantomkv/vllm_plugin.py +34 -0
|
@@ -0,0 +1,731 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: phantom-kv
|
|
3
|
+
Version: 0.6.1
|
|
4
|
+
Summary: Deterministic simulation testing for LLM-serving KV caches
|
|
5
|
+
Author: Sameer Ahmed
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/sameer-sde/phantom-kv
|
|
8
|
+
Project-URL: Repository, https://github.com/sameer-sde/phantom-kv
|
|
9
|
+
Project-URL: Changelog, https://github.com/sameer-sde/phantom-kv/blob/main/CHANGELOG.md
|
|
10
|
+
Project-URL: Blog, https://github.com/sameer-sde/phantom-kv/blob/main/docs/blog/phantom.md
|
|
11
|
+
Project-URL: Issues, https://github.com/sameer-sde/phantom-kv/issues
|
|
12
|
+
Keywords: llm,kv-cache,vllm,lmcache,testing,fuzzing,deterministic-simulation,prefix-caching,inference
|
|
13
|
+
Classifier: Development Status :: 4 - Beta
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Intended Audience :: Science/Research
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
22
|
+
Classifier: Topic :: Software Development :: Testing
|
|
23
|
+
Requires-Python: >=3.10
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE
|
|
26
|
+
Provides-Extra: dev
|
|
27
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
28
|
+
Dynamic: license-file
|
|
29
|
+
|
|
30
|
+
# Phantom — Deterministic Simulation Testing for LLM-Serving KV Caches
|
|
31
|
+
|
|
32
|
+
[](https://github.com/sameer-sde/phantom-kv/actions/workflows/ci.yml)
|
|
33
|
+
[](https://github.com/sameer-sde/phantom-kv/actions/workflows/vllm.yml)
|
|
34
|
+
[](https://pypi.org/project/phantom-kv/)
|
|
35
|
+
[](https://python.org)
|
|
36
|
+
[](#quick-start)
|
|
37
|
+
[](#testing)
|
|
38
|
+
[](https://github.com/sameer-sde/phantom-kv/blob/main/LICENSE)
|
|
39
|
+
|
|
40
|
+
A testing framework that finds **cross-request data leaks and KV-cache
|
|
41
|
+
corruption** in LLM inference engines such as vLLM, SGLang and LMCache.
|
|
42
|
+
It does this **exactly** (hash equality, not floating-point comparison),
|
|
43
|
+
**on a laptop CPU** (no GPU, no model weights), and **reproducibly** (one
|
|
44
|
+
seed replays the entire run, event for event, and failures shrink automatically
|
|
45
|
+
to a handful of events).
|
|
46
|
+
|
|
47
|
+
**Blog post:** [A model with no weights found two real bugs in LMCache](https://github.com/sameer-sde/phantom-kv/blob/main/docs/blog/phantom.md)
|
|
48
|
+
(how Phantom works, and the two LMCache bugs it found independently).
|
|
49
|
+
|
|
50
|
+
> Phantom is a model with no weights. To the serving engine it looks like a
|
|
51
|
+
> real LLM: it is scheduled, batched, preempted, prefix-cached, and its KV
|
|
52
|
+
> cache is shipped between machines. But instead of K/V vectors it writes a
|
|
53
|
+
> **provenance tag** into every cache slot, so any slot read by the wrong
|
|
54
|
+
> request is caught and the request that wrote it is named.
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
## Why this project
|
|
59
|
+
|
|
60
|
+
LLM servers squeeze throughput out of GPUs by **sharing memory between
|
|
61
|
+
users**: paged KV blocks, prefix caching, preemption, CPU offloading, and
|
|
62
|
+
**prefill/decode (PD) disaggregation**, where one machine computes the
|
|
63
|
+
prompt's KV cache and streams it to another with one-sided RDMA writes.
|
|
64
|
+
When that bookkeeping is wrong, nothing crashes. A user silently gets
|
|
65
|
+
corrupted output, or **another user's data**.
|
|
66
|
+
|
|
67
|
+
These bugs are hard to catch today:
|
|
68
|
+
|
|
69
|
+
| Problem | Why existing testing struggles |
|
|
70
|
+
|---|---|
|
|
71
|
+
| **Noisy oracle** | Comparing outputs on real GPUs is ambiguous: a *legitimate* prefix-cache hit already perturbs floating-point numerics, so "different output" ≠ "bug". |
|
|
72
|
+
| **Rare interleavings** | The dangerous cases (a client aborts *while* a KV write is in flight) depend on exact timing and almost never reproduce. |
|
|
73
|
+
| **Cost** | GPU fuzzing campaigns need expensive hardware and run slowly. |
|
|
74
|
+
|
|
75
|
+
Phantom addresses all three with two ideas from the distributed-systems
|
|
76
|
+
world:
|
|
77
|
+
|
|
78
|
+
- **An exact oracle.** Provenance tags + prefix chain hashes turn "is this
|
|
79
|
+
KV correct?" into an equality check.
|
|
80
|
+
- **Deterministic simulation testing**, the technique behind FoundationDB,
|
|
81
|
+
TigerBeetle's VOPR and Antithesis. A single PRNG seed controls every
|
|
82
|
+
arrival, abort, network delay and memory-pressure event, so any failure
|
|
83
|
+
replays exactly.
|
|
84
|
+
|
|
85
|
+
---
|
|
86
|
+
|
|
87
|
+
## Architecture
|
|
88
|
+
|
|
89
|
+
```mermaid
|
|
90
|
+
flowchart TD
|
|
91
|
+
Seed(["seed: u64"])
|
|
92
|
+
|
|
93
|
+
subgraph SIM ["Deterministic Simulator — sim.py"]
|
|
94
|
+
RNG["Single PRNG<br/>random.Random(seed)"]
|
|
95
|
+
WL["Workload generator<br/>arrivals · shared prefixes<br/>prompt lengths · aborts"]
|
|
96
|
+
EQ["Event queue<br/>virtual clock<br/>heap ordered by (time, seq)"]
|
|
97
|
+
NET["Simulated KV network<br/>random delay per block<br/>one-sided writes"]
|
|
98
|
+
RNG --> WL --> EQ
|
|
99
|
+
RNG --> NET
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
subgraph ENG ["Serving engine under test — engine.py"]
|
|
103
|
+
SCH["Scheduler<br/>continuous batching<br/>admission · preemption"]
|
|
104
|
+
KVM["KV block manager<br/>refcounts · free list<br/>LRU evictable list"]
|
|
105
|
+
PC["Prefix cache<br/>chain-hash → block"]
|
|
106
|
+
MEM[("Paged KV memory<br/>slot = Tag | empty")]
|
|
107
|
+
SCH --> KVM
|
|
108
|
+
KVM <--> PC
|
|
109
|
+
KVM --> MEM
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
subgraph PH ["Phantom model — provenance.py"]
|
|
113
|
+
W["write: Tag(writer, pos, chain_hash)"]
|
|
114
|
+
R["read: every context slot<br/>via the engine's block table"]
|
|
115
|
+
O{"chain_hash ==<br/>expected?"}
|
|
116
|
+
R --> O
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
Seed --> RNG
|
|
120
|
+
EQ -->|"submit / abort"| SCH
|
|
121
|
+
NET -->|"remote prefill KV lands"| MEM
|
|
122
|
+
SCH -->|"forward step"| W
|
|
123
|
+
W --> MEM
|
|
124
|
+
MEM --> R
|
|
125
|
+
O -->|"yes"| TOK["next token<br/>H(fold of read hashes)"]
|
|
126
|
+
O -->|"no"| V["Violation<br/>cross-request-leak · stale-read<br/>unwritten-read · block-leak · liveness"]
|
|
127
|
+
V --> TR["Trace + fingerprint<br/>phantomkv replay --seed N"]
|
|
128
|
+
TR --> SH["Shrinker (ddmin)<br/>~50 events → ~6<br/>phantomkv shrink --seed N"]
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
**Request flow:** the simulator submits a request → the scheduler admits it
|
|
132
|
+
(reusing prefix-cache blocks where the chain hash matches) or, for a PD
|
|
133
|
+
request, allocates destination blocks and lets the remote prefill stream
|
|
134
|
+
KV into them → each forward step, the Phantom model writes tags for new
|
|
135
|
+
positions and reads back **every** context slot through the engine's block
|
|
136
|
+
table → any mismatch is recorded with the full event trace.
|
|
137
|
+
|
|
138
|
+
---
|
|
139
|
+
|
|
140
|
+
## The oracle
|
|
141
|
+
|
|
142
|
+
Every request has a chain hash per position:
|
|
143
|
+
`chain[p] = H(chain[p-1], token[p])`, so `chain[p]` uniquely identifies
|
|
144
|
+
the prefix `tokens[0..p]`. The Phantom model writes
|
|
145
|
+
`Tag(writer, pos, chain[pos])` into the slot the engine assigns to `pos`.
|
|
146
|
+
When it reads the context back:
|
|
147
|
+
|
|
148
|
+
| What the slot holds | Verdict |
|
|
149
|
+
|---|---|
|
|
150
|
+
| same chain hash, **any** writer | ✅ correct, including legal prefix sharing between requests |
|
|
151
|
+
| nothing (never written) | ❌ `unwritten-read` |
|
|
152
|
+
| different chain hash, **another** request | ❌ `cross-request-leak` |
|
|
153
|
+
| different chain hash, same request | ❌ `stale-self-read` |
|
|
154
|
+
|
|
155
|
+
Two slots are interchangeable **iff** their chain hashes match. That is
|
|
156
|
+
exactly the rule prefix caching is allowed to exploit, so correct sharing
|
|
157
|
+
passes and everything else is flagged, with **zero false positives by
|
|
158
|
+
construction**. The next token is a hash of everything read, so a bad read
|
|
159
|
+
also changes the visible output, as it would with a real model.
|
|
160
|
+
|
|
161
|
+
---
|
|
162
|
+
|
|
163
|
+
## Core mechanisms
|
|
164
|
+
|
|
165
|
+
| Mechanism | What it does | Where |
|
|
166
|
+
|---|---|---|
|
|
167
|
+
| **Provenance tags** | Replace K/V tensors with `(writer, pos, chain_hash)`; BLAKE2b-64 chain over the prefix | `phantomkv/provenance.py` |
|
|
168
|
+
| **Paged KV block pool** | Fixed-size blocks, refcounts, free list, LRU eviction of unreferenced cached blocks | `phantomkv/engine.py` |
|
|
169
|
+
| **Prefix caching** | Full blocks indexed by the chain hash at the block boundary; hits always leave ≥ 1 token to compute so shared blocks are never written | `phantomkv/engine.py` |
|
|
170
|
+
| **Recompute preemption** | Under memory pressure the newest running request is evicted and later recomputed | `phantomkv/engine.py` |
|
|
171
|
+
| **PD disaggregation** | Decode allocates blocks; remote prefill streams one-sided writes per block with random delay. Writes don't check ownership, just like RDMA | `phantomkv/engine.py`, `phantomkv/sim.py` |
|
|
172
|
+
| **Abort handling** | Aborts at any state; blocks with in-flight writes are freed only after the last write lands | `phantomkv/engine.py` |
|
|
173
|
+
| **Deterministic simulator** | Seed → explicit event list (arrivals, prompts, shared prefixes, aborts); network delay is a hash of (seed, request, block), so it is stable when events are removed; virtual clock; trace fingerprint | `phantomkv/sim.py` |
|
|
174
|
+
| **Liveness + leak checks** | Requests that never finish; blocks still referenced after quiescence | `phantomkv/sim.py`, `phantomkv/engine.py` |
|
|
175
|
+
| **Test-case shrinking** | Delta debugging (ddmin) reduces a failing event list to a 1-minimal reproduction with the same violation kind | `phantomkv/shrink.py` |
|
|
176
|
+
| **Planted bugs** | Switch on known bug classes to prove the oracle catches them before targeting real engines | `--plant` flag |
|
|
177
|
+
|
|
178
|
+
---
|
|
179
|
+
|
|
180
|
+
## The bug class Phantom is built for
|
|
181
|
+
|
|
182
|
+
The abort-during-transfer race in PD disaggregation, the same class SGLang
|
|
183
|
+
disclosed in its
|
|
184
|
+
[October 2026 security update](https://www.lmsys.org/blog/2026-10-26-cf-security/):
|
|
185
|
+
|
|
186
|
+
```mermaid
|
|
187
|
+
sequenceDiagram
|
|
188
|
+
participant C as Client A
|
|
189
|
+
participant P as Prefill engine
|
|
190
|
+
participant D as Decode engine
|
|
191
|
+
participant B as Request B
|
|
192
|
+
|
|
193
|
+
D->>D: allocate block 4 for A
|
|
194
|
+
P-->>D: RDMA write (A, block 4) — in flight
|
|
195
|
+
C->>D: abort A
|
|
196
|
+
Note over D: BUG: free block 4 immediately
|
|
197
|
+
B->>D: admitted → gets block 4, writes its own KV
|
|
198
|
+
P-->>D: A's late write lands in block 4
|
|
199
|
+
B->>D: forward step reads block 4
|
|
200
|
+
Note over B: Phantom: cross-request-leak<br/>B read KV written by A
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
**Correct behaviour:** blocks with outstanding writes are released only when
|
|
204
|
+
the last write lands (or is cancelled and acknowledged).
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
## Validation
|
|
209
|
+
|
|
210
|
+
Before Phantom is trusted on real engines, it must (1) never raise a false
|
|
211
|
+
alarm on a correct engine and (2) catch known bug classes. Measured on a
|
|
212
|
+
MacBook-class CPU, 1,000 seeds per row, 40 requests per seed with mixed
|
|
213
|
+
local/PD traffic, shared prefixes and 25% client aborts:
|
|
214
|
+
|
|
215
|
+
| Engine configuration | Failing seeds | Violations reported |
|
|
216
|
+
|---|---|---|
|
|
217
|
+
| Correct engine | **0 / 1,000** | none (no false positives) |
|
|
218
|
+
| Planted `abort-free-race`: free blocks on abort while KV writes are in flight | **545 / 1,000** | `cross-request-leak` |
|
|
219
|
+
| Planted `stale-prefix-cache`: evict a cached block but keep its hash entry | **1,000 / 1,000** | `cross-request-leak`, `stale-self-read` |
|
|
220
|
+
|
|
221
|
+
| Metric | Value |
|
|
222
|
+
|---|---|
|
|
223
|
+
| Throughput | **~100 seeds/s** (≈ 4,000 simulated requests/s) on one CPU core |
|
|
224
|
+
| Determinism | same seed → identical trace fingerprint (tested) |
|
|
225
|
+
| Shrinking | ~50 events → **median 6, max 13** (92 failing seeds), ~0.2 s per shrink |
|
|
226
|
+
| GPU required | none |
|
|
227
|
+
|
|
228
|
+
---
|
|
229
|
+
|
|
230
|
+
## Proof: a real failure, captured end-to-end
|
|
231
|
+
|
|
232
|
+
```
|
|
233
|
+
$ python -m phantomkv fuzz --seeds 1000 --plant abort-free-race
|
|
234
|
+
phantomkv: 1000 seeds in 10.6s (94 seeds/s), plant=abort-free-race
|
|
235
|
+
545 failing seeds; violation kinds: {'cross-request-leak': 1151}
|
|
236
|
+
first failure: seed 0
|
|
237
|
+
[t=10] cross-request-leak: req 12 pos 0: read KV of req 9 pos 4 (prefix differs)
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
53 events are too many for a bug report, so we shrink them:
|
|
241
|
+
|
|
242
|
+
```
|
|
243
|
+
$ python -m phantomkv shrink --seed 0 --plant abort-free-race
|
|
244
|
+
shrunk seed 0: 53 events -> 6 (29 runs, 0.06s)
|
|
245
|
+
|
|
246
|
+
minimal reproduction:
|
|
247
|
+
t=0 submit req 18 prompt_len=15 max_new=1 local
|
|
248
|
+
t=8 submit req 19 prompt_len=22 max_new=2 local
|
|
249
|
+
t=39 submit req 21 prompt_len=23 max_new=2 remote-prefill
|
|
250
|
+
t=40 submit req 17 prompt_len=16 max_new=7 remote-prefill
|
|
251
|
+
t=45 submit req 20 prompt_len=27 max_new=10 local
|
|
252
|
+
t=48 abort req 21
|
|
253
|
+
|
|
254
|
+
trace (excerpt):
|
|
255
|
+
t=48 abort req 21 (was waiting_kv, inflight=2) # client cancels mid-transfer
|
|
256
|
+
t=48 kv-land req 17 blk#2->18 left=1 # buggy engine already freed req 21's
|
|
257
|
+
t=49 kv-land req 17 blk#0->16 left=0 # blocks; req 17 now owns block 10
|
|
258
|
+
t=51 kv-land req 21 blk#0->10 left=0 # req 21's late write lands in block 10
|
|
259
|
+
t=51 VIOLATION cross-request-leak: req 17 pos 16: read KV of req 21 pos 0
|
|
260
|
+
|
|
261
|
+
=> [t=51] cross-request-leak: req 17 pos 16: read KV of req 21 pos 0 (prefix differs)
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
Every one of the 6 events is necessary: removing any single one makes the
|
|
265
|
+
failure disappear (checked in `tests/test_shrink.py`). Same seed, same
|
|
266
|
+
output, on any machine.
|
|
267
|
+
|
|
268
|
+
---
|
|
269
|
+
|
|
270
|
+
## Running inside real vLLM
|
|
271
|
+
|
|
272
|
+
Phantom now runs **inside vLLM's real engine** (v0.33 dev, CPU build on Apple
|
|
273
|
+
Silicon) as an out-of-tree model plugin. vLLM's own scheduler, KV-cache
|
|
274
|
+
manager, prefix cache and block tables decide where every token's KV goes;
|
|
275
|
+
Phantom writes a tag there and re-reads the whole context through vLLM's
|
|
276
|
+
block table at every step.
|
|
277
|
+
|
|
278
|
+
```
|
|
279
|
+
$ python scripts/vllm_phantom_check.py
|
|
280
|
+
STATS: {'steps': 34, 'tokens_written': 748, 'slots_checked': 19776, 'violations': 0}
|
|
281
|
+
OUTPUT MISMATCHES: 0
|
|
282
|
+
VIOLATIONS: 0
|
|
283
|
+
RESULT: PASS
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
8 prompts with long shared prefixes, generated twice: vLLM served 608 of
|
|
287
|
+
1,356 positions from its prefix cache, and every one of 19,776 slot reads was
|
|
288
|
+
correct. No GPU, no model weights. How it plugs in:
|
|
289
|
+
[docs/VLLM_INTEGRATION.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/VLLM_INTEGRATION.md).
|
|
290
|
+
|
|
291
|
+
### Catching a bug planted inside real vLLM
|
|
292
|
+
|
|
293
|
+
To prove Phantom catches real failures in the real engine, we plant a known
|
|
294
|
+
bug class at runtime (vLLM's source is untouched): `stale-prefix-cache` makes
|
|
295
|
+
`BlockPool._maybe_evict_cached_block` forget to remove the hash → block index
|
|
296
|
+
entry when it evicts a cached block. Under memory pressure (48 KV blocks plus
|
|
297
|
+
a churn phase that forces eviction and reuse):
|
|
298
|
+
|
|
299
|
+
| vLLM | Slots checked | Violations | Wrong outputs | Result |
|
|
300
|
+
|---|---|---|---|---|
|
|
301
|
+
| unmodified | 55,872 | 0 | 0 | ✅ PASS |
|
|
302
|
+
| `stale-prefix-cache` planted | 47,488 | **96** `misplaced-block` | **7 of 8** | ❌ caught |
|
|
303
|
+
|
|
304
|
+
```
|
|
305
|
+
$ python scripts/vllm_phantom_check.py --pressure --plant stale-prefix-cache
|
|
306
|
+
VIOLATIONS: 96 {'misplaced-block': 96}
|
|
307
|
+
{'kind': 'misplaced-block', 'step': 99, 'batch_row': 0, 'pos': 0, 'detail': 'slot 32 holds position 64'}
|
|
308
|
+
RESULT: FAIL
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
A request asked for its first prompt block; the stale index pointed it at a
|
|
312
|
+
block that had been recycled for another request's positions 64–95. Phantom
|
|
313
|
+
names the exact slot.
|
|
314
|
+
|
|
315
|
+
### Seeded, step-by-step fuzzing of real vLLM
|
|
316
|
+
|
|
317
|
+
`phantomkv vllm-fuzz` drives vLLM's real `LLMEngine` one `step()` at a time
|
|
318
|
+
from a seeded event list: 24 requests per seed with shared prefixes of 1–3
|
|
319
|
+
whole blocks, ~25% of them aborted at chosen steps, under a 48-block KV cache.
|
|
320
|
+
Every output (including partial outputs of aborted requests) is compared with
|
|
321
|
+
the reference.
|
|
322
|
+
|
|
323
|
+
| Run | Result |
|
|
324
|
+
|---|---|
|
|
325
|
+
| unmodified vLLM, 20 seeds, each run twice (`--check-determinism`) | ✅ 0 violations; every seed identical across both runs |
|
|
326
|
+
| `--plant stale-prefix-cache`, 20 seeds | ❌ **caught on 15 of 20 seeds** |
|
|
327
|
+
|
|
328
|
+
Each seed sends 24 requests, then a second wave of 10 that come back for the
|
|
329
|
+
first blocks of earlier prompts after the cache has churned, which is exactly when
|
|
330
|
+
a stale cache entry does damage. ~190 requests/s on one CPU core. Along the way Phantom raised one false alarm
|
|
331
|
+
on unmodified vLLM; it was traced to Phantom (vLLM caches blocks at schedule
|
|
332
|
+
time, so the model must write the whole batch before checking, like real
|
|
333
|
+
attention) and fixed. See the [devlog](https://github.com/sameer-sde/phantom-kv/blob/main/docs/DEVLOG.md#day-7--2026-11-02).
|
|
334
|
+
|
|
335
|
+
### Shrinking a real-vLLM failure
|
|
336
|
+
|
|
337
|
+
`phantomkv vllm-shrink` runs delta debugging against the real engine,
|
|
338
|
+
re-running vLLM for every candidate event list:
|
|
339
|
+
|
|
340
|
+
```
|
|
341
|
+
$ python -m phantomkv vllm-shrink --seed 1 --plant stale-prefix-cache
|
|
342
|
+
shrunk seed 1: 38 events -> 13 (235 vLLM runs, 12.5s), kind=misplaced-block
|
|
343
|
+
reproduces 3/3 times on fresh re-runs
|
|
344
|
+
|
|
345
|
+
minimal reproduction (same letter = same first 32-token block):
|
|
346
|
+
step 5 submit req 11 prompt_len=110 first_block=A max_new=13
|
|
347
|
+
... 9 requests with prefixes B..H fill the 48-block cache
|
|
348
|
+
step 87 submit req 33 prompt_len=52 first_block=A max_new=6
|
|
349
|
+
step 97 submit req 31 prompt_len=113 first_block=A max_new=3
|
|
350
|
+
|
|
351
|
+
violations in the minimal run:
|
|
352
|
+
{'kind': 'misplaced-block', ..., 'pos': 64, 'detail': 'slot 224 holds position 96'}
|
|
353
|
+
```
|
|
354
|
+
|
|
355
|
+
Prefix A is cached, evicted under pressure, and requested again; the stale
|
|
356
|
+
index entry hands the new request recycled memory. Every one of the 13 events
|
|
357
|
+
is necessary.
|
|
358
|
+
|
|
359
|
+
|
|
360
|
+
### Phase 3: stressing different parts of vLLM
|
|
361
|
+
|
|
362
|
+
Each profile runs vLLM with a different configuration; every run reports
|
|
363
|
+
vLLM's own counters as proof that the stressed path was really exercised.
|
|
364
|
+
|
|
365
|
+
| Profile | Unmodified vLLM (20 seeds × 2 runs) | `stale-prefix-cache` planted | Evidence the path was exercised |
|
|
366
|
+
|---|---|---|---|
|
|
367
|
+
| baseline | ✅ PASS | ❌ caught 15/20 | 33% prefix-cache hit rate |
|
|
368
|
+
| chunked-prefill (≤64 tokens/step) | ✅ PASS | ❌ caught 15/20 | **498 chunked-prefill continuations**, max 64 tokens/step |
|
|
369
|
+
| preemption (24 blocks, 8 seqs) | ✅ PASS | ❌ caught **20/20** | **16 preemptions** |
|
|
370
|
+
| block64 (64-token blocks) | ✅ PASS | ❌ caught 16/20 | 29% hit rate |
|
|
371
|
+
|
|
372
|
+
```
|
|
373
|
+
bash scripts/vllm_campaign.sh
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
**A second bug class, and what it taught us.** `missing-touch` plants a
|
|
377
|
+
refcount bug (a shared block isn't counted for its second owner). Here vLLM
|
|
378
|
+
defends itself: `assert block.ref_cnt == 0` in `BlockPool.get_new_blocks()`
|
|
379
|
+
(`block_pool.py:730`) fires when the wrongly freed block is handed out again,
|
|
380
|
+
before any corruption is visible to Phantom. Bugs that make vLLM's bookkeeping
|
|
381
|
+
*inconsistent* are caught by vLLM's own assertions; bugs that leave it
|
|
382
|
+
*consistent but wrong*, like a stale index entry, trip nothing, and that is
|
|
383
|
+
where Phantom is needed.
|
|
384
|
+
|
|
385
|
+
Two things the counters and determinism checks exposed along the way: vLLM's
|
|
386
|
+
CPU platform silently disables async scheduling (so that profile was dropped),
|
|
387
|
+
and the multiprocess engine let aborts arrive a step late. The harness now
|
|
388
|
+
runs the engine in-process so every add, abort and step is exact. Details in
|
|
389
|
+
the [devlog](https://github.com/sameer-sde/phantom-kv/blob/main/docs/DEVLOG.md#day-10--2026-11-03).
|
|
390
|
+
|
|
391
|
+
|
|
392
|
+
### Phase 4: two vLLM engines, KV transferred between them
|
|
393
|
+
|
|
394
|
+
Disaggregated serving splits a request across a *prefill* engine and a
|
|
395
|
+
*decode* engine; the KV cache is shipped between them. Phantom now runs on
|
|
396
|
+
both sides: the prefill engine (one process) saves each prompt's KV through
|
|
397
|
+
vLLM's `ExampleConnector`, and the decode engine (another process) loads it
|
|
398
|
+
into its own slots and continues.
|
|
399
|
+
|
|
400
|
+
```
|
|
401
|
+
$ python scripts/vllm_pd_check.py
|
|
402
|
+
PREFILL: 8 prompts, KV files saved: 8
|
|
403
|
+
DECODE STATS: {... 'slots_checked': 12624, 'violations': 0, 'context_not_computed_here': 864}
|
|
404
|
+
DECODE OUTPUT MISMATCHES: 0
|
|
405
|
+
RESULT: PASS
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
864 is exactly the number of positions the connector should transfer for
|
|
409
|
+
these prompts; every one, plus the seam where locally computed KV takes over,
|
|
410
|
+
was verified in the decode engine.
|
|
411
|
+
|
|
412
|
+
**Faults in the transfer.** `PhantomFaultConnector` (vLLM's connector plus
|
|
413
|
+
planted bugs, loaded via `kv_connector_module_path`) breaks the transfer
|
|
414
|
+
silently, so nothing raises and shapes match:
|
|
415
|
+
|
|
416
|
+
| Fault | What goes wrong | Phantom saw | Result |
|
|
417
|
+
|---|---|---|---|
|
|
418
|
+
| none | — | 0 violations, 0 wrong outputs | ✅ PASS |
|
|
419
|
+
| `swap` | decode receives another prompt's KV | **4 wrong outputs** for the 4 swapped requests | ❌ caught |
|
|
420
|
+
| `silent-drop` | load reported done, nothing written | **96 `unwritten-read`** violations | ❌ caught |
|
|
421
|
+
|
|
422
|
+
**Abort while KV is still in flight (the SGLang Oct-2026 bug class).** With an
|
|
423
|
+
asynchronous connector, KV lands 3 steps after the load starts, one-sided like
|
|
424
|
+
RDMA. Four requests are aborted while their KV is in flight, then filler
|
|
425
|
+
requests take whatever blocks are free:
|
|
426
|
+
|
|
427
|
+
| vLLM | Late landings | Phantom saw | Result |
|
|
428
|
+
|---|---|---|---|
|
|
429
|
+
| unmodified (holds an aborted request's blocks until its load lands) | 4 | 0 violations, 0 wrong outputs | ✅ PASS |
|
|
430
|
+
| `abort-free-race` planted (frees them immediately) | 4 | **54 `misplaced-block`** at the step the loads landed, **5 fillers corrupted** | ❌ caught |
|
|
431
|
+
|
|
432
|
+
```
|
|
433
|
+
$ python scripts/vllm_pd_check.py --abort-mid-transfer --plant abort-free-race
|
|
434
|
+
ABORTED while KV in flight: ['d0', 'd2', 'd4', 'd6']
|
|
435
|
+
DECODE VIOLATIONS: 54
|
|
436
|
+
{'kind': 'misplaced-block', 'step': 4, 'batch_row': 0, 'pos': 0, 'detail': 'slot 800 holds position 96'}
|
|
437
|
+
RESULT: CAUGHT
|
|
438
|
+
```
|
|
439
|
+
|
|
440
|
+
**Seeded two-engine fuzzing.** `phantomkv pd-fuzz` turns this into a search:
|
|
441
|
+
each seed picks the load delay (1-5 steps), which prefilled prompts to
|
|
442
|
+
transfer and when, which to abort (before the load starts, while KV is in
|
|
443
|
+
flight, after it lands) and when local requests arrive to take freed blocks.
|
|
444
|
+
Every abort is classified from the connector's log, so a campaign proves the
|
|
445
|
+
race was actually exercised.
|
|
446
|
+
|
|
447
|
+
| 20 seeds | Failing | Evidence |
|
|
448
|
+
|---|---|---|
|
|
449
|
+
| unmodified vLLM (each seed run twice) | **0**, deterministic | 137 async loads, 94 aborts, **35 while KV was in flight** |
|
|
450
|
+
| `abort-free-race` planted | **14 caught** | 93 `misplaced-block`, 18 `chain-break`, 19 wrong outputs |
|
|
451
|
+
|
|
452
|
+
A failing seed shrinks to the whole bug in two events:
|
|
453
|
+
|
|
454
|
+
```
|
|
455
|
+
$ python -m phantomkv pd-shrink --seed 0 --plant abort-free-race
|
|
456
|
+
shrunk seed 0 (load delay 4 steps): 24 events -> 2 (36 vLLM runs, 0.4s), kind=misplaced-block
|
|
457
|
+
reproduces 3/3 times on fresh re-runs
|
|
458
|
+
|
|
459
|
+
step 3 submit req 1 prompt_len=130 max_new=9 TRANSFERRED (KV from prefill engine)
|
|
460
|
+
step 4 abort req 1
|
|
461
|
+
{'kind': 'misplaced-block', 'pos': 0, 'detail': 'slot 416 holds position 96'}
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
**Campaign, 200 seeds per run** (`bash scripts/pd_campaign.sh`): the same
|
|
465
|
+
scenarios against unmodified vLLM and against every planted bug and fault.
|
|
466
|
+
|
|
467
|
+
| Decode engine | Failing seeds | Result |
|
|
468
|
+
|---|---|---|
|
|
469
|
+
| unmodified vLLM (each seed run twice) | **0/200**, deterministic | ✅ PASS |
|
|
470
|
+
| `abort-free-race` (frees blocks while KV is in flight) | 115/200 | ❌ caught |
|
|
471
|
+
| `swap` (another request's KV) | 199/200 | ❌ caught |
|
|
472
|
+
| `silent-drop` (load reported done, nothing written) | 199/200 | ❌ caught |
|
|
473
|
+
| `duplicate` (stale copy of a write lands later) | 184/200 | ❌ caught |
|
|
474
|
+
|
|
475
|
+
1,371 async loads, 995 aborts (335 while KV was in flight), 73,376 transferred
|
|
476
|
+
tokens read. The one seed swap and silent-drop did not break is seed 159: every
|
|
477
|
+
request whose KV was transferred was aborted before reading it
|
|
478
|
+
(`transferred_tokens=0`), so there was nothing to corrupt.
|
|
479
|
+
|
|
480
|
+
```
|
|
481
|
+
python scripts/vllm_pd_check.py --fault swap
|
|
482
|
+
```
|
|
483
|
+
|
|
484
|
+
A swapped context is a *valid chain for the wrong prompt*: every slot extends
|
|
485
|
+
the one before it, so the in-model check can't see it, but the output check
|
|
486
|
+
against the true prompt does. An empty slot is caught in-model on the first
|
|
487
|
+
read. Two independent oracles, each catching what the other can't. All of these
|
|
488
|
+
runs are part of CI. Design: [docs/PD_DESIGN.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/PD_DESIGN.md).
|
|
489
|
+
|
|
490
|
+
### Phase 5: hunting in vLLM code nobody planted bugs in
|
|
491
|
+
|
|
492
|
+
Phases 2-4 proved Phantom catches bugs by planting known ones. Phase 5 points
|
|
493
|
+
it at vLLM's KV **offloading** code: when the GPU cache is full, blocks are
|
|
494
|
+
copied to CPU memory and loaded back when a later request needs the same
|
|
495
|
+
prefix. The bookkeeping that decides what goes where is exactly the
|
|
496
|
+
"consistent but wrong" kind of code Phantom is built for.
|
|
497
|
+
|
|
498
|
+
Both of vLLM's offloading systems run on the CPU backend with their decision
|
|
499
|
+
logic **unmodified**. Only the copy engine (CUDA streams, which the CPU
|
|
500
|
+
backend lacks) is replaced with tensor copies that follow the same
|
|
501
|
+
instructions, on separate load and store queues like the real streams. For
|
|
502
|
+
target 2 this goes through vLLM's own pluggable `OffloadingSpec`, so no vLLM
|
|
503
|
+
code is patched at all.
|
|
504
|
+
|
|
505
|
+
| Target | vLLM code under test | Design |
|
|
506
|
+
|---|---|---|
|
|
507
|
+
| 1. `SimpleCPUOffloadConnector` | ~1,500-line scheduler-side manager | [OFFLOAD_DESIGN.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/OFFLOAD_DESIGN.md) |
|
|
508
|
+
| 2. `OffloadingConnector` | ~2,600-line scheduler + `CPUOffloadingManager` | [OFFLOAD2_DESIGN.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/OFFLOAD2_DESIGN.md) |
|
|
509
|
+
|
|
510
|
+
**Target 2 campaign, 50 seeds per run** (`bash scripts/offload_campaign.sh`):
|
|
511
|
+
|
|
512
|
+
| Run | Result | Evidence |
|
|
513
|
+
|---|---|---|
|
|
514
|
+
| 8 profiles on unmodified vLLM, every seed replayed | **PASS, 0/50 each**, deterministic | 21,181 blocks stored, 2,243 loaded back; 56 partial-chunk loads; 96 sync loads; 314 flush waits |
|
|
515
|
+
| `store-reported-early` planted | ❌ **caught 24/50** | |
|
|
516
|
+
| every 3rd load **fails** (storage failure) | ✅ **recovered**, outputs correct | 117 failed loads, requests recomputed |
|
|
517
|
+
| every 3rd store **fails** (storage failure) | ✅ **recovered**, outputs correct | 757 failed stores, never served |
|
|
518
|
+
|
|
519
|
+
The last two rows are not bugs we planted: they are failures real storage
|
|
520
|
+
produces, which vLLM's API says it handles. It does.
|
|
521
|
+
|
|
522
|
+
**Result so far: no bug found in either offloading system.** That is a real
|
|
523
|
+
result for this code under these conditions, and the harness has been
|
|
524
|
+
validated in both directions (planted faults caught, unmodified runs clean
|
|
525
|
+
and replayable).
|
|
526
|
+
|
|
527
|
+
Two "mysteries" on the way were my own harness, and `phantomkv vllm-diff`
|
|
528
|
+
(run a seed twice, print the first step where the runs differ) found the
|
|
529
|
+
second one in one line:
|
|
530
|
+
|
|
531
|
+
```
|
|
532
|
+
$ python -m phantomkv vllm-diff --profile cpu-offload-very-slow --seed 0
|
|
533
|
+
first divergence at trace entry 6:
|
|
534
|
+
--- FIRST RUN
|
|
535
|
+
start [reset_prefix_cache -> True]
|
|
536
|
+
--- REPLAY
|
|
537
|
+
start [reset_prefix_cache -> False]
|
|
538
|
+
```
|
|
539
|
+
|
|
540
|
+
vLLM was right to refuse the reset (it still had stores queued); the harness
|
|
541
|
+
had stopped stepping while vLLM had pending work. Fixed by stepping while
|
|
542
|
+
`scheduler.has_requests()` is true, as vLLM's own loop does.
|
|
543
|
+
|
|
544
|
+
---
|
|
545
|
+
|
|
546
|
+
## Quick start
|
|
547
|
+
|
|
548
|
+
### Requirements
|
|
549
|
+
|
|
550
|
+
- Python 3.10+
|
|
551
|
+
- No third-party dependencies (pytest for tests)
|
|
552
|
+
|
|
553
|
+
### Install
|
|
554
|
+
|
|
555
|
+
```bash
|
|
556
|
+
pip install phantom-kv
|
|
557
|
+
phantomkv fuzz --seeds 1000 --plant abort-free-race
|
|
558
|
+
```
|
|
559
|
+
|
|
560
|
+
To work on Phantom itself, or to run the real-vLLM and LMCache scripts:
|
|
561
|
+
|
|
562
|
+
```bash
|
|
563
|
+
git clone https://github.com/sameer-sde/phantom-kv.git
|
|
564
|
+
cd phantom-kv
|
|
565
|
+
pip install -e ".[dev]"
|
|
566
|
+
```
|
|
567
|
+
|
|
568
|
+
### Run
|
|
569
|
+
|
|
570
|
+
```bash
|
|
571
|
+
# Correct engine: expect "no violations found"
|
|
572
|
+
python -m phantomkv fuzz --seeds 1000
|
|
573
|
+
|
|
574
|
+
# Prove the oracle catches a known bug class
|
|
575
|
+
python -m phantomkv fuzz --seeds 1000 --plant abort-free-race
|
|
576
|
+
|
|
577
|
+
# Replay one failing seed and print the story leading to the violation
|
|
578
|
+
python -m phantomkv replay --seed 0 --plant abort-free-race
|
|
579
|
+
|
|
580
|
+
# Shrink it to a minimal reproduction for a bug report
|
|
581
|
+
python -m phantomkv shrink --seed 0 --plant abort-free-race
|
|
582
|
+
|
|
583
|
+
# Heavier fault mix
|
|
584
|
+
python -m phantomkv fuzz --seeds 500 --remote-frac 0.9 --abort-frac 0.5
|
|
585
|
+
```
|
|
586
|
+
|
|
587
|
+
### CLI reference
|
|
588
|
+
|
|
589
|
+
| Command | Flags |
|
|
590
|
+
|---|---|
|
|
591
|
+
| `fuzz` | `--seeds N` · `--start S` · `--plant {abort-free-race, stale-prefix-cache}` · `--remote-frac F` · `--abort-frac F` · `--stop-first` |
|
|
592
|
+
| `replay` | `--seed N` · `--plant …` · `--context K` (trace lines before the violation) · `--full` |
|
|
593
|
+
| `shrink` | `--seed N` · `--plant …` |
|
|
594
|
+
| `vllm-fuzz` | `--seeds N` · `--start S` · `--profile NAME` · `--plant NAME` · `--check-determinism` · `-v` (needs the vLLM environment) |
|
|
595
|
+
| `vllm-shrink` | `--seed N` · `--profile NAME` · `--plant NAME` (needs the vLLM environment) |
|
|
596
|
+
| `vllm-diff` | `--seed N` · `--profile NAME` · `--before SEED` · `--context K` (first step where two runs of a seed differ) |
|
|
597
|
+
| `pd-fuzz` | `--seeds N` · `--fault {swap, silent-drop, duplicate}` · `--plant NAME` · `--check-determinism` · `-v` |
|
|
598
|
+
| `pd-shrink` | `--seed N` · `--fault …` · `--plant …` |
|
|
599
|
+
|
|
600
|
+
---
|
|
601
|
+
|
|
602
|
+
## Testing
|
|
603
|
+
|
|
604
|
+
```bash
|
|
605
|
+
pytest -q
|
|
606
|
+
```
|
|
607
|
+
|
|
608
|
+
| Test | Guarantees |
|
|
609
|
+
|---|---|
|
|
610
|
+
| `test_no_false_positives_on_correct_engine` | 300 seeds, all requests finish or abort, zero violations |
|
|
611
|
+
| `test_no_false_positives_under_heavy_pressure` | 12 blocks, 80% PD, 50% aborts: still zero violations |
|
|
612
|
+
| `test_prefix_cache_is_exercised` | the workload actually produces prefix-cache hits (no vacuous passes) |
|
|
613
|
+
| `test_planted_bug_is_caught[*]` | every planted bug class is detected |
|
|
614
|
+
| `test_abort_race_is_reported_as_cross_request_leak` | the race is classified correctly |
|
|
615
|
+
| `test_same_seed_same_run` / `test_different_seeds_differ` | determinism |
|
|
616
|
+
| `test_event_list_reproduces_seed_exactly` | seed → event list → identical run |
|
|
617
|
+
| `test_network_delay_independent_of_other_events` | removing events doesn't perturb other requests' timing |
|
|
618
|
+
| `test_shrunk_repro_is_small_and_still_fails[*]` | ≤ 15 events, same violation kind |
|
|
619
|
+
| `test_shrunk_repro_is_one_minimal[*]` | removing any single event makes the failure disappear |
|
|
620
|
+
| `test_ddmin_*` | the minimiser itself |
|
|
621
|
+
|
|
622
|
+
CI runs on every push:
|
|
623
|
+
|
|
624
|
+
* **`ci`**: the test suite on Python 3.10 and 3.12, plus a 1,000-seed clean fuzz of the simulator.
|
|
625
|
+
* **`real-vllm`**: installs vLLM's Linux CPU wheel (pinned) and runs
|
|
626
|
+
[`scripts/ci_vllm.sh`](https://github.com/sameer-sde/phantom-kv/blob/main/scripts/ci_vllm.sh). Unmodified vLLM must pass every check
|
|
627
|
+
(no false positives), and with a bug planted in vLLM's block pool the checks
|
|
628
|
+
must fail (no false negatives).
|
|
629
|
+
|
|
630
|
+
---
|
|
631
|
+
|
|
632
|
+
## Roadmap
|
|
633
|
+
|
|
634
|
+
| Phase | Status |
|
|
635
|
+
|---|---|
|
|
636
|
+
| 0. Exact oracle + deterministic simulator, validated on a toy engine | ✅ done |
|
|
637
|
+
| 1. Test-case shrinking: failing seed → minimal event trace | ✅ done |
|
|
638
|
+
| 2. Phantom as an out-of-tree **vLLM** model on the CPU backend, writing tags through vLLM's real block tables and slot mappings | ✅ done: runs inside real vLLM, planted bug caught (15/20 seeds), seeded step-by-step fuzzing + shrinking, real-vLLM checks in CI on every push |
|
|
639
|
+
| 3. More chaos on real vLLM: chunked prefill, preemption, block sizes | ✅ 4 profiles clean on unmodified vLLM, `stale-prefix-cache` caught in each, chunking/preemption proven by counters; refcount bugs shown to be caught by vLLM's own assertions |
|
|
640
|
+
| 4. Two-engine PD disaggregation with transfer faults (swap, drop, duplicate, abort mid-transfer) | ✅ v0.5.0: SGLang Oct-2026 class reproduced and caught; seeded PD fuzzing + shrinking; 200-seed campaign 0 false positives, every planted bug and fault caught; all in CI |
|
|
641
|
+
| 5. Bug-hunting campaigns on vLLM, SGLang, LMCache, llm-d; issues and fix PRs upstream | ✅ v0.6.0: vLLM's `SimpleCPUOffloadConnector` ([design](https://github.com/sameer-sde/phantom-kv/blob/main/docs/OFFLOAD_DESIGN.md)) and `OffloadingConnector` ([design](https://github.com/sameer-sde/phantom-kv/blob/main/docs/OFFLOAD2_DESIGN.md)), each with its own code unmodified: clean, planted faults caught, failed loads/stores recovered, in CI. LMCache (CPU-only server): planted bug caught 10/10, and two real open bugs found independently: a fast server restart is never noticed by vLLM (issue #4642) and aborting a request whose lookup finished before it was scheduled leaks read locks for 300 s (issue #5656, shrunk to 7 events). Phantom's A/B tests show both open fixes work (PR #6106: stores per server lifetime 19, 0, 0, 0 → 19, 20, 21, 19; PR #6540: leaked locks 2 and 3 → 0). Not covered: SGLang, llm-d, NIXL (need GPUs); no new issues filed, both bugs were already reported |
|
|
642
|
+
|
|
643
|
+
Releases: [v0.1.0 – v0.6.1](https://github.com/sameer-sde/phantom-kv/releases), one per finished phase ([CHANGELOG](https://github.com/sameer-sde/phantom-kv/blob/main/CHANGELOG.md)).
|
|
644
|
+
|
|
645
|
+
Details: [docs/ROADMAP.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/ROADMAP.md) · design: [docs/DESIGN.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/DESIGN.md) · vLLM integration: [docs/VLLM_INTEGRATION.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/VLLM_INTEGRATION.md) · daily progress: [docs/DEVLOG.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/DEVLOG.md)
|
|
646
|
+
|
|
647
|
+
---
|
|
648
|
+
|
|
649
|
+
## Project structure
|
|
650
|
+
|
|
651
|
+
```
|
|
652
|
+
phantom-kv/
|
|
653
|
+
├── phantomkv/
|
|
654
|
+
│ ├── provenance.py # Tag, chain hashes, phantom attention + LM head
|
|
655
|
+
│ ├── engine.py # ToyEngine: paged KV, prefix cache, preemption, PD, aborts, oracle
|
|
656
|
+
│ ├── sim.py # Event list, workload generator, deterministic simulator
|
|
657
|
+
│ ├── shrink.py # ddmin delta debugging → minimal reproductions
|
|
658
|
+
│ ├── cli.py # fuzz / replay / shrink commands
|
|
659
|
+
│ ├── tags.py # float32-exact tag encoding + reference outputs
|
|
660
|
+
│ ├── vllm_model.py # PhantomForCausalLM: writes/verifies tags in vLLM's KV cache
|
|
661
|
+
│ ├── vllm_plugin.py # registers it with vLLM (vllm.general_plugins)
|
|
662
|
+
│ ├── vllm_plants.py # known bug classes planted into vLLM at runtime
|
|
663
|
+
│ ├── vllm_fault_connector.py # ExampleConnector + planted transfer faults
|
|
664
|
+
│ ├── vllm_harness.py # seeded step-by-step driver for real vLLM
|
|
665
|
+
│ ├── vllm_pd.py # seeded two-engine (prefill/decode) fuzzing + shrinking
|
|
666
|
+
│ ├── vllm_offload.py # vLLM's CPU KV offloading on the CPU backend (copy engine only replaced)
|
|
667
|
+
│ ├── vllm_offload2.py # vLLM's OffloadingConnector, via its pluggable OffloadingSpec
|
|
668
|
+
│ ├── offload_map.py # block <-> chunk mapping for offloaded KV
|
|
669
|
+
│ └── __main__.py # python -m phantomkv
|
|
670
|
+
├── tests/
|
|
671
|
+
│ ├── test_oracle.py # no false positives · planted bugs caught · determinism
|
|
672
|
+
│ ├── test_shrink.py # minimal, still-failing, 1-minimal reproductions
|
|
673
|
+
│ ├── test_tags.py # tags round-trip exactly through float32
|
|
674
|
+
│ └── test_pd_workload.py # seeded PD scenarios: delays, abort timings, fillers
|
|
675
|
+
├── docs/
|
|
676
|
+
│ ├── DESIGN.md # oracle, determinism, mapping to vLLM
|
|
677
|
+
│ ├── VLLM_INTEGRATION.md # phase 2: code map + PhantomForCausalLM design
|
|
678
|
+
│ ├── PD_DESIGN.md # phase 4: disaggregated prefill/decode + fault plan
|
|
679
|
+
│ ├── OFFLOAD_DESIGN.md # phase 5 target 1: vLLM's CPU KV offloading
|
|
680
|
+
│ ├── OFFLOAD2_DESIGN.md # phase 5 target 2: vLLM's OffloadingConnector
|
|
681
|
+
│ ├── ROADMAP.md # phases and exit criteria
|
|
682
|
+
│ ├── blog/phantom.md # blog post: how Phantom works and what it found
|
|
683
|
+
│ └── DEVLOG.md # daily engineering log
|
|
684
|
+
├── scripts/
|
|
685
|
+
│ ├── vllm_smoke.py # does vLLM (CPU) run on this machine?
|
|
686
|
+
│ ├── ci_vllm.sh # real-vLLM acceptance checks (CI + local)
|
|
687
|
+
│ ├── vllm_campaign.sh # every profile, unmodified + planted
|
|
688
|
+
│ ├── pd_campaign.sh # two engines: unmodified + every planted bug and fault
|
|
689
|
+
│ ├── offload_campaign.sh # OffloadingConnector: every profile, planted fault, storage failures
|
|
690
|
+
│ ├── lmcache_smoke.py # Phantom in vLLM with an LMCache server as the KV store
|
|
691
|
+
│ ├── lmcache_abort_probe.py # one aborted request at a chosen moment: LMCache locks left?
|
|
692
|
+
│ ├── lmcache_lock_shrink.py # shrink a seed to the fewest events that leak LMCache locks
|
|
693
|
+
│ ├── lmcache_lock_replay.py # replay the saved minimal lock-leak cases
|
|
694
|
+
│ ├── vllm_phantom_hello.py # does real vLLM load and run Phantom?
|
|
695
|
+
│ ├── vllm_phantom_check.py # end-to-end check: --pressure, --plant
|
|
696
|
+
│ └── vllm_pd_check.py # two engines (prefill + decode) via ExampleConnector
|
|
697
|
+
├── CHANGELOG.md # what each release (phase) added
|
|
698
|
+
├── .github/workflows/
|
|
699
|
+
│ ├── ci.yml # unit tests + simulator fuzz
|
|
700
|
+
│ ├── publish.yml # GitHub release -> PyPI (Trusted Publishing)
|
|
701
|
+
│ └── vllm.yml # Phantom inside real vLLM (pinned CPU wheel)
|
|
702
|
+
└── pyproject.toml
|
|
703
|
+
```
|
|
704
|
+
|
|
705
|
+
---
|
|
706
|
+
|
|
707
|
+
## References
|
|
708
|
+
|
|
709
|
+
- Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). SOSP 2023. https://arxiv.org/abs/2309.06180
|
|
710
|
+
- Zheng, L., et al. (2024). SGLang: Efficient Execution of Structured Language Model Programs. https://arxiv.org/abs/2312.07104
|
|
711
|
+
- Zhou, J., et al. (2021). FoundationDB: A Distributed Unbundled Transactional Key Value Store. SIGMOD 2021. Simulation testing: https://apple.github.io/foundationdb/testing.html
|
|
712
|
+
- TigerBeetle VOPR simulator. https://github.com/tigerbeetle/tigerbeetle
|
|
713
|
+
- Kingsbury, K. Jepsen: distributed systems safety research. https://jepsen.io
|
|
714
|
+
- Zeller, A., & Hildebrandt, R. (2002). Simplifying and Isolating Failure-Inducing Input. IEEE Transactions on Software Engineering 28(2).
|
|
715
|
+
- Meng, R., et al. (2023). Greybox Fuzzing of Distributed Systems (Mallory). ACM CCS 2023. https://arxiv.org/abs/2305.02601
|
|
716
|
+
- GRIEF: Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing (2026). https://arxiv.org/abs/2605.11202
|
|
717
|
+
- Gu, Y., et al. (2026). Ekka: Automated Diagnosis of Silent Errors in LLM Inference. ICML 2026.
|
|
718
|
+
- LMSYS (2026). SGLang security update: improving request isolation. https://www.lmsys.org/blog/2026-10-26-cf-security/
|
|
719
|
+
|
|
720
|
+
---
|
|
721
|
+
|
|
722
|
+
## Author
|
|
723
|
+
|
|
724
|
+
**Sameer Ahmed** · [GitHub](https://github.com/sameer-sde)
|
|
725
|
+
|
|
726
|
+
Related work: [vLLM PR #54832](https://github.com/vllm-project/vllm/pull/54832) ·
|
|
727
|
+
[RaftKV](https://github.com/sameer-sde/raftKv) ·
|
|
728
|
+
[Sentinel](https://github.com/sameer-sde/sentinel) ·
|
|
729
|
+
[Vektr](https://github.com/sameer-sde/vektr)
|
|
730
|
+
|
|
731
|
+
License: [MIT](https://github.com/sameer-sde/phantom-kv/blob/main/LICENSE)
|