phantom-kv 0.6.1__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,731 @@
1
+ Metadata-Version: 2.4
2
+ Name: phantom-kv
3
+ Version: 0.6.1
4
+ Summary: Deterministic simulation testing for LLM-serving KV caches
5
+ Author: Sameer Ahmed
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/sameer-sde/phantom-kv
8
+ Project-URL: Repository, https://github.com/sameer-sde/phantom-kv
9
+ Project-URL: Changelog, https://github.com/sameer-sde/phantom-kv/blob/main/CHANGELOG.md
10
+ Project-URL: Blog, https://github.com/sameer-sde/phantom-kv/blob/main/docs/blog/phantom.md
11
+ Project-URL: Issues, https://github.com/sameer-sde/phantom-kv/issues
12
+ Keywords: llm,kv-cache,vllm,lmcache,testing,fuzzing,deterministic-simulation,prefix-caching,inference
13
+ Classifier: Development Status :: 4 - Beta
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Intended Audience :: Science/Research
16
+ Classifier: Operating System :: OS Independent
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Programming Language :: Python :: 3.10
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
22
+ Classifier: Topic :: Software Development :: Testing
23
+ Requires-Python: >=3.10
24
+ Description-Content-Type: text/markdown
25
+ License-File: LICENSE
26
+ Provides-Extra: dev
27
+ Requires-Dist: pytest>=8; extra == "dev"
28
+ Dynamic: license-file
29
+
30
+ # Phantom — Deterministic Simulation Testing for LLM-Serving KV Caches
31
+
32
+ [![CI](https://github.com/sameer-sde/phantom-kv/actions/workflows/ci.yml/badge.svg)](https://github.com/sameer-sde/phantom-kv/actions/workflows/ci.yml)
33
+ [![real-vllm](https://github.com/sameer-sde/phantom-kv/actions/workflows/vllm.yml/badge.svg)](https://github.com/sameer-sde/phantom-kv/actions/workflows/vllm.yml)
34
+ [![PyPI](https://img.shields.io/pypi/v/phantom-kv)](https://pypi.org/project/phantom-kv/)
35
+ [![Python](https://img.shields.io/badge/python-3.10+-3776AB?logo=python&logoColor=white)](https://python.org)
36
+ [![Zero Dependencies](https://img.shields.io/badge/dependencies-zero-brightgreen)](#quick-start)
37
+ [![Tests](https://img.shields.io/badge/tests-36%20passing-success)](#testing)
38
+ [![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/sameer-sde/phantom-kv/blob/main/LICENSE)
39
+
40
+ A testing framework that finds **cross-request data leaks and KV-cache
41
+ corruption** in LLM inference engines such as vLLM, SGLang and LMCache.
42
+ It does this **exactly** (hash equality, not floating-point comparison),
43
+ **on a laptop CPU** (no GPU, no model weights), and **reproducibly** (one
44
+ seed replays the entire run, event for event, and failures shrink automatically
45
+ to a handful of events).
46
+
47
+ **Blog post:** [A model with no weights found two real bugs in LMCache](https://github.com/sameer-sde/phantom-kv/blob/main/docs/blog/phantom.md)
48
+ (how Phantom works, and the two LMCache bugs it found independently).
49
+
50
+ > Phantom is a model with no weights. To the serving engine it looks like a
51
+ > real LLM: it is scheduled, batched, preempted, prefix-cached, and its KV
52
+ > cache is shipped between machines. But instead of K/V vectors it writes a
53
+ > **provenance tag** into every cache slot, so any slot read by the wrong
54
+ > request is caught and the request that wrote it is named.
55
+
56
+ ---
57
+
58
+ ## Why this project
59
+
60
+ LLM servers squeeze throughput out of GPUs by **sharing memory between
61
+ users**: paged KV blocks, prefix caching, preemption, CPU offloading, and
62
+ **prefill/decode (PD) disaggregation**, where one machine computes the
63
+ prompt's KV cache and streams it to another with one-sided RDMA writes.
64
+ When that bookkeeping is wrong, nothing crashes. A user silently gets
65
+ corrupted output, or **another user's data**.
66
+
67
+ These bugs are hard to catch today:
68
+
69
+ | Problem | Why existing testing struggles |
70
+ |---|---|
71
+ | **Noisy oracle** | Comparing outputs on real GPUs is ambiguous: a *legitimate* prefix-cache hit already perturbs floating-point numerics, so "different output" ≠ "bug". |
72
+ | **Rare interleavings** | The dangerous cases (a client aborts *while* a KV write is in flight) depend on exact timing and almost never reproduce. |
73
+ | **Cost** | GPU fuzzing campaigns need expensive hardware and run slowly. |
74
+
75
+ Phantom addresses all three with two ideas from the distributed-systems
76
+ world:
77
+
78
+ - **An exact oracle.** Provenance tags + prefix chain hashes turn "is this
79
+ KV correct?" into an equality check.
80
+ - **Deterministic simulation testing**, the technique behind FoundationDB,
81
+ TigerBeetle's VOPR and Antithesis. A single PRNG seed controls every
82
+ arrival, abort, network delay and memory-pressure event, so any failure
83
+ replays exactly.
84
+
85
+ ---
86
+
87
+ ## Architecture
88
+
89
+ ```mermaid
90
+ flowchart TD
91
+ Seed(["seed: u64"])
92
+
93
+ subgraph SIM ["Deterministic Simulator — sim.py"]
94
+ RNG["Single PRNG<br/>random.Random(seed)"]
95
+ WL["Workload generator<br/>arrivals · shared prefixes<br/>prompt lengths · aborts"]
96
+ EQ["Event queue<br/>virtual clock<br/>heap ordered by (time, seq)"]
97
+ NET["Simulated KV network<br/>random delay per block<br/>one-sided writes"]
98
+ RNG --> WL --> EQ
99
+ RNG --> NET
100
+ end
101
+
102
+ subgraph ENG ["Serving engine under test — engine.py"]
103
+ SCH["Scheduler<br/>continuous batching<br/>admission · preemption"]
104
+ KVM["KV block manager<br/>refcounts · free list<br/>LRU evictable list"]
105
+ PC["Prefix cache<br/>chain-hash → block"]
106
+ MEM[("Paged KV memory<br/>slot = Tag | empty")]
107
+ SCH --> KVM
108
+ KVM <--> PC
109
+ KVM --> MEM
110
+ end
111
+
112
+ subgraph PH ["Phantom model — provenance.py"]
113
+ W["write: Tag(writer, pos, chain_hash)"]
114
+ R["read: every context slot<br/>via the engine's block table"]
115
+ O{"chain_hash ==<br/>expected?"}
116
+ R --> O
117
+ end
118
+
119
+ Seed --> RNG
120
+ EQ -->|"submit / abort"| SCH
121
+ NET -->|"remote prefill KV lands"| MEM
122
+ SCH -->|"forward step"| W
123
+ W --> MEM
124
+ MEM --> R
125
+ O -->|"yes"| TOK["next token<br/>H(fold of read hashes)"]
126
+ O -->|"no"| V["Violation<br/>cross-request-leak · stale-read<br/>unwritten-read · block-leak · liveness"]
127
+ V --> TR["Trace + fingerprint<br/>phantomkv replay --seed N"]
128
+ TR --> SH["Shrinker (ddmin)<br/>~50 events → ~6<br/>phantomkv shrink --seed N"]
129
+ ```
130
+
131
+ **Request flow:** the simulator submits a request → the scheduler admits it
132
+ (reusing prefix-cache blocks where the chain hash matches) or, for a PD
133
+ request, allocates destination blocks and lets the remote prefill stream
134
+ KV into them → each forward step, the Phantom model writes tags for new
135
+ positions and reads back **every** context slot through the engine's block
136
+ table → any mismatch is recorded with the full event trace.
137
+
138
+ ---
139
+
140
+ ## The oracle
141
+
142
+ Every request has a chain hash per position:
143
+ `chain[p] = H(chain[p-1], token[p])`, so `chain[p]` uniquely identifies
144
+ the prefix `tokens[0..p]`. The Phantom model writes
145
+ `Tag(writer, pos, chain[pos])` into the slot the engine assigns to `pos`.
146
+ When it reads the context back:
147
+
148
+ | What the slot holds | Verdict |
149
+ |---|---|
150
+ | same chain hash, **any** writer | ✅ correct, including legal prefix sharing between requests |
151
+ | nothing (never written) | ❌ `unwritten-read` |
152
+ | different chain hash, **another** request | ❌ `cross-request-leak` |
153
+ | different chain hash, same request | ❌ `stale-self-read` |
154
+
155
+ Two slots are interchangeable **iff** their chain hashes match. That is
156
+ exactly the rule prefix caching is allowed to exploit, so correct sharing
157
+ passes and everything else is flagged, with **zero false positives by
158
+ construction**. The next token is a hash of everything read, so a bad read
159
+ also changes the visible output, as it would with a real model.
160
+
161
+ ---
162
+
163
+ ## Core mechanisms
164
+
165
+ | Mechanism | What it does | Where |
166
+ |---|---|---|
167
+ | **Provenance tags** | Replace K/V tensors with `(writer, pos, chain_hash)`; BLAKE2b-64 chain over the prefix | `phantomkv/provenance.py` |
168
+ | **Paged KV block pool** | Fixed-size blocks, refcounts, free list, LRU eviction of unreferenced cached blocks | `phantomkv/engine.py` |
169
+ | **Prefix caching** | Full blocks indexed by the chain hash at the block boundary; hits always leave ≥ 1 token to compute so shared blocks are never written | `phantomkv/engine.py` |
170
+ | **Recompute preemption** | Under memory pressure the newest running request is evicted and later recomputed | `phantomkv/engine.py` |
171
+ | **PD disaggregation** | Decode allocates blocks; remote prefill streams one-sided writes per block with random delay. Writes don't check ownership, just like RDMA | `phantomkv/engine.py`, `phantomkv/sim.py` |
172
+ | **Abort handling** | Aborts at any state; blocks with in-flight writes are freed only after the last write lands | `phantomkv/engine.py` |
173
+ | **Deterministic simulator** | Seed → explicit event list (arrivals, prompts, shared prefixes, aborts); network delay is a hash of (seed, request, block), so it is stable when events are removed; virtual clock; trace fingerprint | `phantomkv/sim.py` |
174
+ | **Liveness + leak checks** | Requests that never finish; blocks still referenced after quiescence | `phantomkv/sim.py`, `phantomkv/engine.py` |
175
+ | **Test-case shrinking** | Delta debugging (ddmin) reduces a failing event list to a 1-minimal reproduction with the same violation kind | `phantomkv/shrink.py` |
176
+ | **Planted bugs** | Switch on known bug classes to prove the oracle catches them before targeting real engines | `--plant` flag |
177
+
178
+ ---
179
+
180
+ ## The bug class Phantom is built for
181
+
182
+ The abort-during-transfer race in PD disaggregation, the same class SGLang
183
+ disclosed in its
184
+ [October 2026 security update](https://www.lmsys.org/blog/2026-10-26-cf-security/):
185
+
186
+ ```mermaid
187
+ sequenceDiagram
188
+ participant C as Client A
189
+ participant P as Prefill engine
190
+ participant D as Decode engine
191
+ participant B as Request B
192
+
193
+ D->>D: allocate block 4 for A
194
+ P-->>D: RDMA write (A, block 4) — in flight
195
+ C->>D: abort A
196
+ Note over D: BUG: free block 4 immediately
197
+ B->>D: admitted → gets block 4, writes its own KV
198
+ P-->>D: A's late write lands in block 4
199
+ B->>D: forward step reads block 4
200
+ Note over B: Phantom: cross-request-leak<br/>B read KV written by A
201
+ ```
202
+
203
+ **Correct behaviour:** blocks with outstanding writes are released only when
204
+ the last write lands (or is cancelled and acknowledged).
205
+
206
+ ---
207
+
208
+ ## Validation
209
+
210
+ Before Phantom is trusted on real engines, it must (1) never raise a false
211
+ alarm on a correct engine and (2) catch known bug classes. Measured on a
212
+ MacBook-class CPU, 1,000 seeds per row, 40 requests per seed with mixed
213
+ local/PD traffic, shared prefixes and 25% client aborts:
214
+
215
+ | Engine configuration | Failing seeds | Violations reported |
216
+ |---|---|---|
217
+ | Correct engine | **0 / 1,000** | none (no false positives) |
218
+ | Planted `abort-free-race`: free blocks on abort while KV writes are in flight | **545 / 1,000** | `cross-request-leak` |
219
+ | Planted `stale-prefix-cache`: evict a cached block but keep its hash entry | **1,000 / 1,000** | `cross-request-leak`, `stale-self-read` |
220
+
221
+ | Metric | Value |
222
+ |---|---|
223
+ | Throughput | **~100 seeds/s** (≈ 4,000 simulated requests/s) on one CPU core |
224
+ | Determinism | same seed → identical trace fingerprint (tested) |
225
+ | Shrinking | ~50 events → **median 6, max 13** (92 failing seeds), ~0.2 s per shrink |
226
+ | GPU required | none |
227
+
228
+ ---
229
+
230
+ ## Proof: a real failure, captured end-to-end
231
+
232
+ ```
233
+ $ python -m phantomkv fuzz --seeds 1000 --plant abort-free-race
234
+ phantomkv: 1000 seeds in 10.6s (94 seeds/s), plant=abort-free-race
235
+ 545 failing seeds; violation kinds: {'cross-request-leak': 1151}
236
+ first failure: seed 0
237
+ [t=10] cross-request-leak: req 12 pos 0: read KV of req 9 pos 4 (prefix differs)
238
+ ```
239
+
240
+ 53 events are too many for a bug report, so we shrink them:
241
+
242
+ ```
243
+ $ python -m phantomkv shrink --seed 0 --plant abort-free-race
244
+ shrunk seed 0: 53 events -> 6 (29 runs, 0.06s)
245
+
246
+ minimal reproduction:
247
+ t=0 submit req 18 prompt_len=15 max_new=1 local
248
+ t=8 submit req 19 prompt_len=22 max_new=2 local
249
+ t=39 submit req 21 prompt_len=23 max_new=2 remote-prefill
250
+ t=40 submit req 17 prompt_len=16 max_new=7 remote-prefill
251
+ t=45 submit req 20 prompt_len=27 max_new=10 local
252
+ t=48 abort req 21
253
+
254
+ trace (excerpt):
255
+ t=48 abort req 21 (was waiting_kv, inflight=2) # client cancels mid-transfer
256
+ t=48 kv-land req 17 blk#2->18 left=1 # buggy engine already freed req 21's
257
+ t=49 kv-land req 17 blk#0->16 left=0 # blocks; req 17 now owns block 10
258
+ t=51 kv-land req 21 blk#0->10 left=0 # req 21's late write lands in block 10
259
+ t=51 VIOLATION cross-request-leak: req 17 pos 16: read KV of req 21 pos 0
260
+
261
+ => [t=51] cross-request-leak: req 17 pos 16: read KV of req 21 pos 0 (prefix differs)
262
+ ```
263
+
264
+ Every one of the 6 events is necessary: removing any single one makes the
265
+ failure disappear (checked in `tests/test_shrink.py`). Same seed, same
266
+ output, on any machine.
267
+
268
+ ---
269
+
270
+ ## Running inside real vLLM
271
+
272
+ Phantom now runs **inside vLLM's real engine** (v0.33 dev, CPU build on Apple
273
+ Silicon) as an out-of-tree model plugin. vLLM's own scheduler, KV-cache
274
+ manager, prefix cache and block tables decide where every token's KV goes;
275
+ Phantom writes a tag there and re-reads the whole context through vLLM's
276
+ block table at every step.
277
+
278
+ ```
279
+ $ python scripts/vllm_phantom_check.py
280
+ STATS: {'steps': 34, 'tokens_written': 748, 'slots_checked': 19776, 'violations': 0}
281
+ OUTPUT MISMATCHES: 0
282
+ VIOLATIONS: 0
283
+ RESULT: PASS
284
+ ```
285
+
286
+ 8 prompts with long shared prefixes, generated twice: vLLM served 608 of
287
+ 1,356 positions from its prefix cache, and every one of 19,776 slot reads was
288
+ correct. No GPU, no model weights. How it plugs in:
289
+ [docs/VLLM_INTEGRATION.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/VLLM_INTEGRATION.md).
290
+
291
+ ### Catching a bug planted inside real vLLM
292
+
293
+ To prove Phantom catches real failures in the real engine, we plant a known
294
+ bug class at runtime (vLLM's source is untouched): `stale-prefix-cache` makes
295
+ `BlockPool._maybe_evict_cached_block` forget to remove the hash → block index
296
+ entry when it evicts a cached block. Under memory pressure (48 KV blocks plus
297
+ a churn phase that forces eviction and reuse):
298
+
299
+ | vLLM | Slots checked | Violations | Wrong outputs | Result |
300
+ |---|---|---|---|---|
301
+ | unmodified | 55,872 | 0 | 0 | ✅ PASS |
302
+ | `stale-prefix-cache` planted | 47,488 | **96** `misplaced-block` | **7 of 8** | ❌ caught |
303
+
304
+ ```
305
+ $ python scripts/vllm_phantom_check.py --pressure --plant stale-prefix-cache
306
+ VIOLATIONS: 96 {'misplaced-block': 96}
307
+ {'kind': 'misplaced-block', 'step': 99, 'batch_row': 0, 'pos': 0, 'detail': 'slot 32 holds position 64'}
308
+ RESULT: FAIL
309
+ ```
310
+
311
+ A request asked for its first prompt block; the stale index pointed it at a
312
+ block that had been recycled for another request's positions 64–95. Phantom
313
+ names the exact slot.
314
+
315
+ ### Seeded, step-by-step fuzzing of real vLLM
316
+
317
+ `phantomkv vllm-fuzz` drives vLLM's real `LLMEngine` one `step()` at a time
318
+ from a seeded event list: 24 requests per seed with shared prefixes of 1–3
319
+ whole blocks, ~25% of them aborted at chosen steps, under a 48-block KV cache.
320
+ Every output (including partial outputs of aborted requests) is compared with
321
+ the reference.
322
+
323
+ | Run | Result |
324
+ |---|---|
325
+ | unmodified vLLM, 20 seeds, each run twice (`--check-determinism`) | ✅ 0 violations; every seed identical across both runs |
326
+ | `--plant stale-prefix-cache`, 20 seeds | ❌ **caught on 15 of 20 seeds** |
327
+
328
+ Each seed sends 24 requests, then a second wave of 10 that come back for the
329
+ first blocks of earlier prompts after the cache has churned, which is exactly when
330
+ a stale cache entry does damage. ~190 requests/s on one CPU core. Along the way Phantom raised one false alarm
331
+ on unmodified vLLM; it was traced to Phantom (vLLM caches blocks at schedule
332
+ time, so the model must write the whole batch before checking, like real
333
+ attention) and fixed. See the [devlog](https://github.com/sameer-sde/phantom-kv/blob/main/docs/DEVLOG.md#day-7--2026-11-02).
334
+
335
+ ### Shrinking a real-vLLM failure
336
+
337
+ `phantomkv vllm-shrink` runs delta debugging against the real engine,
338
+ re-running vLLM for every candidate event list:
339
+
340
+ ```
341
+ $ python -m phantomkv vllm-shrink --seed 1 --plant stale-prefix-cache
342
+ shrunk seed 1: 38 events -> 13 (235 vLLM runs, 12.5s), kind=misplaced-block
343
+ reproduces 3/3 times on fresh re-runs
344
+
345
+ minimal reproduction (same letter = same first 32-token block):
346
+ step 5 submit req 11 prompt_len=110 first_block=A max_new=13
347
+ ... 9 requests with prefixes B..H fill the 48-block cache
348
+ step 87 submit req 33 prompt_len=52 first_block=A max_new=6
349
+ step 97 submit req 31 prompt_len=113 first_block=A max_new=3
350
+
351
+ violations in the minimal run:
352
+ {'kind': 'misplaced-block', ..., 'pos': 64, 'detail': 'slot 224 holds position 96'}
353
+ ```
354
+
355
+ Prefix A is cached, evicted under pressure, and requested again; the stale
356
+ index entry hands the new request recycled memory. Every one of the 13 events
357
+ is necessary.
358
+
359
+
360
+ ### Phase 3: stressing different parts of vLLM
361
+
362
+ Each profile runs vLLM with a different configuration; every run reports
363
+ vLLM's own counters as proof that the stressed path was really exercised.
364
+
365
+ | Profile | Unmodified vLLM (20 seeds × 2 runs) | `stale-prefix-cache` planted | Evidence the path was exercised |
366
+ |---|---|---|---|
367
+ | baseline | ✅ PASS | ❌ caught 15/20 | 33% prefix-cache hit rate |
368
+ | chunked-prefill (≤64 tokens/step) | ✅ PASS | ❌ caught 15/20 | **498 chunked-prefill continuations**, max 64 tokens/step |
369
+ | preemption (24 blocks, 8 seqs) | ✅ PASS | ❌ caught **20/20** | **16 preemptions** |
370
+ | block64 (64-token blocks) | ✅ PASS | ❌ caught 16/20 | 29% hit rate |
371
+
372
+ ```
373
+ bash scripts/vllm_campaign.sh
374
+ ```
375
+
376
+ **A second bug class, and what it taught us.** `missing-touch` plants a
377
+ refcount bug (a shared block isn't counted for its second owner). Here vLLM
378
+ defends itself: `assert block.ref_cnt == 0` in `BlockPool.get_new_blocks()`
379
+ (`block_pool.py:730`) fires when the wrongly freed block is handed out again,
380
+ before any corruption is visible to Phantom. Bugs that make vLLM's bookkeeping
381
+ *inconsistent* are caught by vLLM's own assertions; bugs that leave it
382
+ *consistent but wrong*, like a stale index entry, trip nothing, and that is
383
+ where Phantom is needed.
384
+
385
+ Two things the counters and determinism checks exposed along the way: vLLM's
386
+ CPU platform silently disables async scheduling (so that profile was dropped),
387
+ and the multiprocess engine let aborts arrive a step late. The harness now
388
+ runs the engine in-process so every add, abort and step is exact. Details in
389
+ the [devlog](https://github.com/sameer-sde/phantom-kv/blob/main/docs/DEVLOG.md#day-10--2026-11-03).
390
+
391
+
392
+ ### Phase 4: two vLLM engines, KV transferred between them
393
+
394
+ Disaggregated serving splits a request across a *prefill* engine and a
395
+ *decode* engine; the KV cache is shipped between them. Phantom now runs on
396
+ both sides: the prefill engine (one process) saves each prompt's KV through
397
+ vLLM's `ExampleConnector`, and the decode engine (another process) loads it
398
+ into its own slots and continues.
399
+
400
+ ```
401
+ $ python scripts/vllm_pd_check.py
402
+ PREFILL: 8 prompts, KV files saved: 8
403
+ DECODE STATS: {... 'slots_checked': 12624, 'violations': 0, 'context_not_computed_here': 864}
404
+ DECODE OUTPUT MISMATCHES: 0
405
+ RESULT: PASS
406
+ ```
407
+
408
+ 864 is exactly the number of positions the connector should transfer for
409
+ these prompts; every one, plus the seam where locally computed KV takes over,
410
+ was verified in the decode engine.
411
+
412
+ **Faults in the transfer.** `PhantomFaultConnector` (vLLM's connector plus
413
+ planted bugs, loaded via `kv_connector_module_path`) breaks the transfer
414
+ silently, so nothing raises and shapes match:
415
+
416
+ | Fault | What goes wrong | Phantom saw | Result |
417
+ |---|---|---|---|
418
+ | none | — | 0 violations, 0 wrong outputs | ✅ PASS |
419
+ | `swap` | decode receives another prompt's KV | **4 wrong outputs** for the 4 swapped requests | ❌ caught |
420
+ | `silent-drop` | load reported done, nothing written | **96 `unwritten-read`** violations | ❌ caught |
421
+
422
+ **Abort while KV is still in flight (the SGLang Oct-2026 bug class).** With an
423
+ asynchronous connector, KV lands 3 steps after the load starts, one-sided like
424
+ RDMA. Four requests are aborted while their KV is in flight, then filler
425
+ requests take whatever blocks are free:
426
+
427
+ | vLLM | Late landings | Phantom saw | Result |
428
+ |---|---|---|---|
429
+ | unmodified (holds an aborted request's blocks until its load lands) | 4 | 0 violations, 0 wrong outputs | ✅ PASS |
430
+ | `abort-free-race` planted (frees them immediately) | 4 | **54 `misplaced-block`** at the step the loads landed, **5 fillers corrupted** | ❌ caught |
431
+
432
+ ```
433
+ $ python scripts/vllm_pd_check.py --abort-mid-transfer --plant abort-free-race
434
+ ABORTED while KV in flight: ['d0', 'd2', 'd4', 'd6']
435
+ DECODE VIOLATIONS: 54
436
+ {'kind': 'misplaced-block', 'step': 4, 'batch_row': 0, 'pos': 0, 'detail': 'slot 800 holds position 96'}
437
+ RESULT: CAUGHT
438
+ ```
439
+
440
+ **Seeded two-engine fuzzing.** `phantomkv pd-fuzz` turns this into a search:
441
+ each seed picks the load delay (1-5 steps), which prefilled prompts to
442
+ transfer and when, which to abort (before the load starts, while KV is in
443
+ flight, after it lands) and when local requests arrive to take freed blocks.
444
+ Every abort is classified from the connector's log, so a campaign proves the
445
+ race was actually exercised.
446
+
447
+ | 20 seeds | Failing | Evidence |
448
+ |---|---|---|
449
+ | unmodified vLLM (each seed run twice) | **0**, deterministic | 137 async loads, 94 aborts, **35 while KV was in flight** |
450
+ | `abort-free-race` planted | **14 caught** | 93 `misplaced-block`, 18 `chain-break`, 19 wrong outputs |
451
+
452
+ A failing seed shrinks to the whole bug in two events:
453
+
454
+ ```
455
+ $ python -m phantomkv pd-shrink --seed 0 --plant abort-free-race
456
+ shrunk seed 0 (load delay 4 steps): 24 events -> 2 (36 vLLM runs, 0.4s), kind=misplaced-block
457
+ reproduces 3/3 times on fresh re-runs
458
+
459
+ step 3 submit req 1 prompt_len=130 max_new=9 TRANSFERRED (KV from prefill engine)
460
+ step 4 abort req 1
461
+ {'kind': 'misplaced-block', 'pos': 0, 'detail': 'slot 416 holds position 96'}
462
+ ```
463
+
464
+ **Campaign, 200 seeds per run** (`bash scripts/pd_campaign.sh`): the same
465
+ scenarios against unmodified vLLM and against every planted bug and fault.
466
+
467
+ | Decode engine | Failing seeds | Result |
468
+ |---|---|---|
469
+ | unmodified vLLM (each seed run twice) | **0/200**, deterministic | ✅ PASS |
470
+ | `abort-free-race` (frees blocks while KV is in flight) | 115/200 | ❌ caught |
471
+ | `swap` (another request's KV) | 199/200 | ❌ caught |
472
+ | `silent-drop` (load reported done, nothing written) | 199/200 | ❌ caught |
473
+ | `duplicate` (stale copy of a write lands later) | 184/200 | ❌ caught |
474
+
475
+ 1,371 async loads, 995 aborts (335 while KV was in flight), 73,376 transferred
476
+ tokens read. The one seed swap and silent-drop did not break is seed 159: every
477
+ request whose KV was transferred was aborted before reading it
478
+ (`transferred_tokens=0`), so there was nothing to corrupt.
479
+
480
+ ```
481
+ python scripts/vllm_pd_check.py --fault swap
482
+ ```
483
+
484
+ A swapped context is a *valid chain for the wrong prompt*: every slot extends
485
+ the one before it, so the in-model check can't see it, but the output check
486
+ against the true prompt does. An empty slot is caught in-model on the first
487
+ read. Two independent oracles, each catching what the other can't. All of these
488
+ runs are part of CI. Design: [docs/PD_DESIGN.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/PD_DESIGN.md).
489
+
490
+ ### Phase 5: hunting in vLLM code nobody planted bugs in
491
+
492
+ Phases 2-4 proved Phantom catches bugs by planting known ones. Phase 5 points
493
+ it at vLLM's KV **offloading** code: when the GPU cache is full, blocks are
494
+ copied to CPU memory and loaded back when a later request needs the same
495
+ prefix. The bookkeeping that decides what goes where is exactly the
496
+ "consistent but wrong" kind of code Phantom is built for.
497
+
498
+ Both of vLLM's offloading systems run on the CPU backend with their decision
499
+ logic **unmodified**. Only the copy engine (CUDA streams, which the CPU
500
+ backend lacks) is replaced with tensor copies that follow the same
501
+ instructions, on separate load and store queues like the real streams. For
502
+ target 2 this goes through vLLM's own pluggable `OffloadingSpec`, so no vLLM
503
+ code is patched at all.
504
+
505
+ | Target | vLLM code under test | Design |
506
+ |---|---|---|
507
+ | 1. `SimpleCPUOffloadConnector` | ~1,500-line scheduler-side manager | [OFFLOAD_DESIGN.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/OFFLOAD_DESIGN.md) |
508
+ | 2. `OffloadingConnector` | ~2,600-line scheduler + `CPUOffloadingManager` | [OFFLOAD2_DESIGN.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/OFFLOAD2_DESIGN.md) |
509
+
510
+ **Target 2 campaign, 50 seeds per run** (`bash scripts/offload_campaign.sh`):
511
+
512
+ | Run | Result | Evidence |
513
+ |---|---|---|
514
+ | 8 profiles on unmodified vLLM, every seed replayed | **PASS, 0/50 each**, deterministic | 21,181 blocks stored, 2,243 loaded back; 56 partial-chunk loads; 96 sync loads; 314 flush waits |
515
+ | `store-reported-early` planted | ❌ **caught 24/50** | |
516
+ | every 3rd load **fails** (storage failure) | ✅ **recovered**, outputs correct | 117 failed loads, requests recomputed |
517
+ | every 3rd store **fails** (storage failure) | ✅ **recovered**, outputs correct | 757 failed stores, never served |
518
+
519
+ The last two rows are not bugs we planted: they are failures real storage
520
+ produces, which vLLM's API says it handles. It does.
521
+
522
+ **Result so far: no bug found in either offloading system.** That is a real
523
+ result for this code under these conditions, and the harness has been
524
+ validated in both directions (planted faults caught, unmodified runs clean
525
+ and replayable).
526
+
527
+ Two "mysteries" on the way were my own harness, and `phantomkv vllm-diff`
528
+ (run a seed twice, print the first step where the runs differ) found the
529
+ second one in one line:
530
+
531
+ ```
532
+ $ python -m phantomkv vllm-diff --profile cpu-offload-very-slow --seed 0
533
+ first divergence at trace entry 6:
534
+ --- FIRST RUN
535
+ start [reset_prefix_cache -> True]
536
+ --- REPLAY
537
+ start [reset_prefix_cache -> False]
538
+ ```
539
+
540
+ vLLM was right to refuse the reset (it still had stores queued); the harness
541
+ had stopped stepping while vLLM had pending work. Fixed by stepping while
542
+ `scheduler.has_requests()` is true, as vLLM's own loop does.
543
+
544
+ ---
545
+
546
+ ## Quick start
547
+
548
+ ### Requirements
549
+
550
+ - Python 3.10+
551
+ - No third-party dependencies (pytest for tests)
552
+
553
+ ### Install
554
+
555
+ ```bash
556
+ pip install phantom-kv
557
+ phantomkv fuzz --seeds 1000 --plant abort-free-race
558
+ ```
559
+
560
+ To work on Phantom itself, or to run the real-vLLM and LMCache scripts:
561
+
562
+ ```bash
563
+ git clone https://github.com/sameer-sde/phantom-kv.git
564
+ cd phantom-kv
565
+ pip install -e ".[dev]"
566
+ ```
567
+
568
+ ### Run
569
+
570
+ ```bash
571
+ # Correct engine: expect "no violations found"
572
+ python -m phantomkv fuzz --seeds 1000
573
+
574
+ # Prove the oracle catches a known bug class
575
+ python -m phantomkv fuzz --seeds 1000 --plant abort-free-race
576
+
577
+ # Replay one failing seed and print the story leading to the violation
578
+ python -m phantomkv replay --seed 0 --plant abort-free-race
579
+
580
+ # Shrink it to a minimal reproduction for a bug report
581
+ python -m phantomkv shrink --seed 0 --plant abort-free-race
582
+
583
+ # Heavier fault mix
584
+ python -m phantomkv fuzz --seeds 500 --remote-frac 0.9 --abort-frac 0.5
585
+ ```
586
+
587
+ ### CLI reference
588
+
589
+ | Command | Flags |
590
+ |---|---|
591
+ | `fuzz` | `--seeds N` · `--start S` · `--plant {abort-free-race, stale-prefix-cache}` · `--remote-frac F` · `--abort-frac F` · `--stop-first` |
592
+ | `replay` | `--seed N` · `--plant …` · `--context K` (trace lines before the violation) · `--full` |
593
+ | `shrink` | `--seed N` · `--plant …` |
594
+ | `vllm-fuzz` | `--seeds N` · `--start S` · `--profile NAME` · `--plant NAME` · `--check-determinism` · `-v` (needs the vLLM environment) |
595
+ | `vllm-shrink` | `--seed N` · `--profile NAME` · `--plant NAME` (needs the vLLM environment) |
596
+ | `vllm-diff` | `--seed N` · `--profile NAME` · `--before SEED` · `--context K` (first step where two runs of a seed differ) |
597
+ | `pd-fuzz` | `--seeds N` · `--fault {swap, silent-drop, duplicate}` · `--plant NAME` · `--check-determinism` · `-v` |
598
+ | `pd-shrink` | `--seed N` · `--fault …` · `--plant …` |
599
+
600
+ ---
601
+
602
+ ## Testing
603
+
604
+ ```bash
605
+ pytest -q
606
+ ```
607
+
608
+ | Test | Guarantees |
609
+ |---|---|
610
+ | `test_no_false_positives_on_correct_engine` | 300 seeds, all requests finish or abort, zero violations |
611
+ | `test_no_false_positives_under_heavy_pressure` | 12 blocks, 80% PD, 50% aborts: still zero violations |
612
+ | `test_prefix_cache_is_exercised` | the workload actually produces prefix-cache hits (no vacuous passes) |
613
+ | `test_planted_bug_is_caught[*]` | every planted bug class is detected |
614
+ | `test_abort_race_is_reported_as_cross_request_leak` | the race is classified correctly |
615
+ | `test_same_seed_same_run` / `test_different_seeds_differ` | determinism |
616
+ | `test_event_list_reproduces_seed_exactly` | seed → event list → identical run |
617
+ | `test_network_delay_independent_of_other_events` | removing events doesn't perturb other requests' timing |
618
+ | `test_shrunk_repro_is_small_and_still_fails[*]` | ≤ 15 events, same violation kind |
619
+ | `test_shrunk_repro_is_one_minimal[*]` | removing any single event makes the failure disappear |
620
+ | `test_ddmin_*` | the minimiser itself |
621
+
622
+ CI runs on every push:
623
+
624
+ * **`ci`**: the test suite on Python 3.10 and 3.12, plus a 1,000-seed clean fuzz of the simulator.
625
+ * **`real-vllm`**: installs vLLM's Linux CPU wheel (pinned) and runs
626
+ [`scripts/ci_vllm.sh`](https://github.com/sameer-sde/phantom-kv/blob/main/scripts/ci_vllm.sh). Unmodified vLLM must pass every check
627
+ (no false positives), and with a bug planted in vLLM's block pool the checks
628
+ must fail (no false negatives).
629
+
630
+ ---
631
+
632
+ ## Roadmap
633
+
634
+ | Phase | Status |
635
+ |---|---|
636
+ | 0. Exact oracle + deterministic simulator, validated on a toy engine | ✅ done |
637
+ | 1. Test-case shrinking: failing seed → minimal event trace | ✅ done |
638
+ | 2. Phantom as an out-of-tree **vLLM** model on the CPU backend, writing tags through vLLM's real block tables and slot mappings | ✅ done: runs inside real vLLM, planted bug caught (15/20 seeds), seeded step-by-step fuzzing + shrinking, real-vLLM checks in CI on every push |
639
+ | 3. More chaos on real vLLM: chunked prefill, preemption, block sizes | ✅ 4 profiles clean on unmodified vLLM, `stale-prefix-cache` caught in each, chunking/preemption proven by counters; refcount bugs shown to be caught by vLLM's own assertions |
640
+ | 4. Two-engine PD disaggregation with transfer faults (swap, drop, duplicate, abort mid-transfer) | ✅ v0.5.0: SGLang Oct-2026 class reproduced and caught; seeded PD fuzzing + shrinking; 200-seed campaign 0 false positives, every planted bug and fault caught; all in CI |
641
+ | 5. Bug-hunting campaigns on vLLM, SGLang, LMCache, llm-d; issues and fix PRs upstream | ✅ v0.6.0: vLLM's `SimpleCPUOffloadConnector` ([design](https://github.com/sameer-sde/phantom-kv/blob/main/docs/OFFLOAD_DESIGN.md)) and `OffloadingConnector` ([design](https://github.com/sameer-sde/phantom-kv/blob/main/docs/OFFLOAD2_DESIGN.md)), each with its own code unmodified: clean, planted faults caught, failed loads/stores recovered, in CI. LMCache (CPU-only server): planted bug caught 10/10, and two real open bugs found independently: a fast server restart is never noticed by vLLM (issue #4642) and aborting a request whose lookup finished before it was scheduled leaks read locks for 300 s (issue #5656, shrunk to 7 events). Phantom's A/B tests show both open fixes work (PR #6106: stores per server lifetime 19, 0, 0, 0 → 19, 20, 21, 19; PR #6540: leaked locks 2 and 3 → 0). Not covered: SGLang, llm-d, NIXL (need GPUs); no new issues filed, both bugs were already reported |
642
+
643
+ Releases: [v0.1.0 – v0.6.1](https://github.com/sameer-sde/phantom-kv/releases), one per finished phase ([CHANGELOG](https://github.com/sameer-sde/phantom-kv/blob/main/CHANGELOG.md)).
644
+
645
+ Details: [docs/ROADMAP.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/ROADMAP.md) · design: [docs/DESIGN.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/DESIGN.md) · vLLM integration: [docs/VLLM_INTEGRATION.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/VLLM_INTEGRATION.md) · daily progress: [docs/DEVLOG.md](https://github.com/sameer-sde/phantom-kv/blob/main/docs/DEVLOG.md)
646
+
647
+ ---
648
+
649
+ ## Project structure
650
+
651
+ ```
652
+ phantom-kv/
653
+ ├── phantomkv/
654
+ │ ├── provenance.py # Tag, chain hashes, phantom attention + LM head
655
+ │ ├── engine.py # ToyEngine: paged KV, prefix cache, preemption, PD, aborts, oracle
656
+ │ ├── sim.py # Event list, workload generator, deterministic simulator
657
+ │ ├── shrink.py # ddmin delta debugging → minimal reproductions
658
+ │ ├── cli.py # fuzz / replay / shrink commands
659
+ │ ├── tags.py # float32-exact tag encoding + reference outputs
660
+ │ ├── vllm_model.py # PhantomForCausalLM: writes/verifies tags in vLLM's KV cache
661
+ │ ├── vllm_plugin.py # registers it with vLLM (vllm.general_plugins)
662
+ │ ├── vllm_plants.py # known bug classes planted into vLLM at runtime
663
+ │ ├── vllm_fault_connector.py # ExampleConnector + planted transfer faults
664
+ │ ├── vllm_harness.py # seeded step-by-step driver for real vLLM
665
+ │ ├── vllm_pd.py # seeded two-engine (prefill/decode) fuzzing + shrinking
666
+ │ ├── vllm_offload.py # vLLM's CPU KV offloading on the CPU backend (copy engine only replaced)
667
+ │ ├── vllm_offload2.py # vLLM's OffloadingConnector, via its pluggable OffloadingSpec
668
+ │ ├── offload_map.py # block <-> chunk mapping for offloaded KV
669
+ │ └── __main__.py # python -m phantomkv
670
+ ├── tests/
671
+ │ ├── test_oracle.py # no false positives · planted bugs caught · determinism
672
+ │ ├── test_shrink.py # minimal, still-failing, 1-minimal reproductions
673
+ │ ├── test_tags.py # tags round-trip exactly through float32
674
+ │ └── test_pd_workload.py # seeded PD scenarios: delays, abort timings, fillers
675
+ ├── docs/
676
+ │ ├── DESIGN.md # oracle, determinism, mapping to vLLM
677
+ │ ├── VLLM_INTEGRATION.md # phase 2: code map + PhantomForCausalLM design
678
+ │ ├── PD_DESIGN.md # phase 4: disaggregated prefill/decode + fault plan
679
+ │ ├── OFFLOAD_DESIGN.md # phase 5 target 1: vLLM's CPU KV offloading
680
+ │ ├── OFFLOAD2_DESIGN.md # phase 5 target 2: vLLM's OffloadingConnector
681
+ │ ├── ROADMAP.md # phases and exit criteria
682
+ │ ├── blog/phantom.md # blog post: how Phantom works and what it found
683
+ │ └── DEVLOG.md # daily engineering log
684
+ ├── scripts/
685
+ │ ├── vllm_smoke.py # does vLLM (CPU) run on this machine?
686
+ │ ├── ci_vllm.sh # real-vLLM acceptance checks (CI + local)
687
+ │ ├── vllm_campaign.sh # every profile, unmodified + planted
688
+ │ ├── pd_campaign.sh # two engines: unmodified + every planted bug and fault
689
+ │ ├── offload_campaign.sh # OffloadingConnector: every profile, planted fault, storage failures
690
+ │ ├── lmcache_smoke.py # Phantom in vLLM with an LMCache server as the KV store
691
+ │ ├── lmcache_abort_probe.py # one aborted request at a chosen moment: LMCache locks left?
692
+ │ ├── lmcache_lock_shrink.py # shrink a seed to the fewest events that leak LMCache locks
693
+ │ ├── lmcache_lock_replay.py # replay the saved minimal lock-leak cases
694
+ │ ├── vllm_phantom_hello.py # does real vLLM load and run Phantom?
695
+ │ ├── vllm_phantom_check.py # end-to-end check: --pressure, --plant
696
+ │ └── vllm_pd_check.py # two engines (prefill + decode) via ExampleConnector
697
+ ├── CHANGELOG.md # what each release (phase) added
698
+ ├── .github/workflows/
699
+ │ ├── ci.yml # unit tests + simulator fuzz
700
+ │ ├── publish.yml # GitHub release -> PyPI (Trusted Publishing)
701
+ │ └── vllm.yml # Phantom inside real vLLM (pinned CPU wheel)
702
+ └── pyproject.toml
703
+ ```
704
+
705
+ ---
706
+
707
+ ## References
708
+
709
+ - Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). SOSP 2023. https://arxiv.org/abs/2309.06180
710
+ - Zheng, L., et al. (2024). SGLang: Efficient Execution of Structured Language Model Programs. https://arxiv.org/abs/2312.07104
711
+ - Zhou, J., et al. (2021). FoundationDB: A Distributed Unbundled Transactional Key Value Store. SIGMOD 2021. Simulation testing: https://apple.github.io/foundationdb/testing.html
712
+ - TigerBeetle VOPR simulator. https://github.com/tigerbeetle/tigerbeetle
713
+ - Kingsbury, K. Jepsen: distributed systems safety research. https://jepsen.io
714
+ - Zeller, A., & Hildebrandt, R. (2002). Simplifying and Isolating Failure-Inducing Input. IEEE Transactions on Software Engineering 28(2).
715
+ - Meng, R., et al. (2023). Greybox Fuzzing of Distributed Systems (Mallory). ACM CCS 2023. https://arxiv.org/abs/2305.02601
716
+ - GRIEF: Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing (2026). https://arxiv.org/abs/2605.11202
717
+ - Gu, Y., et al. (2026). Ekka: Automated Diagnosis of Silent Errors in LLM Inference. ICML 2026.
718
+ - LMSYS (2026). SGLang security update: improving request isolation. https://www.lmsys.org/blog/2026-10-26-cf-security/
719
+
720
+ ---
721
+
722
+ ## Author
723
+
724
+ **Sameer Ahmed** · [GitHub](https://github.com/sameer-sde)
725
+
726
+ Related work: [vLLM PR #54832](https://github.com/vllm-project/vllm/pull/54832) ·
727
+ [RaftKV](https://github.com/sameer-sde/raftKv) ·
728
+ [Sentinel](https://github.com/sameer-sde/sentinel) ·
729
+ [Vektr](https://github.com/sameer-sde/vektr)
730
+
731
+ License: [MIT](https://github.com/sameer-sde/phantom-kv/blob/main/LICENSE)