kvkernels 0.1.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,899 @@
1
+ """Cascade decode attention: shared KV read once per tree node, suffixes once per request.
2
+
3
+ Requests that share leading KV blocks form a prefix tree. Each node of the tree
4
+ (requests below it, the KV blocks they share beyond its parent) is one work
5
+ item, and each request's own remaining tokens are another. The cascade computes
6
+ attention over every work item, then merges each request's partial results:
7
+
8
+ work items partial results merge
9
+ rows of r0-r3 · node A blocks, range 1 ──▶ p0
10
+ rows of r0-r3 · node A blocks, range 2 ──▶ p1 r0: p0 p1 p2 p4 ──▶ out[r0]
11
+ rows of r0-r1 · node B blocks (in A) ──▶ p2
12
+ rows of r0 · r0's suffix blocks ──▶ p4
13
+ …
14
+
15
+ A request may have several new query tokens (query_tokens, the same for every
16
+ request of a plan): token t sees the request's keys up to length - query_tokens
17
+ + t, so the causal rule applies in its suffix item only. The cache may be fp8
18
+ (e4m3) with one K and one V scale per layer.
19
+
20
+ A plan, computed once per decode step and used by every layer, lists the tasks
21
+ (one tile of an item's rows over one token range), assigns each partial result
22
+ its slot in the workspace, and lists each (request, query head) row's partials
23
+ for the merge. Its buffers are allocated once for a maximum
24
+ batch and rewritten in place each step, so the launches can be captured in a
25
+ CUDA graph and replayed after later steps. Two ways to fill them:
26
+
27
+ plan_from_block_table, plan_from_slots on the GPU, from the index the engine
28
+ already passes to attention; groups are
29
+ found from shared leading KV blocks
30
+ update on the host, from groups the caller names
31
+
32
+ Requests are read through a permutation, so a group's requests need not be
33
+ consecutive in the batch; out[r] is always request r.
34
+
35
+ Three launches: prefix tasks, suffix tasks, merge. The two attention launches
36
+ run the same kernel with different tiles:
37
+
38
+ launch rows per tile warps stages programs per SM
39
+ prefix 128, 64 or 32 4 2 2 many requests' rows per item
40
+ (64, 32 or 16 below sm_80 at head_dim 128)
41
+ suffix 16 4 2 2 one request's GQA group per item
42
+
43
+ Both launches read keys in tiles of 32 tokens (block_n). A prefix task of at
44
+ most 64 or 32 rows runs in a 64- or 32-row tile (prefix_min_block_m, default a
45
+ quarter of the prefix tile; see prefix_attention). A group of at most
46
+ small_item_rows rows (four requests with a GQA group of 4) runs in the suffix
47
+ launch's tiles instead.
48
+
49
+ With 4 warps each warp holds 32 query rows of a 128-row tile. Compiled by
50
+ Triton 3.8.0 for sm_89 at head_dim 128 the prefix launch uses 255 registers per
51
+ thread, 24 bytes of stack and 48 KB of shared memory, so two programs fit in an
52
+ L4 SM's 100 KB; with 64-token key tiles it spilled 1 KB per thread. For sm_75
53
+ the same tile needs 80 KB, more than a T4 allows one program, so below compute
54
+ capability 8.0 the prefix tile at head_dim 128 is 64 rows (48 KB) and its
55
+ shorter tiles 32 and 16 rows. On an L4, against 8 warps, 3 stages and 64-token
56
+ key tiles, it took 0.83-0.88x the time at batch 64-128 with prefixes of 8,192
57
+ and more, and 0.97x over the probe grid.
58
+
59
+ Split-K cuts an item's tokens into s ranges, each its own task, so a launch
60
+ with few tasks still fills the GPU. Each launch is counted alone, since the
61
+ two run one after the other. With base programs before splitting and R
62
+ programs that run at once (prefix_programs_per_sm · SMs for the prefix launch,
63
+ programs_per_sm · SMs for the suffix launch), a launch takes about
64
+ ceil(base · s / R) rounds of ranges 1/s as long, so s is the value from 1 to
65
+ its cap that makes ceil(base · s / R) / s smallest, ties to the smaller:
66
+
67
+ base 32, R 116 s = 1: 1 round 1/1 = 1.00
68
+ s = 3: 1 round 1/3 = 0.33
69
+ s = 7: 2 rounds 2/7 = 0.29 ◀ chosen
70
+
71
+ A launch with base >= R fills a round already and is not split: more ranges
72
+ would add partial results for the merge to read. The cap is the smallest of
73
+ MAX_SPLITS, ceil(length / shortest range), with the shortest range
74
+ clamp(SPLIT_TOKENS_PER_ROW · tile rows, MIN_SPLIT_FLOOR, min_split_tokens),
75
+ and the larger of two bounds on the item's partial results: no larger than its
76
+ K and V (length / (SPLIT_TOKENS_PER_ROW · rows)), or within a third of L2
77
+ (part_budget_rows / rows).
78
+ On an L4 over 144 cases (batch 1-128, prefix 512-32,768, both models) the
79
+ round rule took 0.935x the time of the earlier one (ceil(2 SMs / base) ranges
80
+ of at least 512 tokens); the guard took a further 0.99x over 72 cases with
81
+ suffixes of 128-2,048 tokens.
82
+ """
83
+
84
+ from __future__ import annotations
85
+
86
+ import functools
87
+ import math
88
+ import os
89
+ from collections.abc import Sequence
90
+ from dataclasses import dataclass
91
+
92
+ import torch
93
+
94
+ from kvkernels.ops._registry import registered_op
95
+ from kvkernels.ops.cascade_plan import MAX_PLAN_REQUESTS, PlanGraphs, plan_on_device
96
+ from kvkernels.ops.merge_states import merge_ragged
97
+ from kvkernels.ops.prefix_attention import attend
98
+
99
+ MAX_SPLITS = 16
100
+ MAX_QUERY_TOKENS = 16 # more new tokens per request is prefill
101
+ MAX_LEVELS = 8 # rounds of tree finding a plan may ask for
102
+ DEFAULT_LEVELS = 4
103
+ # Split-K ranges are at least SPLIT_TOKENS_PER_ROW tokens per query row of the
104
+ # item's tile, and at least MIN_SPLIT_FLOOR tokens, up to min_split_tokens.
105
+ SPLIT_TOKENS_PER_ROW = 2
106
+ # An item's split-K partial results may take at most this share of L2: on an L4
107
+ # (48 MB), 4,096-token ranges for a 1,024-row item spilled 75 MB of partial
108
+ # results to device memory and took 1.5x the time of 2 ranges.
109
+ PARTIAL_L2_SHARE = 3
110
+ DEFAULT_L2_BYTES = 48 << 20 # an L4's L2, for plans without a CUDA device (the interpreter)
111
+ MIN_SPLIT_FLOOR = 128
112
+ # Shortest shared prefix that forms a group. On an L4 (48 MB of L2), grouping
113
+ # requests that share 16-256 tokens took 0.98-1.07x the time of reading the
114
+ # shared tokens with each request; from 512 tokens it took 0.61-1.03x.
115
+ MIN_SHARED_TOKENS = 512
116
+ TASK_WAVES = 8 # attention programs per KV head: TASK_WAVES · programs_per_sm · SMs / KV heads
117
+ MERGE_PROGRAMS_PER_SM = 32 # merge programs are small; many fit on one SM
118
+ # Warps and pipeline stages of each attention launch; see the module docstring.
119
+ PREFIX_WARPS, PREFIX_STAGES = 4, 2
120
+ SUFFIX_WARPS, SUFFIX_STAGES = 4, 2
121
+ HEAD_DIMS = (64, 128)
122
+ _IN_DTYPES = (torch.float16, torch.bfloat16)
123
+
124
+
125
+ @dataclass(frozen=True)
126
+ class PrefixGroup:
127
+ """Requests that share tokens start … start + length - 1, held in `blocks`.
128
+
129
+ requests are consecutive request indices. Groups may nest: a group inside
130
+ another covers a subset of its requests and starts where it ends. start is
131
+ a multiple of the block size; length may end inside the last block.
132
+ """
133
+
134
+ requests: range
135
+ blocks: Sequence[int]
136
+ length: int
137
+ start: int = 0
138
+
139
+
140
+ def _interpreting() -> bool:
141
+ return os.environ.get("TRITON_INTERPRET") == "1"
142
+
143
+
144
+ # Triton has an e4m3 type only from compute capability 8.9 (L4, RTX 4090, H100).
145
+ FP8_MIN_CAPABILITY = (8, 9)
146
+
147
+
148
+ # Compiled by Triton 3.8.0 at head_dim 128, one prefix program with a 128-row
149
+ # tile needs 80 KB of shared memory for sm_75 and 48 KB for sm_89. A T4 (sm_75)
150
+ # allows 64 KB per program, so below SHORT_PREFIX_TILE_BELOW the prefix tile at
151
+ # head_dim 128 is SHORT_PREFIX_TILE rows (48 KB for sm_75).
152
+ SHORT_PREFIX_TILE_BELOW = (8, 0)
153
+ SHORT_PREFIX_TILE = 64
154
+
155
+
156
+ @functools.cache
157
+ def _capability(index: int) -> tuple[int, int]:
158
+ major, minor = torch.cuda.get_device_capability(index)
159
+ return major, minor
160
+
161
+
162
+ def _device_capability(device: torch.device) -> tuple[int, int]:
163
+ index = device.index if device.index is not None else torch.cuda.current_device()
164
+ return _capability(index)
165
+
166
+
167
+ def fp8_kv_supported(device: torch.device) -> bool:
168
+ """Whether the cascade can read an fp8 (e4m3) cache on this device."""
169
+ if device.type != "cuda":
170
+ return True # the Triton interpreter
171
+ return _device_capability(device) >= FP8_MIN_CAPABILITY
172
+
173
+
174
+ def default_prefix_block_m(device: torch.device, head_dim: int) -> int:
175
+ """Rows per prefix tile: 128, or 64 where 128 rows do not fit in shared memory."""
176
+ if (
177
+ device.type == "cuda"
178
+ and head_dim == 128
179
+ and _device_capability(device) < SHORT_PREFIX_TILE_BELOW
180
+ ):
181
+ return SHORT_PREFIX_TILE
182
+ return 128
183
+
184
+
185
+ def same_device(a: torch.device, b: torch.device) -> bool:
186
+ """Whether two devices are the same; a CUDA device without an index is the current one."""
187
+ if a.type != b.type:
188
+ return False
189
+ if a.type != "cuda" or a.index == b.index:
190
+ return True
191
+ current = torch.cuda.current_device()
192
+ return (a.index if a.index is not None else current) == (
193
+ b.index if b.index is not None else current
194
+ )
195
+
196
+
197
+ class CascadePlan:
198
+ """Work items, split-K ranges and merge lists for one decode step, in fixed device buffers.
199
+
200
+ Build one plan per model (one shape of attention) and maximum batch, fill it
201
+ once per decode step with plan_from_block_table, plan_from_slots or update,
202
+ and pass it to cascade_decode_attention for every layer.
203
+
204
+ Sizes, fixed for the plan's life:
205
+ max_requests: the largest batch; at most 4,096 for planning on the GPU.
206
+ max_groups: the most prefix-tree nodes a step may have. Planning on the
207
+ GPU folds nodes past it into their children, which read those
208
+ tokens themselves.
209
+ max_blocks: the most KV block ids a host plan (update) may list.
210
+ num_q_heads, num_kv_heads, head_dim: the attention shape; head_dim 64
211
+ or 128, and at most 16 query heads per KV head.
212
+ block_size, num_blocks: tokens per KV block and KV blocks in the cache.
213
+ device: where the plan's buffers live.
214
+ max_tasks, max_parts: workspace sizes; the defaults fit any plan.
215
+ Behaviour:
216
+ min_shared_tokens: the fewest tokens a group of requests must share to
217
+ be read once for all of them (default 512).
218
+ batch_invariant: no groups, and split-K by each request's length alone,
219
+ so a request's output does not depend on the rest of its batch.
220
+ query_tokens: new tokens per request, 1 to 16 (more than 1 for
221
+ speculative decoding's verification step).
222
+ max_levels: levels of shared prefix planning on the GPU looks for, 1
223
+ to 8 (default 4).
224
+ The other arguments tune tiles and split-K for a GPU; their defaults come
225
+ from L4 measurements, and a release may change or remove them.
226
+ """
227
+
228
+ def __init__(
229
+ self,
230
+ *,
231
+ max_requests: int,
232
+ max_groups: int,
233
+ max_blocks: int,
234
+ num_q_heads: int,
235
+ num_kv_heads: int,
236
+ head_dim: int,
237
+ block_size: int,
238
+ num_blocks: int,
239
+ device: torch.device,
240
+ max_tasks: int | None = None,
241
+ max_parts: int | None = None,
242
+ num_sms: int | None = None,
243
+ programs_per_sm: int = 2,
244
+ prefix_programs_per_sm: int = 2,
245
+ min_split_tokens: int = 512,
246
+ prefix_block_m: int | None = None,
247
+ prefix_min_block_m: int | None = None,
248
+ suffix_block_m: int = 16,
249
+ block_n: int | None = None,
250
+ min_shared_tokens: int | None = None,
251
+ small_item_rows: int = 16,
252
+ batch_invariant: bool = False,
253
+ query_tokens: int = 1,
254
+ max_levels: int = DEFAULT_LEVELS,
255
+ ) -> None:
256
+ if num_q_heads % num_kv_heads:
257
+ raise ValueError(f"num_q_heads {num_q_heads} is not a multiple of {num_kv_heads}")
258
+ self.group = num_q_heads // num_kv_heads
259
+ if not 1 <= self.group <= suffix_block_m:
260
+ raise ValueError(f"GQA group {self.group} must be 1 to {suffix_block_m}")
261
+ if not 1 <= query_tokens <= MAX_QUERY_TOKENS:
262
+ raise ValueError(f"query_tokens {query_tokens} must be 1 to {MAX_QUERY_TOKENS}")
263
+ if not 1 <= max_levels <= MAX_LEVELS:
264
+ raise ValueError(f"max_levels {max_levels} must be 1 to {MAX_LEVELS}")
265
+ if head_dim not in HEAD_DIMS:
266
+ raise ValueError(f"head_dim must be one of {HEAD_DIMS}, got {head_dim}")
267
+ prefix_block_m = prefix_block_m or default_prefix_block_m(device, head_dim)
268
+ if prefix_block_m == suffix_block_m:
269
+ raise ValueError("prefix and suffix row tiles must differ in height")
270
+ prefix_min_block_m = prefix_min_block_m or max(16, prefix_block_m // 4)
271
+ if prefix_min_block_m not in (prefix_block_m, prefix_block_m // 2, prefix_block_m // 4):
272
+ raise ValueError(
273
+ f"prefix_min_block_m {prefix_min_block_m}: must be prefix_block_m, /2 or /4"
274
+ )
275
+ if prefix_min_block_m < 16:
276
+ raise ValueError(f"prefix_min_block_m {prefix_min_block_m} is below 16")
277
+ self.max_requests = max_requests
278
+ self.max_groups = max_groups
279
+ self.max_blocks = max_blocks
280
+ self.num_q_heads = num_q_heads
281
+ self.num_kv_heads = num_kv_heads
282
+ self.head_dim = head_dim
283
+ self.block_size = block_size
284
+ self.num_blocks = num_blocks
285
+ self.device = device
286
+ self.max_splits = MAX_SPLITS
287
+ self.min_shared_tokens = (
288
+ MIN_SHARED_TOKENS if min_shared_tokens is None else min_shared_tokens
289
+ )
290
+ self.small_item_rows = small_item_rows
291
+ self.query_tokens = query_tokens
292
+ self.max_levels = max_levels
293
+ # A node's own tokens must pay for the partial results of its rows, which
294
+ # grow with the query tokens per request.
295
+ self.min_shared_blocks = max(
296
+ 1, math.ceil(self.min_shared_tokens * query_tokens / block_size)
297
+ )
298
+ self.batch_invariant = batch_invariant
299
+ rows_per_request = self.group * query_tokens
300
+ suffix_tiles = math.ceil(rows_per_request / suffix_block_m)
301
+ self.max_tasks = max_tasks or (64 * max_groups + MAX_SPLITS * max_requests * suffix_tiles)
302
+ self.max_parts = max_parts or (2 * max_requests * query_tokens * num_q_heads * MAX_SPLITS)
303
+ # Without split-K every plan fits: one part per row for each node item on
304
+ # its path and one for its suffix. A batch-invariant plan must fit with
305
+ # split-K, which it never turns off.
306
+ splits = MAX_SPLITS if batch_invariant else 1
307
+ levels = 0 if batch_invariant else max_levels
308
+ need_parts = (levels + 1) * max_requests * query_tokens * num_q_heads * splits
309
+ small_groups = 0 if batch_invariant else min(max_groups, max_requests // 2)
310
+ need_tasks = max(
311
+ max_requests * suffix_tiles * splits
312
+ + small_groups * math.ceil(small_item_rows / suffix_block_m),
313
+ 0
314
+ if batch_invariant
315
+ else max_groups + math.ceil(levels * max_requests * rows_per_request / prefix_block_m),
316
+ )
317
+ if self.max_parts < need_parts:
318
+ raise ValueError(f"max_parts {self.max_parts} is below {need_parts}")
319
+ if self.max_tasks < need_tasks:
320
+ raise ValueError(f"max_tasks {self.max_tasks} is below {need_tasks}")
321
+ if num_sms is None:
322
+ num_sms = (
323
+ torch.cuda.get_device_properties(device).multi_processor_count
324
+ if device.type == "cuda"
325
+ else 8
326
+ )
327
+ self.num_sms = num_sms
328
+ self.programs_per_sm = programs_per_sm
329
+ self.prefix_programs_per_sm = prefix_programs_per_sm
330
+ self.min_split_tokens = min_split_tokens
331
+ self.split_tokens_per_row = SPLIT_TOKENS_PER_ROW
332
+ self.min_split_floor = MIN_SPLIT_FLOOR
333
+ # Split-K keeps each item's partial results within a share of L2: rows of
334
+ # an item times its ranges at most part_budget_rows. Each row costs
335
+ # kv_heads · head_dim · 8 bytes per range (fp32 output, written and read).
336
+ l2_bytes = (
337
+ torch.cuda.get_device_properties(device).L2_cache_size
338
+ if device.type == "cuda"
339
+ else DEFAULT_L2_BYTES
340
+ )
341
+ self.part_budget_rows = max(
342
+ 1, l2_bytes // PARTIAL_L2_SHARE // (num_kv_heads * head_dim * 8)
343
+ )
344
+ self.prefix_block_m = prefix_block_m
345
+ self.prefix_min_block_m = prefix_min_block_m
346
+ self.suffix_block_m = suffix_block_m
347
+ self.block_n = block_n or (16 if _interpreting() else 32)
348
+ self.max_rows = max_requests * query_tokens * num_q_heads
349
+ # Fixed grids that loop over the batch's work, a few times larger than
350
+ # what the GPU runs at once so that uneven tasks still spread out.
351
+ resident = programs_per_sm * num_sms
352
+ self.task_programs = min(self.max_tasks, math.ceil(TASK_WAVES * resident / num_kv_heads))
353
+ self.merge_programs = min(self.max_rows, MERGE_PROGRAMS_PER_SM * num_sms)
354
+
355
+ def ints(n: int) -> torch.Tensor:
356
+ return torch.zeros(max(n, 1), dtype=torch.int32, device=device)
357
+
358
+ items = max_groups + max_requests
359
+ self.blocks = ints(max_blocks)
360
+ self.perm = ints(max_requests) # row i of the sorted batch is request perm[i]
361
+ # Device planning, per round and sorted row: the first row of the row's
362
+ # node (-1: none) and the node's depth in KV blocks.
363
+ self.tree = torch.zeros(
364
+ (2, max_levels, max(max_requests, 1)), dtype=torch.int32, device=device
365
+ )
366
+ self.item_req_start, self.item_num_reqs, self.item_block_start = (
367
+ ints(items),
368
+ ints(items),
369
+ ints(items),
370
+ )
371
+ # A suffix item's token t sees its tokens up to key_limit + t; node items see all.
372
+ self.item_key_limit = ints(items)
373
+ # Device planning: length, ranges, span, task start, part start, merge offset per item.
374
+ self.item_info = torch.zeros((6, items), dtype=torch.int32, device=device)
375
+ # Item, row start, token start, token end, part base; prefix tasks, then suffix tasks.
376
+ self.tasks = torch.zeros((2, 5, self.max_tasks), dtype=torch.int32, device=device)
377
+ self.prefix_tasks = tuple(self.tasks[0])
378
+ self.suffix_tasks = tuple(self.tasks[1])
379
+ self.counts = ints(6) # prefix tasks, suffix tasks, merge rows, node items, items, rounds
380
+ self.planned_requests = ints(1) # device planning: the batch size, read by the kernels
381
+ self.graphs = PlanGraphs() # device planning: recorded launches, replayed each step
382
+ self.merge_start = ints(self.max_rows)
383
+ self.merge_end = ints(self.max_rows)
384
+ self.merge_indices = ints(self.max_parts)
385
+ self.part_o = torch.zeros((self.max_parts, head_dim), dtype=torch.float32, device=device)
386
+ self.part_lse = torch.zeros(self.max_parts, dtype=torch.float32, device=device)
387
+ self.num_requests = 0
388
+ # Whether the prefix launch can have tasks. An empty launch still costs
389
+ # about 2 us on an L4 (batch 1, Llama 3.1 8B: 19.7 us with it, per call
390
+ # replayed from a CUDA graph), so it is left out when no node can exist.
391
+ self.prefix_launch = True
392
+ self.index = self.blocks # what item_block_start points into
393
+ self.per_token = False # index holds one slot per token (a slot list)
394
+
395
+ # ----------------------------------------------------------------------------------
396
+
397
+ def _check(
398
+ self,
399
+ groups: Sequence[PrefixGroup],
400
+ suffix_blocks: Sequence[Sequence[int]],
401
+ suffix_lens: Sequence[int],
402
+ ) -> list[int]:
403
+ """Check the groups form a tree; return their order, parents before children."""
404
+ num_req = len(suffix_lens)
405
+ if num_req > self.max_requests:
406
+ raise ValueError(f"{num_req} requests exceed max_requests {self.max_requests}")
407
+ if len(suffix_blocks) != num_req:
408
+ raise ValueError(f"{len(suffix_blocks)} suffix block lists for {num_req} requests")
409
+ if len(groups) > self.max_groups:
410
+ raise ValueError(f"{len(groups)} groups exceed max_groups {self.max_groups}")
411
+ for gi, g in enumerate(groups):
412
+ if g.requests.step != 1 or g.requests.start < 0 or g.requests.stop > num_req:
413
+ raise ValueError(f"group {gi}: requests {g.requests} outside 0 … {num_req - 1}")
414
+ if len(g.requests) == 0:
415
+ raise ValueError(f"group {gi} has no requests")
416
+ if g.start < 0 or g.start % self.block_size:
417
+ raise ValueError(f"group {gi}: start {g.start} is not a multiple of the block size")
418
+ # Each request's groups, outermost first, must hold tokens 0 … end with no gap.
419
+ path: list[list[int]] = [[] for _ in range(num_req)]
420
+ for gi in sorted(range(len(groups)), key=lambda gi: groups[gi].start):
421
+ for r in groups[gi].requests:
422
+ end = groups[path[r][-1]].start + groups[path[r][-1]].length if path[r] else 0
423
+ if groups[gi].start < end:
424
+ raise ValueError(
425
+ f"groups {path[r][-1]} and {gi} overlap at request {r}, token "
426
+ f"{groups[gi].start}"
427
+ )
428
+ if groups[gi].start > end:
429
+ raise ValueError(
430
+ f"group {gi} starts at token {groups[gi].start}, but request {r}'s "
431
+ f"groups end at {end}: a gap"
432
+ )
433
+ if path[r]:
434
+ outer = groups[path[r][-1]].requests
435
+ if not (
436
+ outer.start <= groups[gi].requests.start
437
+ and groups[gi].requests.stop <= outer.stop
438
+ ):
439
+ raise ValueError(f"group {gi} is not inside group {path[r][-1]}")
440
+ path[r].append(gi)
441
+ depth = [0] * len(groups)
442
+ for p in path:
443
+ for d, gi in enumerate(p):
444
+ depth[gi] = d
445
+ for r, p in enumerate(path):
446
+ if p and self.query_tokens > 1 and suffix_lens[r] < self.query_tokens - 1:
447
+ raise ValueError(
448
+ f"request {r}: groups hold keys after its first new query token "
449
+ f"(suffix {suffix_lens[r]} tokens, query_tokens {self.query_tokens})"
450
+ )
451
+ lists = [(f"group {gi}", g.blocks, g.length) for gi, g in enumerate(groups)]
452
+ lists += [
453
+ (f"request {r}", b, n)
454
+ for r, (b, n) in enumerate(zip(suffix_blocks, suffix_lens, strict=True))
455
+ ]
456
+ for name, blocks, length in lists:
457
+ if length < 0 or len(blocks) * self.block_size < length:
458
+ raise ValueError(f"{name}: {length} tokens do not fit in {len(blocks)} blocks")
459
+ for b in blocks:
460
+ if not 0 <= b < self.num_blocks:
461
+ raise ValueError(f"{name}: block {b} is outside 0 … {self.num_blocks - 1}")
462
+ return sorted(range(len(groups)), key=lambda gi: (depth[gi], groups[gi].requests.start))
463
+
464
+ def min_range_tokens(self, rows: int) -> int:
465
+ """Shortest split-K range for an item whose tile holds rows query rows."""
466
+ by_rows = max(self.min_split_floor, self.split_tokens_per_row * rows)
467
+ return min(self.min_split_tokens, by_rows)
468
+
469
+ def target_programs(self, block_m: int) -> int:
470
+ """Programs of one launch that run at once: what split-K aims to fill."""
471
+ per_sm = (
472
+ self.prefix_programs_per_sm if block_m == self.prefix_block_m else self.programs_per_sm
473
+ )
474
+ return per_sm * self.num_sms
475
+
476
+ def _splits(self, length: int, base_programs: int, rows: int, block_m: int) -> int:
477
+ if self.batch_invariant:
478
+ return max(1, min(MAX_SPLITS, math.ceil(length / self.min_split_tokens)))
479
+ resident = self.target_programs(block_m)
480
+ if base_programs >= resident:
481
+ return 1 # the launch fills a round already; more ranges add merge work
482
+ # The ranges whose programs fill the launch's rounds best: time is about
483
+ # (rounds of programs) x (tokens per range) = ceil(base · s / R) / s.
484
+ # Each range also writes, and the merge reads, an fp32 partial result per
485
+ # row: 8 bytes per row and head dimension, against 4 per token of K and V.
486
+ # s ranges are allowed when the item's partial results are no larger than
487
+ # its K and V (ranges of at least 2 tokens per row), or fit in a share of
488
+ # L2 (s · rows at most part_budget_rows).
489
+ rows = max(rows, 1)
490
+ by_kv = length // max(self.split_tokens_per_row * rows, 1)
491
+ by_bytes = max(by_kv, self.part_budget_rows // rows)
492
+ cap = math.ceil(length / self.min_range_tokens(min(rows, block_m)))
493
+ cap = max(1, min(MAX_SPLITS, cap, by_bytes))
494
+ base = max(base_programs, 1)
495
+ best_s, best_rounds = 1, math.ceil(base / resident)
496
+ for s in range(2, cap + 1):
497
+ rounds = math.ceil(base * s / resident)
498
+ if rounds * best_s < best_rounds * s:
499
+ best_s, best_rounds = s, rounds
500
+ return best_s
501
+
502
+ def update(
503
+ self,
504
+ *,
505
+ groups: Sequence[PrefixGroup],
506
+ suffix_blocks: Sequence[Sequence[int]],
507
+ suffix_lens: Sequence[int],
508
+ ) -> None:
509
+ """Plan one decode step on the host. Requests are numbered 0 … len(suffix_lens) - 1."""
510
+ if self.batch_invariant and groups:
511
+ raise ValueError("a batch-invariant plan makes no groups")
512
+ order = self._check(groups, suffix_blocks, suffix_lens)
513
+ num_req = len(suffix_lens)
514
+ g, hkv, hq, nt = self.group, self.num_kv_heads, self.num_q_heads, self.query_tokens
515
+ per_req = g * nt # item rows per request
516
+ no_limit = 1 << 30 # node items: every query token sees all keys
517
+
518
+ # Work items, parents before children, then suffixes: (first request,
519
+ # requests, block list, length, row-tile height, key limit).
520
+ items: list[tuple[int, int, Sequence[int], int, int, int]] = []
521
+ for grp in (groups[gi] for gi in order):
522
+ if grp.length > 0:
523
+ items.append(
524
+ (
525
+ grp.requests.start,
526
+ len(grp.requests),
527
+ grp.blocks,
528
+ grp.length,
529
+ self.suffix_block_m
530
+ if len(grp.requests) * per_req <= self.small_item_rows
531
+ else self.prefix_block_m,
532
+ no_limit,
533
+ )
534
+ )
535
+ for r, (blocks, length) in enumerate(zip(suffix_blocks, suffix_lens, strict=True)):
536
+ if length > 0:
537
+ items.append((r, 1, blocks, length, self.suffix_block_m, length - nt))
538
+ # Programs of each launch before splitting; the launches run one after the other.
539
+ base_programs = {self.prefix_block_m: 0, self.suffix_block_m: 0}
540
+ for _, n, _, _, bm, _ in items:
541
+ base_programs[bm] += math.ceil(n * per_req / bm) * hkv
542
+
543
+ flat: list[int] = []
544
+ req_start, num_reqs, block_start = [], [], []
545
+ tasks: dict[int, list[list[int]]] = {
546
+ self.prefix_block_m: [[], [], [], [], []],
547
+ self.suffix_block_m: [[], [], [], [], []],
548
+ }
549
+ key_limit: list[int] = []
550
+ parts_of_row: list[list[int]] = [[] for _ in range(num_req * nt * hq)]
551
+ next_part = 0
552
+ for index, (first_req, n_req, blocks, length, bm, limit) in enumerate(items):
553
+ req_start.append(first_req)
554
+ num_reqs.append(n_req)
555
+ block_start.append(len(flat))
556
+ key_limit.append(limit)
557
+ flat.extend(blocks[: math.ceil(length / self.block_size)])
558
+ splits = self._splits(length, base_programs[bm], n_req * per_req, bm)
559
+ span = math.ceil(math.ceil(length / splits) / self.block_n) * self.block_n
560
+ ranges = [(s, min(s + span, length)) for s in range(0, length, span)]
561
+ rows = n_req * per_req
562
+ for row_start in range(0, rows, bm):
563
+ tile_rows = min(bm, rows - row_start)
564
+ for tok_start, tok_end in ranges:
565
+ table = tasks[bm]
566
+ for column, value in zip(
567
+ table, (index, row_start, tok_start, tok_end, next_part), strict=True
568
+ ):
569
+ column.append(value)
570
+ for kv in range(hkv):
571
+ for i in range(tile_rows):
572
+ item_row = row_start + i
573
+ request = first_req + item_row // per_req
574
+ token = (item_row // g) % nt
575
+ head = kv * g + item_row % g
576
+ parts_of_row[(request * nt + token) * hq + head].append(
577
+ next_part + kv * tile_rows + i
578
+ )
579
+ next_part += hkv * tile_rows
580
+
581
+ prefix_tasks, suffix_tasks = tasks[self.prefix_block_m], tasks[self.suffix_block_m]
582
+ if len(flat) > self.max_blocks:
583
+ raise ValueError(f"{len(flat)} blocks in the batch exceed max_blocks {self.max_blocks}")
584
+ if max(len(prefix_tasks[0]), len(suffix_tasks[0])) > self.max_tasks:
585
+ raise ValueError(f"the batch needs more than max_tasks {self.max_tasks} tasks")
586
+ if next_part > self.max_parts:
587
+ raise ValueError(
588
+ f"the batch needs {next_part} partial results, above max_parts {self.max_parts}"
589
+ )
590
+
591
+ starts: list[int] = []
592
+ indices: list[int] = []
593
+ for parts in parts_of_row:
594
+ starts.append(len(indices))
595
+ indices.extend(parts)
596
+
597
+ def put(buffer: torch.Tensor, values: Sequence[int]) -> None:
598
+ if values:
599
+ buffer[: len(values)].copy_(torch.tensor(values, dtype=torch.int32))
600
+
601
+ put(self.blocks, flat)
602
+ put(self.perm, range(num_req))
603
+ put(self.item_req_start, req_start)
604
+ put(self.item_num_reqs, num_reqs)
605
+ put(self.item_block_start, block_start)
606
+ put(self.item_key_limit, key_limit)
607
+ for buffers, columns in (
608
+ (self.prefix_tasks, prefix_tasks),
609
+ (self.suffix_tasks, suffix_tasks),
610
+ ):
611
+ for buffer, column in zip(buffers, columns, strict=True):
612
+ put(buffer, column)
613
+ put(self.merge_start, starts)
614
+ put(self.merge_end, [*starts[1:], len(indices)])
615
+ put(self.merge_indices, indices)
616
+ put(self.counts, [len(prefix_tasks[0]), len(suffix_tasks[0]), num_req * nt * hq])
617
+ self.num_requests = num_req
618
+ # As for device planning: by the batch size alone, so a CUDA graph captured
619
+ # before a later update still holds the prefix launch the update needs.
620
+ self.prefix_launch = num_req >= 2 and not self.batch_invariant
621
+ self.index, self.per_token = self.blocks, False
622
+
623
+ def plan_from_block_table(self, block_table: torch.Tensor, seq_lens: torch.Tensor) -> None:
624
+ """Plan one decode step on the GPU from a vLLM-style index, without a host sync.
625
+
626
+ block_table: [requests, width] int32, row r holding request r's KV block ids
627
+ (entries past its length are never read). seq_lens: [requests] int32; a
628
+ request of length 0 is a padded row and gets out = 0. Requests whose rows
629
+ start with the same KV blocks form a group when they share at least
630
+ min_shared_tokens; the group covers the whole KV blocks every member
631
+ shares. The attention reads block_table in place until the next plan, so
632
+ keep it unchanged until then.
633
+ """
634
+ if block_table.dim() != 2 or block_table.dtype != torch.int32:
635
+ raise ValueError(f"block_table must be 2-D int32, got {block_table.dtype}")
636
+ if block_table.shape[1] > 0 and block_table.stride(1) != 1:
637
+ raise ValueError("block_table rows must be contiguous")
638
+ self._plan_on_device(block_table, None, seq_lens, block_table.stride(0))
639
+
640
+ def plan_from_slots(
641
+ self, slot_indptr: torch.Tensor, slots: torch.Tensor, seq_lens: torch.Tensor
642
+ ) -> None:
643
+ """Plan one decode step on the GPU from an SGLang-style index, without a host sync.
644
+
645
+ slots: one slot per token, int32 or int64, request r's tokens at
646
+ slots[slot_indptr[r] : slot_indptr[r] + seq_lens[r]]; slot s is token
647
+ s % block_size of page s // block_size. slot_indptr: [requests + 1] or
648
+ [requests] int32. Groups are found as in plan_from_block_table, comparing
649
+ each page's first slot; the attention reads slots in place.
650
+ """
651
+ if slots.dim() != 1 or slots.dtype not in (torch.int32, torch.int64):
652
+ raise ValueError(f"slots must be 1-D int32 or int64, got {slots.dtype}")
653
+ if slot_indptr.dim() != 1 or slot_indptr.dtype != torch.int32:
654
+ raise ValueError(f"slot_indptr must be 1-D int32, got {slot_indptr.dtype}")
655
+ if slot_indptr.shape[0] not in (seq_lens.shape[0], seq_lens.shape[0] + 1):
656
+ raise ValueError(
657
+ f"slot_indptr has {slot_indptr.shape[0]} entries for {seq_lens.shape[0]} requests"
658
+ )
659
+ self._plan_on_device(slots, slot_indptr, seq_lens, 0)
660
+
661
+ def _plan_on_device(
662
+ self,
663
+ index: torch.Tensor,
664
+ starts: torch.Tensor | None,
665
+ seq_lens: torch.Tensor,
666
+ stride_r: int,
667
+ ) -> None:
668
+ if seq_lens.dim() != 1 or seq_lens.dtype != torch.int32:
669
+ raise ValueError(f"seq_lens must be 1-D int32, got {seq_lens.dtype}")
670
+ num_req = seq_lens.shape[0]
671
+ if num_req > self.max_requests:
672
+ raise ValueError(f"{num_req} requests exceed max_requests {self.max_requests}")
673
+ if starts is None and index.shape[0] != num_req:
674
+ raise ValueError(f"block_table has {index.shape[0]} rows for {num_req} requests")
675
+ if self.max_requests > MAX_PLAN_REQUESTS:
676
+ raise ValueError(f"planning on the GPU supports up to {MAX_PLAN_REQUESTS} requests")
677
+ for t in (index, seq_lens) + ((starts,) if starts is not None else ()):
678
+ if not same_device(t.device, self.device):
679
+ raise ValueError(f"index tensors must be on {self.device}, got {t.device}")
680
+ plan_on_device(self, index=index, starts=starts, seq_lens=seq_lens, stride_r=stride_r)
681
+ self.num_requests = num_req
682
+ # A node needs two requests. The decision rests on the batch size alone,
683
+ # which a captured CUDA graph fixes.
684
+ self.prefix_launch = num_req >= 2 and not self.batch_invariant
685
+ self.index, self.per_token = index, starts is not None
686
+
687
+ # ----------------------------------------------------------------------------------
688
+
689
+ def tensors(self) -> list[torch.Tensor]:
690
+ """Read-only plan tensors, in the order the registered op expects."""
691
+ return [
692
+ self.index,
693
+ self.perm,
694
+ self.item_req_start,
695
+ self.item_num_reqs,
696
+ self.item_block_start,
697
+ self.item_key_limit,
698
+ *self.prefix_tasks,
699
+ *self.suffix_tasks,
700
+ self.counts,
701
+ self.merge_start,
702
+ self.merge_end,
703
+ self.merge_indices,
704
+ ]
705
+
706
+ def settings(self) -> list[int]:
707
+ return [
708
+ self.block_size,
709
+ self.group,
710
+ self.max_tasks,
711
+ self.max_rows,
712
+ self.prefix_block_m,
713
+ self.suffix_block_m,
714
+ self.block_n,
715
+ int(self.per_token),
716
+ self.task_programs,
717
+ self.merge_programs,
718
+ self.prefix_min_block_m,
719
+ self.query_tokens,
720
+ int(self.prefix_launch),
721
+ ]
722
+
723
+
724
+ @registered_op("kvkernels::cascade_decode_attention", mutates_args=("workspace", "out", "lse_out"))
725
+ def _cascade_op(
726
+ q: torch.Tensor,
727
+ k: torch.Tensor,
728
+ v: torch.Tensor,
729
+ plan_tensors: list[torch.Tensor],
730
+ workspace: list[torch.Tensor],
731
+ out: torch.Tensor,
732
+ lse_out: torch.Tensor | None,
733
+ sm_scale: float,
734
+ settings: list[int],
735
+ kv_scales: list[torch.Tensor],
736
+ ) -> None:
737
+ index, perm, req_start, num_reqs, block_start, key_limit = plan_tensors[:6]
738
+ prefix_tasks = tuple(plan_tensors[6:11])
739
+ suffix_tasks = tuple(plan_tensors[11:16])
740
+ counts, merge_start, merge_end, indices = plan_tensors[16:20]
741
+ part_o, part_lse = workspace
742
+ block_size, group, max_tasks, max_rows, prefix_bm, suffix_bm, block_n, per_token = settings[:8]
743
+ task_programs, merge_programs, prefix_min_bm, query_tokens, prefix_launch = settings[8:13]
744
+ common = {
745
+ "index": index,
746
+ "perm": perm,
747
+ "per_token": bool(per_token),
748
+ "item_req_start": req_start,
749
+ "item_num_reqs": num_reqs,
750
+ "item_block_start": block_start,
751
+ "item_key_limit": key_limit,
752
+ "query_tokens": query_tokens,
753
+ "part_o": part_o,
754
+ "part_lse": part_lse,
755
+ "sm_scale": sm_scale,
756
+ "block_size": block_size,
757
+ "group": group,
758
+ "max_tasks": max_tasks,
759
+ "block_n": block_n,
760
+ "num_programs": task_programs,
761
+ "k_scale": kv_scales[0] if kv_scales else None,
762
+ "v_scale": kv_scales[1] if kv_scales else None,
763
+ }
764
+ if prefix_launch:
765
+ attend(
766
+ q,
767
+ k,
768
+ v,
769
+ tasks=prefix_tasks, # type: ignore[arg-type]
770
+ num_tasks=counts[0:1],
771
+ block_m=prefix_bm,
772
+ min_block_m=prefix_min_bm,
773
+ num_warps=PREFIX_WARPS,
774
+ num_stages=PREFIX_STAGES,
775
+ **common, # type: ignore[arg-type]
776
+ )
777
+ attend(
778
+ q,
779
+ k,
780
+ v,
781
+ tasks=suffix_tasks, # type: ignore[arg-type]
782
+ num_tasks=counts[1:2],
783
+ block_m=suffix_bm,
784
+ num_warps=SUFFIX_WARPS,
785
+ num_stages=SUFFIX_STAGES,
786
+ **common, # type: ignore[arg-type]
787
+ )
788
+ merge_ragged(
789
+ part_o,
790
+ part_lse,
791
+ merge_start,
792
+ merge_end,
793
+ indices,
794
+ counts[2:3],
795
+ out,
796
+ lse_out,
797
+ max_rows=max_rows,
798
+ num_programs=merge_programs,
799
+ )
800
+
801
+
802
+ @_cascade_op.register_fake
803
+ def _cascade_fake(
804
+ q: torch.Tensor,
805
+ k: torch.Tensor,
806
+ v: torch.Tensor,
807
+ plan_tensors: list[torch.Tensor],
808
+ workspace: list[torch.Tensor],
809
+ out: torch.Tensor,
810
+ lse_out: torch.Tensor | None,
811
+ sm_scale: float,
812
+ settings: list[int],
813
+ kv_scales: list[torch.Tensor],
814
+ ) -> None:
815
+ return None
816
+
817
+
818
+ def cascade_decode_attention(
819
+ q: torch.Tensor,
820
+ k: torch.Tensor,
821
+ v: torch.Tensor,
822
+ plan: CascadePlan,
823
+ out: torch.Tensor,
824
+ lse_out: torch.Tensor | None = None,
825
+ *,
826
+ sm_scale: float | None = None,
827
+ k_scale: torch.Tensor | None = None,
828
+ v_scale: torch.Tensor | None = None,
829
+ ) -> None:
830
+ """Decode attention for the batch the plan was last filled for.
831
+
832
+ q: [requests · query_tokens, q_heads, head_dim], fp16 or bf16; request r's
833
+ new tokens are rows r · query_tokens … r · query_tokens + query_tokens - 1,
834
+ oldest first, and token t sees the request's keys up to position
835
+ length - query_tokens + t. k, v: one layer's K and V views from
836
+ kvkernels.layouts, [blocks, block_size, kv_heads, head_dim], same dtype; the
837
+ new tokens' K and V are already in them. out: like q. lse_out, if given:
838
+ [requests · query_tokens, q_heads], fp32, natural log. A request with no keys
839
+ gets out = 0 and lse = -inf. sm_scale defaults to 1 / sqrt(head_dim). Runs
840
+ as the custom op kvkernels::cascade_decode_attention.
841
+
842
+ An fp8 cache: k and v are float8_e4m3fn views holding K / k_scale and
843
+ V / v_scale, with k_scale and v_scale one-element fp32 tensors on q's device
844
+ (an engine's per-layer scales).
845
+ """
846
+ rows = plan.num_requests * plan.query_tokens
847
+ if q.dim() != 3 or q.shape[0] != rows:
848
+ raise ValueError(
849
+ f"q must be [{rows} = requests · query_tokens, q_heads, head_dim], got {tuple(q.shape)}"
850
+ )
851
+ if q.shape[1:] != (plan.num_q_heads, plan.head_dim):
852
+ raise ValueError(f"q heads and head_dim {tuple(q.shape[1:])} do not match the plan")
853
+ kv_fp8 = k.dtype == torch.float8_e4m3fn
854
+ kv_ok = k.dtype == v.dtype and k.dtype in (q.dtype, torch.float8_e4m3fn)
855
+ if q.dtype not in _IN_DTYPES or not kv_ok:
856
+ raise ValueError(
857
+ "q must be fp16 or bf16, and k and v q's dtype or both float8_e4m3fn; "
858
+ f"got {q.dtype}, {k.dtype}, {v.dtype}"
859
+ )
860
+ scales: list[torch.Tensor] = []
861
+ if kv_fp8:
862
+ if not fp8_kv_supported(q.device):
863
+ raise ValueError(
864
+ "an fp8 cache needs compute capability 8.9 or later: Triton has no e4m3 type"
865
+ " below it"
866
+ )
867
+ for name, given in (("k_scale", k_scale), ("v_scale", v_scale)):
868
+ if given is None or given.numel() != 1 or given.dtype != torch.float32:
869
+ raise ValueError(f"an fp8 cache needs {name}: a one-element fp32 tensor")
870
+ assert k_scale is not None and v_scale is not None
871
+ scales = [k_scale.reshape(1), v_scale.reshape(1)]
872
+ for name, t in (("k", k), ("v", v)):
873
+ if t.dim() != 4 or t.shape[2:] != (plan.num_kv_heads, plan.head_dim):
874
+ raise ValueError(
875
+ f"{name} must be [blocks, block_size, {plan.num_kv_heads}, {plan.head_dim}]"
876
+ )
877
+ if t.shape[1] != plan.block_size:
878
+ raise ValueError(f"{name} has block_size {t.shape[1]}, the plan {plan.block_size}")
879
+ if tuple(out.shape) != tuple(q.shape) or out.dtype != q.dtype:
880
+ raise ValueError("out must have q's shape and dtype")
881
+ if lse_out is not None and (
882
+ tuple(lse_out.shape) != tuple(q.shape[:2]) or lse_out.dtype != torch.float32
883
+ ):
884
+ raise ValueError(f"lse_out must be fp32 {tuple(q.shape[:2])}")
885
+ if not _interpreting() and not q.is_cuda:
886
+ raise ValueError("tensors must be on a CUDA device")
887
+ scale = sm_scale if sm_scale is not None else 1.0 / math.sqrt(plan.head_dim)
888
+ _cascade_op(
889
+ q,
890
+ k,
891
+ v,
892
+ plan.tensors(),
893
+ [plan.part_o, plan.part_lse],
894
+ out,
895
+ lse_out,
896
+ scale,
897
+ plan.settings(),
898
+ scales,
899
+ )