@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/techniques/fine-grained-quantization.md

@@ -1,253 +0,0 @@
----
-id: technique-fine-grained-quantization
-title: Fine-Grained FP8/FP4 Quantization
-type: technique
-architectures:
-- sm100
-- sm90
-languages:
-- '*'
-tags:
-- fine-grained-quantization
-- fp8
-- fp4
-- nvfp4
-- block-scale
-confidence: source-reported
-evidence_level: spec
-reproducibility: snippet
-prerequisites:
-- hw-nvfp4
-related:
-- hw-nvfp4
-- kernel-deepgemm
-- technique-fine-grained-quantization
-sources:
-- blog-deepgemm
-- doc-nvidia-tuning-guide
-- pr-vllm-23696
----
-## Overview
-
-Fine-grained quantization applies per-block (rather than per-tensor) scaling factors to low-precision data, preventing outlier values from destroying the quantization precision of an entire tensor. DeepSeek pioneered the tile-wise 1x128 scaling for activations and block-wise 128x128 scaling for weights in their FP8 training framework. On Blackwell (SM100), native block scaling support in tcgen05.mma enables hardware-accelerated fine-grained quantization using the UE8M0 scale format, while Hopper requires software-managed CUDA core promotion.
-
-## Scaling Granularities
-
-```
-Per-tensor scaling (coarsest):
-  Entire tensor shares one FP32 scale factor.
-  Problem: a single outlier ruins precision for all elements.
-  
-  [=== entire MxK matrix === ] -> 1 scale
-
-Per-block 128x128 scaling (weights):
-  Each 128x128 block has its own scale factor.
-  128x128 = 16,384 elements per scale -> 0.006% overhead.
-  
-  +---+---+---+
-  |s1 |s2 |s3 |  <- each block has independent scale
-  +---+---+---+
-  |s4 |s5 |s6 |
-  +---+---+---+
-
-Per-tile 1x128 scaling (activations):
-  Each row of 128 elements has its own scale factor.
-  Captures per-channel activation distributions.
-  
-  [s1: ====128 elements====]
-  [s2: ====128 elements====]
-  [s3: ====128 elements====]
-```
-
-## DeepGEMM: Tile-wise and Block-wise Scaling
-
-DeepGEMM implements fine-grained FP8 GEMM with two scaling patterns:
-
-```cuda
-// DeepGEMM FP8 GEMM with fine-grained scaling
-// A (activations): FP8 E4M3 with tile-wise 1x128 FP32 scales
-// B (weights):     FP8 E4M3 with block-wise 128x128 FP32 scales
-// C (output):      FP32 accumulator
-
-// Scale tensor shapes:
-//   scale_A: [M, K/128] -- one FP32 scale per 128 elements along K for each row
-//   scale_B: [K/128, N/128] -- one FP32 scale per 128x128 block
-
-__device__ void deepgemm_fp8_tile(
-    const fp8_e4m3* A,    // [TILE_M, TILE_K] in FP8
-    const fp8_e4m3* B,    // [TILE_K, TILE_N] in FP8
-    const float* scale_A, // [TILE_M, TILE_K/128]
-    const float* scale_B, // [TILE_K/128, TILE_N/128]
-    float* C,             // [TILE_M, TILE_N] accumulator
-    int M, int N, int K)
-{
-    // For each 128-element K-chunk:
-    for (int k_block = 0; k_block < K; k_block += 128) {
-        // 1. Load FP8 A tile [TILE_M, 128] and B tile [128, TILE_N]
-        // 2. Perform MMA: partial = A_fp8 * B_fp8 (in limited-precision FP32)
-        // 3. Apply combined scale: scale_A[m, k_block/128] * scale_B[k_block/128, n_block/128]
-        // 4. Accumulate: C[m, n] += partial * combined_scale
-
-        for (int m = 0; m < TILE_M; m++) {
-            float sa = scale_A[m * (K / 128) + k_block / 128];
-            for (int n_block = 0; n_block < TILE_N; n_block += 128) {
-                float sb = scale_B[(k_block / 128) * (N / 128) + n_block / 128];
-                float combined_scale = sa * sb;
-
-                // Apply scale to the partial MMA result
-                for (int n = n_block; n < n_block + 128; n++) {
-                    C[m * TILE_N + n] += partial[m][n] * combined_scale;
-                }
-            }
-        }
-    }
-}
-```
-
-## Hopper: CUDA Core Promotion (Nc=128)
-
-On Hopper (SM90), the wgmma instruction accumulates in Tensor Core registers with limited precision (~FP22, not true FP32). To maintain numerical accuracy, DeepGEMM promotes the partial sum to a separate FP32 accumulator on CUDA Cores every 4 wgmma instructions (Nc=128, since each wgmma processes 32 K-elements):
-
-```cuda
-// Hopper FP8 GEMM with CUDA Core promotion (DeepGEMM pattern)
-// Every Nc=128 K-elements, promote Tensor Core accumulator to FP32
-
-__device__ void hopper_fp8_gemm_with_promotion(
-    const fp8_e4m3* A, const fp8_e4m3* B,
-    const float* scale_A, const float* scale_B,
-    float* C_accumulator,
-    int K)
-{
-    // Tensor Core limited-precision accumulator (FP22-ish)
-    // These are wgmma output registers
-    float tc_acc[TILE_M_PER_THREAD][TILE_N_PER_THREAD] = {0};
-
-    // True FP32 accumulator on CUDA Cores
-    float fp32_acc[TILE_M_PER_THREAD][TILE_N_PER_THREAD] = {0};
-
-    int promotion_interval = 128;  // Nc = 128 elements = 4 wgmma ops
-    int wgmma_count = 0;
-
-    for (int k = 0; k < K; k += 32) {
-        // Issue wgmma (accumulates in limited-precision tc_acc)
-        wgmma_mma_async(tc_acc, smem_A_ptr, smem_B_ptr);
-        wgmma_count++;
-
-        if (wgmma_count == 4) {  // Every 128 K-elements
-            // Promote: transfer tc_acc to fp32_acc on CUDA Cores
-            wgmma_wait();  // Ensure wgmma is complete
-
-            int k_block = k / promotion_interval;
-            for (int m = 0; m < TILE_M_PER_THREAD; m++) {
-                float sa = scale_A[/*...*/];
-                for (int n = 0; n < TILE_N_PER_THREAD; n++) {
-                    float sb = scale_B[/*...*/];
-                    // Add scaled partial to true FP32 accumulator
-                    fp32_acc[m][n] += tc_acc[m][n] * sa * sb;
-                    // Reset TC accumulator for next interval
-                    tc_acc[m][n] = 0;
-                }
-            }
-            wgmma_count = 0;
-        }
-    }
-}
-```
-
-The Nc=128 interval was chosen because:
-- 4 wgmma operations process 128 K-elements (4 x 32)
-- At this interval, the accumulated FP22 error is bounded to ~0.1% relative error
-- Fewer than 4 ops (Nc=32, Nc=64) adds too much promotion overhead
-- More than 4 ops (Nc=256) allows unacceptable precision loss
-
-## Blackwell: Native Block Scaling
-
-On Blackwell, tcgen05.mma supports native block scaling via the UE8M0 (unsigned 8-bit exponent, no mantissa) format. The hardware applies per-block scale factors directly during the MMA operation, eliminating the software promotion step:
-
-```cuda
-// Blackwell native block scaling with UE8M0
-// Scale format: UE8M0 = pure power-of-two scale (2^exponent)
-// Packed: 4 UE8M0 values per 32-bit integer
-
-// DeepGEMM SM100 kernel: scale_A and scale_B are UE8M0 packed
-// tcgen05.mma applies scales automatically during accumulation
-
-struct BlockScaleDescriptor {
-    // 4 UE8M0 scale values packed into one uint32
-    // Each UE8M0 is an 8-bit unsigned exponent: value = 2^(e - 127)
-    uint32_t packed_scales;  // Contains 4 block scales
-
-    // Dequantization for block [i]:
-    //   scale_i = 2^(((packed >> (i*8)) & 0xFF) - 127)
-};
-
-// PTX for tcgen05.mma with block scaling:
-// The .scale modifier tells the hardware to apply UE8M0 scales
-// from a designated SMEM region alongside the MMA operands
-```
-
-```ptx
-// tcgen05.mma with native block scaling (Blackwell PTX)
-// This instruction applies UE8M0 scales from SMEM during MMA
-tcgen05.mma.cta_group::1.kind::f8f6f4
-    [%tmem_addr],           // TMEM accumulator destination
-    [%desc_a],              // SMEM descriptor for A operand
-    [%desc_b],              // SMEM descriptor for B operand
-    %scale_d,               // Scale descriptor for D (output)
-    %enable_mask,
-    [%scale_a_smem],        // UE8M0 scales for A in SMEM
-    [%scale_b_smem];        // UE8M0 scales for B in SMEM
-```
-
-## UE8M0 vs E4M3 Scale Formats
-
-| Property | UE8M0 (Blackwell native) | E4M3 (NVFP4 hackathon) | FP32 (DeepGEMM Hopper) |
-|----------|-------------------------|------------------------|------------------------|
-| Bits | 8 | 8 | 32 |
-| Representable values | Powers of 2 only | 240 distinct values | Full FP32 range |
-| Range | 2^-127 to 2^128 | ~0 to 448 | Full FP32 |
-| Block size | 32 (MXFP standard) | 16 (NVFP4) | 128 (DeepGEMM) |
-| Hardware support | tcgen05.mma native | Software decode | Software promotion |
-| Precision impact | Coarser (power-of-2 only) | Fine (non-power-of-2) | Best (FP32) |
-
-## NVFP4 Two-Level Scaling
-
-The NVFP4 format used in the GPU Mode hackathon has its own scaling scheme:
-
-```cuda
-// NVFP4 dequantization with two-level scaling
-// Level 1: per-block FP8 E4M3 scale (every 16 FP4 elements)
-// Level 2: per-tensor FP32 global scale
-
-__device__ float dequant_nvfp4(
-    uint8_t fp4_packed,    // Two FP4 values packed in one byte
-    fp8_e4m3 block_scale,  // Per-block scale (every 16 elements)
-    float global_scale)    // Per-tensor global scale
-{
-    // Extract one FP4 value (E2M1 format)
-    // Representable: 0, 0.5, 1, 1.5, 2, 3, 4, 6
-    float fp4_val = decode_e2m1(fp4_packed & 0x0F);
-
-    // Two-level dequantization
-    float result = global_scale * float(block_scale) * fp4_val;
-    return result;
-}
-
-// Block scale layout for NVFP4:
-// For matrix A of shape [M, K]:
-//   scale_A shape: [M, K/16] -- one FP8 E4M3 per 16 FP4 elements
-//   Finer granularity than MXFP4 (block size 32) or DeepGEMM (block size 128)
-```
-
-## When to Use
-
-- **FP8 training**: Use tile-wise 1x128 for activations and block-wise 128x128 for weights (DeepGEMM pattern). This is the validated approach for training 671B+ parameter models.
-- **FP4 inference on Blackwell**: Use NVFP4 with E4M3 block scales (block size 16) for highest precision, or MXFP4 with UE8M0 scales (block size 32) for native hardware acceleration.
-- **Hopper FP8 inference**: Use CUDA core promotion with Nc=128 interval to maintain precision despite limited TC accumulation.
-
-## Caveats
-
-- UE8M0 scales are power-of-two only. Non-power-of-two distributions (common in activations) lose precision compared to E4M3 or FP32 scales.
-- Smaller block sizes (16 for NVFP4 vs 128 for DeepGEMM) provide better precision but higher overhead: more scale values to store, load, and apply.
-- The Nc=128 promotion interval on Hopper is a performance-accuracy tradeoff. Reducing Nc improves accuracy but adds more promotion overhead. Increasing Nc risks precision degradation.
-- On Blackwell, native block scaling only works with UE8M0. Using E4M3 or FP32 scales still requires software handling.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/techniques/kernel-fusion.md

@@ -1,68 +0,0 @@
----
-id: technique-kernel-fusion
-title: Kernel Fusion
-type: technique
-architectures:
-- sm100
-- sm90
-languages:
-- '*'
-tags:
-- kernel-fusion
-- fused-kernel
-- tmem
-confidence: source-reported
-evidence_level: spec
-reproducibility: snippet
-prerequisites:
-- hw-tmem
-related:
-- kernel-fused-moe
-- kernel-nvfp4-gemm
-- technique-epilogue-fusion
-sources:
-- contest-gpumode-p3
-- contest-flashinfer-track-a
-- blog-tflops-gap-fp4-moe
----
-# Kernel Fusion
-
-## Overview
-
-Kernel fusion combines multiple operations into a single kernel launch, eliminating intermediate global memory roundtrips. Critical for MoE and attention pipelines where 5-7 sequential launches each incur latency, synchronization, and memory traffic overhead.
-
-## Examples
-
-### Fused Gate-Up Dual GEMM + SwiGLU
-```cuda
-// Instead of: gate_gemm → up_gemm → silu → multiply (4 kernels)
-// Fused: single kernel with two TMEM accumulators
-__global__ void fused_gate_up_silu(...) {
-    uint32_t tmem_gate = tmem_alloc(256);
-    uint32_t tmem_up = tmem_alloc(256);
-
-    for (int k = 0; k < K; k += BLOCK_K) {
-        tcgen05_mma(x_smem, w_gate_smem, tmem_gate);
-        tcgen05_mma(x_smem, w_up_smem, tmem_up);
-    }
-
-    float g = tmem_load(tmem_gate);
-    float u = tmem_load(tmem_up);
-    output = (g / (1.0f + expf(-g))) * u;  // SwiGLU fused
-}
-```
-
-### MoE Fusion Progression
-- **vLLM (7 kernels)**: softmax → topk → dispatch → gate → up → silu_mul → down+combine
-- **SGLang (5 kernels)**: router+topk → dispatch → fused gate-up-silu → down → combine
-- **Ideal (1-2 kernels)**: all ops in one launch, saves 21.9% activation memory traffic
-
-## Constraints
-
-- TMEM capacity limits how many accumulators can fuse (256 cols total)
-- Register pressure on epilogue if fusing complex activations
-- Fusion opportunities depend on dataflow shape (dependency graph must be DAG-compatible with CTA scope)
-
-## Related
-- [fused-moe](../kernels/fused-moe.md)
-- [epilogue-fusion](epilogue-fusion.md)

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/techniques/persistent-kernel/algo-persistent-kernel.md

@@ -1,218 +0,0 @@
----
-title: Persistent kernel (grid = SM count, internal tile loop)
-status: verified
-evidence_level: measured
-applies_to_pattern_class:
-- tensor-core
-- tensor-core/gemm
-- cuda-core
-applies_to_ops:
-- gemm
-- reduction
-- attention
-requires_sm: '>=7.0'
-requires_features: []
-single_kernel_useful: true
-cuda_version_tested: 12.9.86
-driver_version_tested: 570.124.06
-toolchain: nvcc 12.9 + ptxas 12.9
-measured_on: H200-SXM | sm_90a | cuda 12.9.86 | driver 570.124.06
-source:
-- path: wiki/nvidia/hardware/tma/skill-tma-ptx.md
-  anchor: cutlass-free TMA primitive (composes per-tile in the persistent loop)
-- path: wiki/nvidia/hardware/wgmma/skill-wgmma-ptx.md
-  anchor: cutlass-free wgmma primitive
-artifacts:
-  code: artifacts/experience/api-probes/gemm/gemm_compare_pingpong.cu
-  build: artifacts/experience/api-probes/gemm/build_persistent.sh
-  introspection: artifacts/experience/api-probes/gemm/device.json
-  profile: artifacts/experience/api-probes/gemm/2026-04-28-persistent-kernel-ablation.csv
-  ablation: artifacts/experience/api-probes/gemm/2026-04-28-persistent-kernel-ablation.csv
-upstream_repo: cutlass@f74fea9c (one possible implementation; see "References")
-related_apis: []
-related_skills:
-- warp-specialization
-- gemm-aligned
-- tma-ptx
-- wgmma-ptx
-id: algo-persistent-kernel
-type: algorithm
-vendor: nvidia
-tags:
-- cuda-cpp
-- wgmma
-- tma
-- mbarrier
-- cluster
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- data-reuse
-- shared-memory-optimization
-- swizzling
-- tile-scheduling
-- tma-multicast
-- gemm
-- fused-kernel
-- attention
-- quantization
-- ptx
-- cute-dsl
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L28042
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L11680-L11686
-- source_id: blogs/colfax
-  path: cutlass-tutorial-persistent-kernels-and-stream-k
-  anchor: persistent-kernel + stream-K walkthrough — tile-scheduler abstraction
-- source_id: blogs/colfax
-  path: developing-cuda-kernels-for-gemm-on-nvidia-hopper-architecture-using-cutlass
-  anchor: pingpong vs cooperative scheduling section
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-- tma
-- mbarrier
-- cluster
-techniques:
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- data-reuse
-- shared-memory-optimization
-- swizzling
-- tile-scheduling
-- tma-multicast
-kernel_types:
-- gemm
-- fused-kernel
-- attention
-- quantization
-confidence: experimental
-artifact_dir: artifacts/experience/api-probes/gemm
-scope: "The pattern pays off from a few hundred work tiles upward, where the launch and scheduler-startup cost amortizes; below that a non-persistent launch is cheaper. It assumes a uniform set of work tiles and a grid pinned to the SM count -- it does not apply where the grid must oversubscribe, and it gives no benefit on a single-tile problem."
----
-# Persistent kernel (algorithm skeleton)
-
-## What it is
-
-An *algorithmic pattern* where the launch grid size equals the device's SM count (or a small multiple) and the kernel loops internally over the work units, instead of one CTA per work unit. The tile-to-CTA assignment is computed inside the kernel by a **tile scheduler** rather than by `dim3 grid`.
-
-The pattern is independent of cutlass. cutlass packages it as `PersistentScheduler` / `StreamKScheduler` driving its `KernelTmaWarpSpecialized{Pingpong,Cooperative}` mainloops; the same idea works in any kernel that processes a uniform set of work tiles.
-
-## Why the pattern works
-
-Two structural wins over a non-persistent launch:
-
-1. **Eliminates per-tile launch overhead.** A non-persistent kernel with N tiles incurs one launch + grid setup; a persistent kernel pays once and amortizes across all tiles via the inner loop. For small per-tile work, the overhead can be a noticeable fraction.
-2. **Smarter rasterization without re-launching.** The scheduler can swizzle, linearize, or stream-K the tile order to improve L2 reuse, balance work across SMs, or split a single tile across multiple CTAs — all decided per work-tile inside the kernel.
-
-The pattern composes naturally with warp-specialization: the producer warps stream TMA loads continuously while the consumer warpgroups process tiles handed to them by the scheduler. The two patterns are orthogonal: warp-specialization is about *intra-CTA* parallelism (producer vs consumer); persistent is about *inter-tile* sharing of the same CTA.
-
-## When to use it
-
-- Large work-tile counts (≥ a few hundred tiles) where the launch + scheduler-startup cost can amortize.
-- Stream-K decompositions where work is split across SMs at the K dimension; the persistent grid is the substrate that lets stream-K rebalance load.
-- LLM serving / repeated launches with similar shapes — persistent reduces launch jitter.
-
-## When NOT to use it
-
-- Tiny problems (≤ a few CtaTiles in M and N): the inner-loop setup overhead dominates a non-persistent launch's tiny gain.
-- Kernels that need fine-grained per-tile compile-time specialization (different epilogues per tile region): persistent kernels share the body across tiles.
-
-## Variants
-
-| Variant | Consumer warpgroups per CTA | Per-CTA tile cadence in the persistent loop |
-|---|---|---|
-| **Plain persistent** | 1 | one tile, then advance to next |
-| **Pingpong persistent** | 2 (alternating) | consumer-A computes tile T, consumer-B prepares T+1; on tile T+2 they swap |
-| **Cooperative persistent** | 2 (cooperating on same tile) | both consumers split the wgmma rows of one tile; advance jointly to next; pair with cluster-multicast TMA (cutlass-free PTX measured at `sources/experience/hw-probes/tma-ptx.md`, 1.26× / 1.79× effective-bandwidth at C=2 / C=4) |
-
-Cutlass spells these as `KernelTmaWarpSpecializedPingpong` and `KernelTmaWarpSpecializedCooperative`; those names label *the warp-spec variant + persistent loop combination*. The persistent loop itself is independent of warp-spec — a non-warp-specialized persistent kernel also exists in principle, just not landed in cutlass for sm_90.
-
-## Algorithm skeleton (cutlass-free)
-
-```
-__global__ void persistent_kernel(/* tensor maps, problem dims */) {
-    // 1. Compute the work-tile partition for this CTA.
-    int n_tiles_total  = N_TILES_M * N_TILES_N;
-    int n_tiles_per_cta = (n_tiles_total + gridDim.x - 1) / gridDim.x;
-    int my_first_tile  = blockIdx.x * n_tiles_per_cta;
-    int my_last_tile   = min(my_first_tile + n_tiles_per_cta, n_tiles_total);
-
-    // 2. Persistent loop — process all tiles assigned to this CTA in order.
-    for (int t = my_first_tile; t < my_last_tile; ++t) {
-        int tile_m = scheduler_row(t);   // e.g. linear, swizzled, or stream-K
-        int tile_n = scheduler_col(t);
-
-        // 3. Per-tile work — composes the warp-specialized mainloop from
-        //    wiki/nvidia/techniques/warp-specialization.md (producer/consumer
-        //    + mbarrier-pipelined TMA→wgmma), accumulating into per-thread
-        //    fragment registers across all K iterations of this tile.
-        accum = {0};
-        for (int k = 0; k < N_TILES_K; ++k) {
-            // ... pipelined TMA load A[tile_m, k] + B[k, tile_n] ...
-            // ... wgmma against the loaded smem tiles ...
-            // ... accumulate into accum ...
-        }
-
-        // 4. Per-tile epilogue — quantize / activate / store accum to D[tile_m, tile_n].
-        epilogue.store(accum, D, tile_m, tile_n);
-    }
-}
-
-// Launch with grid = SM count (× cluster size for cooperative).
-persistent_kernel<<<N_SMS, /*threads*/, smem>>>(...);
-```
-
-Variants:
-
-- **Pingpong** — split `for t` into two interleaved streams (even-t vs odd-t) and route them to consumer-A vs consumer-B. The producer warp keeps streaming for both; the two consumers each touch ⌈n_tiles_per_cta / 2⌉ tiles.
-- **Cooperative** — keep `for t` as-is but split the per-tile wgmma work across two consumer warpgroups (rows [0..31] and [32..63] of the m64 atom). Pair with cluster-multicast TMA so the producer's load broadcasts to two CTAs in a 2-CTA cluster.
-- **Stream-K** — replace `scheduler_row/col(t)` with a stream-K decomposition that lets one tile's K-loop be split across multiple CTAs, with the partial reductions joined via a final atomic or split-K commit. The persistent grid is the substrate; stream-K is a different scheduler choice on top.
-
-The scheduler functions `scheduler_row/col` are pure host-precomputable functions of `t, blockIdx, problem_dims`. Common choices:
-
-| Scheduler | When |
-|---|---|
-| Linear (`t / N_TILES_N, t % N_TILES_N`) | Simplest; near-optimal when N_TILES_M × N_TILES_N is small |
-| Swizzled (Hilbert-like) | Improves L2 reuse when adjacent tiles share input rows or cols |
-| Stream-K | Splits one tile's K across SMs; helps load balance when tile count is not a multiple of SM count |
-
-## Implementations on disk
-
-- **Cutlass realization** — `cutlass::gemm::kernel::tile_scheduler.hpp` (`PersistentScheduler` / `StreamKScheduler`), composed into `sm90_gemm_tma_warpspecialized_pingpong.hpp` and `_cooperative.hpp` mainloops. Cluster-shape constraints, multicast pairings, and per-stage smem allocators are cutlass-implementation details; see `pitfalls.md`.
-- **Cutlass-free building blocks** — TMA-PTX (`wiki/nvidia/hardware/tma/skill-tma-ptx.md`) and wgmma-PTX (`wiki/nvidia/hardware/wgmma/skill-wgmma-ptx.md`) provide the per-tile inner kernel; the persistent loop above is plain CUDA C++. No cutlass dependency at the algorithm level.
-
-## Measured Characteristics
-
-H200-SXM, sm_90a. Algorithm validated in the cutlass realization at problem 2048 × 2048 × 2048, TF32 inputs / F32 accumulator (the cutlass-free realization is queued):
-
-| Variant | Persistent? | Cluster | Latency (μs) | TFLOPS | cuBLAS diff |
-|---|---|---|---|---|---|
-| Plain WS (non-persistent) | OFF | `<1, 1, 1>` | 87.94 | 195.4 | bit-identical |
-| Pingpong persistent | ON | `<2, 1, 1>` | 88.64 | 193.8 | bit-identical |
-| Cooperative persistent | ON | `<4, 2, 1>` | 91.92 | 186.9 | bit-identical |
-
-At 2048³ on H200 the persistent advantage is small (within 1% of non-persistent plain WS for pingpong; cooperative pays 4% for its setup cost). The persistent advantage compounds at LARGER problems: the aligned-GEMM 8192³ measurement records 292.7 TFLOPS for the cooperative persistent schedule — substantially above the 2048³ ceiling because the persistent grid keeps all 132 H200 SMs busy across many tiles without launch-overhead bubbles, and cluster-multicast TMA amortizes once the per-CTA tile count is large.
-
-## Cross-references
-
-- Warp-specialization sibling (intra-CTA pattern): `wiki/nvidia/techniques/warp-specialization.md`
-- TMA-PTX (cutlass-free producer side): `wiki/nvidia/hardware/tma/skill-tma-ptx.md`
-- wgmma-PTX (cutlass-free consumer side): `wiki/nvidia/hardware/wgmma/skill-wgmma-ptx.md`
-- Aligned GEMM consumer: `wiki/nvidia/foundations/compute/gemm.md`
-- Cutlass library-usage notes + tuning log: `wiki/nvidia/code-walkthroughs/cutlass-cute/persistent-kernel/{README.md, tuning.md}`
-- Probe + ablation: `sources/experience/api-probes/gemm.md`
-- Underlying TMA hw-probe (composed per tile): `sources/experience/hw-probes/tma-ptx.md`
-- Underlying wgmma hw-probe (composed per tile): `sources/experience/hw-probes/wgmma-ptx.md`
-- Failure modes: `wiki/nvidia/techniques/persistent-kernel/pitfalls.md`

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/techniques/persistent-kernel/pitfalls.md

@@ -1,85 +0,0 @@
----
-id: algo-persistent-kernel-pitfalls
-type: algorithm
-vendor: nvidia
-title: Pitfalls
-tags:
-- cuda-cpp
-- tma
-- cluster
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- shared-memory-optimization
-- swizzling
-- tile-scheduling
-- tma-multicast
-- gemm
-evidence_level: spec
-source:
-- path: spec
-  anchor: Algorithm reference
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-hardware_features:
-- tma
-- cluster
-techniques:
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- shared-memory-optimization
-- swizzling
-- tile-scheduling
-- tma-multicast
-kernel_types:
-- gemm
-confidence: source-reported
-scope: "Applies to a persistent grid pinned to the device SM count (132 on H200). The pingpong tail and cooperative-multicast items assume a cluster shape of at least <2,1,1>; the stream-K item assumes partial-sum reduction is available. None of the pitfalls are about correctness of a non-persistent launch."
----
-# Persistent kernel — pitfalls
-
-## 1. Grid size must equal device SM count, not problem tile count
-
-The whole point of the persistent pattern is to launch one CTA per SM (× cluster size) and have each CTA consume many tiles via the inner loop. Launching with `gridDim = N_TILES_M × N_TILES_N` defeats the pattern and pays both the persistent overhead AND the per-tile launch overhead. Use `cudaDeviceGetAttribute(&n_sms, cudaDevAttrMultiProcessorCount, ...)` (or hard-code 132 for H200) and pass that as the launch grid.
-
-## 2. Pingpong needs an even tile count per CTA, OR the tail runs one consumer
-
-The pingpong variant alternates work between two consumer warpgroups within each CTA. If `n_tiles_per_cta` is odd, one consumer takes the tail tile alone. This is fine for correctness but the per-tile latency at the tail differs from steady-state — don't infer steady-state per-tile timing from a small problem where the tail dominates.
-
-## 3. Cooperative is meaningful only with cluster-multicast TMA
-
-The cooperative variant pairs two consumer warpgroups on the same tile and pairs that with cluster-multicast TMA so the load broadcasts to ≥ 2 CTAs in a cluster. Setting cluster `<1, 1, 1>` defeats the multicast and you pay coordination overhead for nothing. Always set cluster ≥ `<2, 1, 1>` (typically `<4, 2, 1>`) and use the cluster-multicast TMA instruction variant when running cooperative.
-
-## 4. Stream-K is a scheduler choice, not a separate variant
-
-Stream-K decomposes one tile's K-loop across multiple CTAs and joins via partial-sum reductions. It is a *different scheduler* on top of the same persistent grid + warp-specialized mainloop — replace `scheduler_row/col(t)` with the stream-K decomposition; the rest of the kernel doesn't change. Stream-K and the cooperative variant are independent axes.
-
-## 5. The persistent grid does not oversubscribe
-
-A persistent kernel always launches at SM-count × cluster-size occupancy. You cannot scale by launching more CTAs than the grid — the scheduler indexes assume one persistent CTA per SM (or per cluster slot). For concurrency, use multiple streams; each stream gets its own persistent kernel and the GPU schedules them across SMs.
-
-## 6. Per-tile inner work must be uniform for the scheduler to balance
-
-Linear / swizzled schedulers assume every tile takes the same inner-loop time. If some tiles are masked / shorter / cheaper, the scheduler does not adapt and the cheap-tile CTAs sit idle. Stream-K can rebalance, but only along the K dimension; M/N skew requires a different (work-stealing) scheduler.
-
-## 7. The persistent advantage is shape-dependent
-
-At 2048³ on H200, persistent pingpong (193.8 TFLOPS) is within 1 % of non-persistent plain WS (195.4 TFLOPS); cooperative is 4 % slower. Persistent wins at larger problems — at 8192³ the cooperative variant reaches 292.7 TFLOPS (cluster-multicast finally amortizes; the persistent grid keeps all 132 SMs busy without launch-overhead bubbles between tiles). Don't pick persistent + cooperative for small problems; the coordination cost dominates.
-
-## 8. The scheduler functions are host-precomputable — don't recompute per iteration
-
-`scheduler_row(t)` and `scheduler_col(t)` are pure functions of `t, blockIdx, problem_dims`. For complex schedulers (Hilbert / stream-K) precompute the lookup table on host and pass via `__grid_constant__`; recomputing inside the persistent loop wastes cycles in the hot path.
-
-## 9. Cutlass implementation specifics (PersistentScheduler / StreamKScheduler)
-
-These are pitfalls of cutlass's *implementation* of the persistent pattern, not of the pattern itself. Listed for reference:
-
-- `PersistentScheduler` launches `(SMs × CTAs-per-cluster, 1, 1)` — for cluster `<4, 2, 1>` on H200 (132 SMs), that is approximately ⌈132 / 8⌉ × 8 ≈ 136 CTAs total. The grid shape changes if you switch GPUs but the problem doesn't.
-- `TileSchedulerType` is a template parameter independent of the mainloop schedule — set it to `cutlass::gemm::StreamKScheduler` to get stream-K on top of `KernelTmaWarpSpecializedCooperative`.
-- The mainloop's `Stages` parameter is unrelated to persistence. Stages = smem ring depth in the warp-specialized mainloop; persistence = inner tile loop. They compose orthogonally.
-
-A cutlass-free implementation skips all of these by making the scheduler a plain device function and the launch grid an explicit `N_SMS` constant.