@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/techniques/vectorized-loads.md

@@ -1,313 +0,0 @@
----
-id: technique-vectorized-loads
-title: Wide Vectorized Loads and Cache Policies
-type: technique
-architectures:
-- sm100
-- sm90
-languages:
-- '*'
-tags:
-- vectorized-loads
-- cache-policy
-- register-budgeting
-confidence: source-reported
-evidence_level: spec
-reproducibility: snippet
-prerequisites: []
-related:
-- kernel-nvfp4-gemv
-- pattern-memory-bound
-sources:
-- blog-yue-nvfp4
-- blog-amandeep-nvfp4
-- contest-gpumode-p1
----
-## Overview
-
-For memory-bound kernels (low arithmetic intensity), maximizing global memory throughput is critical. Three complementary techniques from the GPU Mode NVFP4 Hackathon achieve this: (1) wide vectorized loads (128-bit and 256-bit) to saturate memory bandwidth per thread, (2) differentiated L1 cache policies to keep reused data hot while bypassing the cache for streaming data, and (3) register budgeting via `-maxrregcount` to increase occupancy. These techniques reduced NVFP4 GEMV latency from 2000us to 22.4us (89x improvement).
-
-## Wide Vectorized Loads
-
-Standard 32-bit loads waste memory bus bandwidth. Wider loads amortize the instruction overhead and saturate the 8 TB/s HBM bandwidth of B200:
-
-```cuda
-// Vectorized load widths comparison for FP4 GEMV
-// Each thread loads more data per instruction
-
-// 32-bit load: 4 bytes per thread per instruction
-float val;
-asm volatile("ld.global.b32 %0, [%1];" : "=f"(val) : "l"(ptr));
-
-// 64-bit load: 8 bytes per thread per instruction
-uint2 val64;
-asm volatile("ld.global.v2.u32 {%0,%1}, [%2];"
-    : "=r"(val64.x), "=r"(val64.y) : "l"(ptr));
-
-// 128-bit load: 16 bytes per thread per instruction (preferred)
-uint4 val128;
-asm volatile("ld.global.v4.u32 {%0,%1,%2,%3}, [%4];"
-    : "=r"(val128.x), "=r"(val128.y), "=r"(val128.z), "=r"(val128.w)
-    : "l"(ptr));
-
-// 256-bit load: 32 bytes per thread per instruction
-// Requires v4.u64 (4 x 64-bit)
-uint64_t v256[4];
-asm volatile("ld.global.v4.u64 {%0,%1,%2,%3}, [%4];"
-    : "=l"(v256[0]), "=l"(v256[1]), "=l"(v256[2]), "=l"(v256[3])
-    : "l"(ptr));
-```
-
-For the NVFP4 GEMV, 128-bit and 256-bit loads are essential because FP4 elements are only 0.5 bytes each. A 256-bit load fetches 64 FP4 values in a single instruction:
-
-```cuda
-// NVFP4 GEMV: each thread loads 64 FP4 values via 256-bit load
-// Then unpacks using PTX byte manipulation instead of bitwise ops
-__device__ void load_and_unpack_nvfp4_256bit(
-    const uint8_t* fp4_data,  // Packed FP4 data (2 values per byte)
-    float* unpacked,          // Output: 64 FP32 values
-    int offset)
-{
-    // 256-bit load: 32 bytes = 64 FP4 values
-    uint64_t raw[4];
-    const uint64_t* ptr = reinterpret_cast<const uint64_t*>(fp4_data + offset);
-    asm volatile(
-        "ld.global.v4.u64 {%0,%1,%2,%3}, [%4];"
-        : "=l"(raw[0]), "=l"(raw[1]), "=l"(raw[2]), "=l"(raw[3])
-        : "l"(ptr)
-    );
-
-    // Unpack using PTX mov.b32 byte extraction
-    // This avoids the bitwise shift-and-mask overhead
-    for (int i = 0; i < 4; i++) {
-        uint32_t lo = (uint32_t)(raw[i]);
-        uint32_t hi = (uint32_t)(raw[i] >> 32);
-
-        // PTX byte unpack: extract individual bytes from 32-bit word
-        uint32_t b0, b1, b2, b3;
-        asm volatile("mov.b32 {%0,%1,%2,%3}, %4;"
-            : "=r"(b0), "=r"(b1), "=r"(b2), "=r"(b3) : "r"(lo));
-
-        // Each byte contains 2 FP4 values — decode low word (8 values)
-        for (int b = 0; b < 4; b++) {
-            uint32_t byte_val = (b == 0) ? b0 : (b == 1) ? b1 : (b == 2) ? b2 : b3;
-            unpacked[i * 16 + b * 2]     = decode_fp4(byte_val & 0xF);
-            unpacked[i * 16 + b * 2 + 1] = decode_fp4((byte_val >> 4) & 0xF);
-        }
-
-        // Unpack high word (next 8 values from same 64-bit element)
-        uint32_t hb0, hb1, hb2, hb3;
-        asm volatile("mov.b32 {%0,%1,%2,%3}, %4;"
-            : "=r"(hb0), "=r"(hb1), "=r"(hb2), "=r"(hb3) : "r"(hi));
-        for (int b = 0; b < 4; b++) {
-            uint32_t byte_val = (b == 0) ? hb0 : (b == 1) ? hb1 : (b == 2) ? hb2 : hb3;
-            unpacked[i * 16 + 8 + b * 2]     = decode_fp4(byte_val & 0xF);
-            unpacked[i * 16 + 8 + b * 2 + 1] = decode_fp4((byte_val >> 4) & 0xF);
-        }
-    }
-}
-```
-
-## L1 Cache Policy Differentiation
-
-Different data streams have different reuse patterns. Applying the correct cache policy per stream avoids L1 pollution:
-
-```cuda
-// Cache policy selection based on data reuse pattern
-//
-// Matrix A (streamed, each row used once): bypass L1
-// Vector B (reused across all rows):      keep in L1
-
-// L1::no_allocate -- data bypasses L1 cache (streaming access)
-// Used for matrix A which is read once per GEMV
-asm volatile(
-    "ld.global.L1::no_allocate.v4.u32 {%0,%1,%2,%3}, [%4];"
-    : "=r"(a.x), "=r"(a.y), "=r"(a.z), "=r"(a.w)
-    : "l"(matrix_a_ptr)
-);
-
-// L1::evict_last -- data stays in L1 as long as possible
-// Used for vector B which is reused across all M rows
-asm volatile(
-    "ld.global.L1::evict_last.v4.u32 {%0,%1,%2,%3}, [%4];"
-    : "=r"(b.x), "=r"(b.y), "=r"(b.z), "=r"(b.w)
-    : "l"(vector_b_ptr)
-);
-```
-
-The impact of cache policies from the GPU Mode Hackathon:
-
-```
-No cache policy differentiation:  39 us
-A: L1::no_allocate, B: L1::evict_last: 27 us  (1.44x faster)
-```
-
-The full set of PTX load cache qualifiers:
-
-```ptx
-// PTX load qualifiers for cache control
-
-// Default: normal L1 and L2 caching
-ld.global.b32          %r, [%addr];
-
-// L1 bypass: skip L1, still cached in L2
-ld.global.L1::no_allocate.b32  %r, [%addr];
-
-// L1 keep: prioritize keeping in L1 (evict last)
-ld.global.L1::evict_last.b32   %r, [%addr];
-
-// L2 promotion hint (256-byte sector)
-// Used by DeepEP for communication overlap
-ld.global.nc.L1::no_allocate.L2::256B.b32  %r, [%addr];
-
-// Non-coherent read-only (nc): uses texture path
-// Avoids coherence traffic, useful for read-only data
-ld.global.nc.b32       %r, [%addr];
-```
-
-## Register Budgeting (-maxrregcount)
-
-For memory-bound kernels, occupancy (number of concurrent warps) matters more than per-thread register count. Limiting registers per thread allows more warps to be resident:
-
-```cuda
-// Compile-time register budgeting
-// Lower register count -> higher occupancy -> better latency hiding
-
-// Problem 1 winner (rank 1): -maxrregcount=32
-// This allows 64 warps per SM (100% occupancy on SM100)
-// Sufficient for GEMV where each thread does minimal computation
-
-// Problem 1 rank 3: -maxrregcount=45
-// Allows 44 warps per SM (~69% occupancy)
-// More registers per thread for wider vectorized accumulation
-```
-
-The tradeoff in a build system:
-
-```python
-# nvcc compilation with register budgeting
-# In CMakeLists.txt or build script:
-
-# For memory-bound GEMV kernel:
-# nvcc -maxrregcount=32 -arch=sm_100a gemv_kernel.cu -o gemv_kernel
-
-# For compute-bound GEMM kernel:
-# nvcc -arch=sm_100a gemm_kernel.cu -o gemm_kernel  (no limit; needs ~128+ regs)
-
-# Per-kernel register limits in the same translation unit:
-# Use __launch_bounds__ to control per-kernel
-
-# Memory-bound: maximize occupancy
-__global__ void __launch_bounds__(256, 8)  // 256 threads, min 8 blocks/SM
-gemv_kernel(/* ... */) { /* ... */ }
-
-# Compute-bound: maximize register availability
-__global__ void __launch_bounds__(512, 1)  // 512 threads, min 1 block/SM
-gemm_kernel(/* ... */) { /* ... */ }
-```
-
-## Complete NVFP4 GEMV Example
-
-Combining all three techniques for the GPU Mode Hackathon Problem 1:
-
-```cuda
-// Optimized NVFP4 Batched GEMV
-// A: [M, K] NVFP4, B: [1, K] NVFP4, C: [M, 1] FP16
-// Memory-bound: maximize bandwidth utilization
-
-// NVFP4 Batched GEMV: each row processed by THREADS_PER_ROW threads
-// Memory-bound: maximize bandwidth with wide loads and cache policies
-
-#define BLOCK_M 4       // Rows per thread block
-#define THREADS 256
-#define THREADS_PER_ROW (THREADS / BLOCK_M)  // 64 threads per row
-
-__global__ void __launch_bounds__(THREADS, 4)
-nvfp4_gemv_optimized(
-    const uint8_t* __restrict__ A,      // [M, K/2] packed FP4
-    const uint8_t* __restrict__ B,      // [1, K/2] packed FP4
-    const fp8_e4m3* __restrict__ sfa,   // [M, K/16] block scales
-    const fp8_e4m3* __restrict__ sfb,   // [1, K/16] block scales
-    half* __restrict__ C,               // [M, 1] output
-    float global_scale_a, float global_scale_b,
-    int M, int K)
-{
-    // Map threads to rows: 64 threads per row, 4 rows per block
-    int local_row = threadIdx.x / THREADS_PER_ROW;      // 0..3
-    int thread_in_row = threadIdx.x % THREADS_PER_ROW;  // 0..63
-    int row = blockIdx.x * BLOCK_M + local_row;
-    if (row >= M) return;
-
-    float acc = 0.0f;
-
-    // Each of the 64 threads handles a distinct K-chunk (no duplication)
-    // 64 elements per load × 64 threads = 4096 K elements per iteration
-    for (int k = thread_in_row * 64; k < K; k += THREADS_PER_ROW * 64) {
-        // Load B (reused across rows): L1::evict_last, 256-bit
-        uint64_t b_raw[4];
-        asm volatile(
-            "ld.global.L1::evict_last.v4.u64 {%0,%1,%2,%3}, [%4];"
-            : "=l"(b_raw[0]), "=l"(b_raw[1]), "=l"(b_raw[2]), "=l"(b_raw[3])
-            : "l"((const uint64_t*)(B + k / 2)));
-
-        // Load A (streamed once): L1::no_allocate, 256-bit
-        uint64_t a_raw[4];
-        asm volatile(
-            "ld.global.L1::no_allocate.v4.u64 {%0,%1,%2,%3}, [%4];"
-            : "=l"(a_raw[0]), "=l"(a_raw[1]), "=l"(a_raw[2]), "=l"(a_raw[3])
-            : "l"((const uint64_t*)(A + row * (K / 2) + k / 2)));
-
-        // Unpack FP4, apply block scale, dot-product
-        for (int i = 0; i < 64; i++) {
-            float a_val = unpack_fp4(a_raw, i) * get_block_scale(sfa, row, k + i);
-            float b_val = unpack_fp4(b_raw, i) * get_block_scale(sfb, 0, k + i);
-            acc += a_val * b_val;
-        }
-    }
-
-    // Two-phase reduction: first within each warp, then across the 2 warps per row
-    // Phase 1: warp-level reduction (32 threads → 1 partial sum per warp)
-    for (int offset = 16; offset > 0; offset >>= 1) {
-        acc += __shfl_xor_sync(0xFFFFFFFF, acc, offset);
-    }
-
-    // Phase 2: shared memory reduction across the 2 warps assigned to this row
-    __shared__ float smem_reduce[BLOCK_M * 2];  // 2 warp partials per row
-    int warp_in_row = thread_in_row / 32;  // 0 or 1
-    int lane = thread_in_row % 32;
-    if (lane == 0) {
-        smem_reduce[local_row * 2 + warp_in_row] = acc;
-    }
-    __syncthreads();
-
-    // Final sum and store (one thread per row)
-    if (thread_in_row == 0) {
-        float result = smem_reduce[local_row * 2] + smem_reduce[local_row * 2 + 1];
-        C[row] = __float2half(result * global_scale_a * global_scale_b);
-    }
-}
-```
-
-## Optimization Progression (GPU Mode Hackathon Problem 1)
-
-| Step | Technique | Latency | Speedup |
-|------|-----------|---------|---------|
-| Baseline | Naive C++ | 2000 us | 1.0x |
-| Coalesced access | Memory layout fix | 443 us | 4.5x |
-| Hardware intrinsics | FP4 decode | 39 us | 51x |
-| PTX assembly | Vectorized loads + cache policy | 27 us | 74x |
-| ILP + register tuning | Unrolling + maxrregcount | 22.4 us | 89x |
-| Speed of light | | ~8.6 us | 233x |
-
-## When to Use
-
-- **GEMV and memory-bound kernels**: Vectorized loads and cache policies are essential. These kernels are entirely limited by memory bandwidth.
-- **FP4/FP8 kernels**: Sub-byte data types make wide loads even more impactful since more elements fit in a single wide load.
-- **Decode-phase inference**: Single-token GEMV during autoregressive decoding is always memory-bound.
-
-## Caveats
-
-- 256-bit loads require 32-byte aligned addresses. Misaligned access falls back to multiple narrower transactions.
-- `L1::no_allocate` is harmful for data that will be reused. Only apply it to truly streaming access patterns.
-- `-maxrregcount` that is too low causes register spilling to local memory, which is slower than the occupancy gain. Profile with Nsight Compute to find the optimal point.
-- PTX inline assembly bypasses the compiler's register allocator. Excessive inline PTX can interfere with compiler optimizations for surrounding code.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/techniques/warp-specialization/algo-warp-specialization.md

@@ -1,208 +0,0 @@
----
-title: Warp-specialized GEMM mainloop (producer/consumer warpgroups)
-status: verified
-evidence_level: measured
-applies_to_pattern_class:
-- tensor-core
-- tensor-core/gemm
-applies_to_ops:
-- gemm
-requires_sm: '>=9.0a'
-requires_features:
-- tma
-- wgmma
-- mbarrier
-single_kernel_useful: true
-cuda_version_tested: 12.9.86
-driver_version_tested: 570.124.06
-toolchain: nvcc 12.9 + ptxas 12.9
-measured_on: H200-SXM | sm_90a | cuda 12.9.86 | driver 570.124.06
-source:
-- path: wiki/nvidia/hardware/tma/skill-tma-ptx.md
-  anchor: cutlass-free TMA primitive (the producer issues these)
-- path: wiki/nvidia/hardware/wgmma/skill-wgmma-ptx.md
-  anchor: cutlass-free wgmma primitive (the consumer issues these)
-artifacts:
-  code: artifacts/experience/api-probes/gemm/gemm_compare_ws.cu
-  build: artifacts/experience/api-probes/gemm/build_warpspec.sh
-  introspection: artifacts/experience/api-probes/gemm/device.json
-  profile: artifacts/experience/api-probes/gemm/2026-04-28-warp-specialization-ablation.csv
-  ablation: artifacts/experience/api-probes/gemm/2026-04-28-warp-specialization-ablation.csv
-upstream_repo: cutlass@f74fea9c (one possible implementation; see "References")
-related_apis: []
-related_skills:
-- tma-ptx
-- wgmma-ptx
-- gemm-aligned
-- persistent-kernel
-id: algo-warp-specialization
-type: algorithm
-vendor: nvidia
-tags:
-- cuda-cpp
-- wgmma
-- tma
-- mbarrier
-- cluster
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- shared-memory-optimization
-- tma-multicast
-- communication-overlap
-- gemm
-- ptx
-- cute-dsl
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L11680-L11686
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L10720
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/parallel-thread-execution/cuda_parallel-thread-execution_index.html.md
-  anchor: L16316
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/parallel-thread-execution/cuda_parallel-thread-execution_index.html.md
-  anchor: L20810-L20825
-- source_id: blogs/colfax
-  path: developing-cuda-kernels-for-gemm-on-nvidia-hopper-architecture-using-cutlass
-  anchor: warp-specialized GEMM mainloop walkthrough — producer/consumer warpgroups + mbarrier-pipelined TMA→wgmma
-- source_id: blogs/colfax
-  path: cutlass-tutorial-efficient-gemm-kernel-designs-with-pipelining
-  anchor: pipelining strategy for warp-specialized GEMM
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-- tma
-- mbarrier
-- cluster
-techniques:
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- shared-memory-optimization
-- tma-multicast
-- communication-overlap
-kernel_types:
-- gemm
-confidence: experimental
-artifact_dir: artifacts/experience/api-probes/gemm
-scope: "Hopper (sm90a) TMA + wgmma mainloop: it assumes an mbarrier-signalled smem ring buffer and asynchronous wgmma. The 1.26x / 1.79x multicast amplification is measured at cluster C=2 / C=4 on H200-SXM and is not a general figure. On sm100 the producer/consumer split still holds but the consumer instruction is tcgen05, not wgmma -- see the migration page."
----
-# Warp-specialized GEMM mainloop (algorithm skeleton)
-
-## What it is
-
-An *algorithmic pattern* for hiding TMA latency behind wgmma compute on Hopper. The threadblock is split into:
-
-- **Producer warps** — issue TMA loads of A and B tiles into a multi-stage smem ring buffer; signal completion via a per-stage mbarrier.
-- **Consumer warpgroup(s)** — wait on the mbarrier (TMA done), issue `wgmma.mma_async` against the loaded tile, signal back that the smem stage is consumed and may be refilled.
-
-The pattern is independent of cutlass; cutlass is one realization (`MainloopSm90TmaGmmaWarpSpecialized`). The cutlass-free realization composes the primitives from `wiki/nvidia/hardware/tma-ptx` (producer side) and `wiki/nvidia/hardware/wgmma-ptx` (consumer side).
-
-## Why the pattern works
-
-A naive Hopper GEMM mainloop interleaves TMA-load and wgmma-issue serially in one warpgroup. The wgmma engine then stalls every iteration waiting on the next TMA tile. By pinning the TMA issuer to a *separate* warp(s) and letting the consumer keep computing on the previous tile, the TMA latency is hidden behind compute.
-
-Three variants of the pattern, each adding an axis of parallelism:
-
-| Variant | Producer warps | Consumer warpgroups | Per-CTA tile cadence |
-|---|---|---|---|
-| **Plain WS** | 1 warp (or 1 warpgroup) | 1 warpgroup | one tile at a time |
-| **Pingpong** | 1 | 2 (alternating) | while consumer-A computes tile T, consumer-B prepares tile T+1 |
-| **Cooperative** | 1 | 2 (cooperating on same tile) | both consumers split the wgmma rows of one tile; usually paired with cluster-multicast TMA so the producer's tile is broadcast to ≥2 CTAs (see `sources/experience/hw-probes/tma-ptx.md` — measured 1.26× / 1.79× effective-bandwidth amplification at C=2 / C=4) |
-
-(Cutlass exposes these as `KernelTmaWarpSpecialized*` dispatch policies; the names are implementation labels, not algorithm names.)
-
-## When to use it
-
-- Hopper sm_90a GEMM-class kernels where TMA latency is non-trivial relative to wgmma compute (essentially all production GEMMs).
-- Any cutlass-free Hopper kernel that loads with TMA and computes with wgmma — the pattern is the canonical way to overlap them.
-
-## When NOT to use it
-
-- Single-tile-grid problems (problem ≤ one CtaTile in M and N). The producer/consumer split adds coordination overhead with no latency to hide.
-- Kernels that don't use TMA (Ampere `cp.async`, manual `cp.async.bulk` without tensor maps). Without TMA's high-latency-but-async issue, there is nothing to hide.
-
-## Algorithm skeleton (cutlass-free)
-
-The pattern below uses `S` smem stages (typical 2–4) and the TMA / wgmma primitives from the sibling skills. The producer stage `s` and consumer stage `s` use two mbarriers per stage: `bar_full[s]` (producer → consumer) signals the load is complete; `bar_empty[s]` (consumer → producer) signals the smem slot may be refilled.
-
-```
-__shared__ {A_tile[S], B_tile[S], bar_full[S], bar_empty[S]};
-
-if (warp_id == PRODUCER_WARP) {
-    // Initial fill of S stages.
-    for (s = 0; s < S; ++s) {
-        wait(bar_empty[s], phase=0);                  // smem slot s is free
-        arrive_expect_tx(bar_full[s], tile_bytes);
-        cp.async.bulk.tensor.2d(A_tile[s], tensor_map_A, coord(0, s));
-        cp.async.bulk.tensor.2d(B_tile[s], tensor_map_B, coord(0, s));
-    }
-    // Steady state.
-    for (k = S; k < N_TILES_K; ++k) {
-        s = k % S;
-        wait(bar_empty[s], phase=(k/S - 1) & 1);
-        arrive_expect_tx(bar_full[s], tile_bytes);
-        cp.async.bulk.tensor.2d(A_tile[s], tensor_map_A, coord(0, k));
-        cp.async.bulk.tensor.2d(B_tile[s], tensor_map_B, coord(0, k));
-    }
-} else {  // CONSUMER_WARPGROUP — 128 threads of 4 warps
-    accum = {0};
-    for (k = 0; k < N_TILES_K; ++k) {
-        s = k % S;
-        wait(bar_full[s], phase=(k/S) & 1);
-        wgmma.fence;
-        wgmma.mma_async.m64nNk16.f32.bf16.bf16 accum,
-             desc(A_tile[s]), desc(B_tile[s]), 1, 1, 1, 0, 0;
-        wgmma.commit_group;
-        wgmma.wait_group 0;
-        arrive(bar_empty[s]);                         // signal slot s is free
-    }
-    epilogue.store(accum, D);
-}
-```
-
-Pingpong variant: keep two consumer warpgroups; route even-k tiles to consumer-A and odd-k to consumer-B; each consumer touches ⌈N_TILES_K/2⌉ tiles. Both consumers share one producer.
-
-Cooperative variant: keep two consumer warpgroups, both wait on the same `bar_full[s]`; consumer-A handles wgmma rows [0..31] of the m64 atom, consumer-B rows [32..63]; both arrive on `bar_empty[s]` (= 2 expected arrivals per slot). Pair with cluster-multicast TMA (`cp.async.bulk.tensor.shared::cluster.global.tile.bulk_group.cta_group::2`) so the producer's load broadcasts to both CTAs in a 2-CTA cluster — single load feeds two consumers' wgmma.
-
-`tile_bytes` per stage = `(BOX_M*BOX_K + BOX_K*BOX_N) * sizeof(elem)`. mbarrier expected-tx counts both A and B in one stage; the producer issues two TMA loads but the barrier expects the sum.
-
-## Implementations on disk
-
-- **Cutlass realization** — `cutlass::gemm::kernel::sm90_gemm_tma_warpspecialized.hpp` (`KernelTmaWarpSpecialized`), `_pingpong.hpp` (Pingpong), `_cooperative.hpp` (Cooperative). Cluster-shape constraints, EpilogueSchedule co-constraints, and stage-count autocarvers are cutlass-implementation details; see `pitfalls.md`.
-- **Cutlass-free building blocks** — `wiki/nvidia/hardware/tma/skill-tma-ptx.md` (producer's `cp.async.bulk.tensor.*` + mbarrier protocol), `wiki/nvidia/hardware/wgmma/skill-wgmma-ptx.md` (consumer's `wgmma.mma_async.*` family). The cutlass-free GEMM at `wiki/nvidia/foundations/compute/gemm-ptx/` composes them but is not yet warp-specialized; adding the producer/consumer split following the skeleton above is the natural extension.
-
-## Measured Characteristics
-
-H200-SXM, sm_90a, CUDA 12.9.86. The algorithm has been validated in the cutlass realization at problem 2048 × 2048 × 2048 in TF32 (the cutlass-free realization is queued as a follow-up). Numbers below characterize the *algorithm pattern* on H200, observed via the cutlass implementation:
-
-| Variant | Latency (μs) | TFLOPS | cuBLAS diff | Note |
-|---|---|---|---|---|
-| Serial baseline (no warp-spec) | 96.02 | 178.9 | bit-identical | One warpgroup interleaves TMA + wgmma |
-| Plain WS | 87.94 | 195.4 | bit-identical | Producer/consumer split, no persistence |
-| Pingpong | 88.64 | 193.8 | bit-identical | 2 consumer warpgroups alternating tiles, persistent |
-| Cooperative | 91.92 | 186.9 | bit-identical | 2 consumers cooperating on same tile + cluster-multicast |
-
-**On/off A/B at 2048³**: enabling warp-specialization (serial → plain WS) gives **+9.2% throughput** (178.9 → 195.4 TFLOPS) at bit-identical correctness. The improvement comes entirely from hiding TMA latency behind wgmma compute via the producer/consumer pipeline.
-
-At 2048³ all three WS variants are within 5% of each other; plain WS leads narrowly. At 8192³ (cooperative + cluster-multicast amortizes its setup) the cooperative variant climbs to 292.7 TFLOPS — see `wiki/nvidia/foundations/compute/gemm.md` for the larger-shape data.
-
-## Cross-references
-
-- TMA-PTX primitive (cutlass-free producer side): `wiki/nvidia/hardware/tma/skill-tma-ptx.md` — measured throughput sweep at `sources/experience/hw-probes/tma-ptx.md`.
-- wgmma-PTX primitive (cutlass-free consumer side): `wiki/nvidia/hardware/wgmma/skill-wgmma-ptx.md` — atom zoo at `sources/experience/hw-probes/wgmma-ptx.md`.
-- Aligned GEMM consumer: `wiki/nvidia/foundations/compute/gemm.md`
-- Persistent-kernel sibling: `wiki/nvidia/techniques/persistent-kernels.md`
-- Cutlass example 48 source-reading notes: `wiki/nvidia/code-walkthroughs/cutlass-cute/example48-hopper-warp-specialized-gemm/{README.md, mainloop_skeleton.md}`
-- Probe + ablation: `sources/experience/api-probes/gemm.md`
-- Failure modes: `wiki/nvidia/techniques/warp-specialization/pitfalls.md`

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/techniques/warp-specialization/pitfalls.md

@@ -1,87 +0,0 @@
----
-id: algo-warp-specialization-pitfalls
-type: algorithm
-vendor: nvidia
-title: Pitfalls
-tags:
-- cuda-cpp
-- wgmma
-- tma
-- mbarrier
-- cluster
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- shared-memory-optimization
-- tma-multicast
-- gemm
-- attention
-- ptx
-evidence_level: spec
-source:
-- path: spec
-  anchor: Algorithm reference
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-hardware_features:
-- wgmma
-- tma
-- mbarrier
-- cluster
-techniques:
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- shared-memory-optimization
-- tma-multicast
-kernel_types:
-- gemm
-- attention
-confidence: source-reported
-scope: "H200-SXM, sm90a, production tile sizes at 8-32 KiB of smem per stage. The 3-4 stage sweet spot is a smem-budget-versus-TMA-latency result for that part and must be re-budgeted after any change to tile size, dtype, or epilogue smem footprint. The expected_tx, phase-parity and pingpong-balance items are structural and hold wherever the pattern is used."
----
-# Warp-specialized GEMM mainloop — pitfalls
-
-## 1. Stage count is a smem-budget × TMA-latency-coverage trade-off
-
-Each smem stage holds one tile's A + B in shared memory plus its mbarrier pair. More stages hide longer TMA latency but cost smem (typically 8–32 KiB per stage at production tile sizes). Two stages is the minimum that overlaps producer with consumer at all; 3–4 is the typical sweet spot on H200; beyond 4 is rarely justified — you're hiding latency you don't have. If you change tile size, dtype, or epilogue smem footprint, re-budget the stage count.
-
-## 2. mbarrier `expected_tx` must equal the *sum* of A and B tile bytes
-
-The producer issues two TMA loads per stage (A tile + B tile) but only signals one `mbarrier.arrive.expect_tx`. The expected-tx count must be `(BOX_M*BOX_K + BOX_K*BOX_N) * sizeof(elem)`, not just one of the two. Get this wrong and the consumer wakes either too early (one load not done — corrupt result) or never (over-counted — kernel hangs).
-
-## 3. Phase tracking on each mbarrier alternates 0,1,0,1,... per cycle through stages
-
-For S smem stages, barrier `bar_full[s]` flips its parity once per use of stage s. The consumer wait must use phase `(k / S) & 1` for iteration k; the producer must wait on `bar_empty[s]` with `((k - S) / S) & 1`. A constant phase causes back-to-back iterations to wake on the same flip — silent correctness bug since the data is from the *previous* iteration of stage s.
-
-## 4. Pingpong assumes the K-loop is uniform across tiles
-
-The pingpong variant routes alternating tiles to consumer-A and consumer-B. For this to balance, every tile must have the same K-loop count (= same N_TILES_K). If different tiles have different K (e.g. masked-out rows in attention), pingpong loses balance and one consumer stalls. Use plain WS or cooperative for non-uniform per-tile work.
-
-## 5. Cooperative needs cluster-multicast TMA
-
-The cooperative variant keeps two consumers on the same tile, halving per-consumer wgmma work. Without cluster-multicast TMA (`cp.async.bulk.tensor.shared::cluster.global.tile.bulk_group.cta_group::2`), the cluster's two CTAs each issue their own load — twice the DRAM bandwidth for the same data. Always pair cooperative with cluster ≥ `<2, 1, 1>` and the multicast variant of the TMA instruction. Otherwise cooperative has higher coordination cost than plain WS for no win.
-
-## 6. Plain WS at small clusters does not always beat cooperative
-
-Measured on H200 at 2048³: plain WS (`<1, 1, 1>` cluster) reaches 195.4 TFLOPS; cooperative with multicast (`<4, 2, 1>` cluster) reaches 186.9 TFLOPS — plain wins. Cooperative + multicast pays a fixed coordination cost per cluster that does not amortize until the problem is large enough to fill many cluster grids. For problems ≤ a few clusters' worth of CTAs in M × N, plain WS is the better default. Don't reflexively reach for cooperative because it's the most-recent variant.
-
-## 7. Producer must `wgmma.wait_group 0` before re-issuing TMA into a stage
-
-This is implicit if the producer and consumer are different warps and the mbarrier protocol is correct, but if you collapse them onto the same warp (e.g. for a degenerate single-stage debug build), you must explicitly fence the wgmma completion before the next TMA into the same smem slot — otherwise the new load races with the still-running wgmma's smem reads.
-
-## 8. Cutlass implementation specifics (KernelTmaWarpSpecialized*)
-
-These are pitfalls of cutlass's *implementation* of the algorithm, not of the algorithm itself. Listed here for reference because cutlass's variant naming is widely cited:
-
-- The cutlass `(Cluster, Schedule)` tuple is co-constrained — `KernelTmaWarpSpecializedPingpong` requires cluster `<2, 1, 1>`; `Cooperative` requires `<4, 2, 1>`. Mismatched tuples either fail `gemm.can_implement` or run at degraded throughput.
-- There is no flag to disable warp-spec on a `KernelTmaWarpSpecialized*` policy; the on/off A/B requires building two binaries (`KernelTma` for off, `KernelTmaWarpSpecialized` for on).
-- `EpilogueSchedule` must match the mainloop's persistence model — pingpong/cooperative epilogues are persistent-aware (`TmaWarpSpecialized*`); plain WS uses `NoSmemWarpSpecialized`. `KernelScheduleAuto` enforces this; explicit-schedule callers must enforce it themselves.
-- Pipeline depth is encoded as the `Stages` template parameter; `StageCountAutoCarveout` computes the maximum stages that fit given the epilogue smem footprint. Recompute when you change epilogue / dtype.
-- `KernelScheduleAuto` is shape-dependent — two callers with the same shape but different SM targets get different schedules. For reproducibility, set the schedule explicitly.
-
-A cutlass-free implementation (composing `wiki/nvidia/hardware/{tma-ptx, wgmma-ptx}/`) skips all of the above by making each of these knobs an explicit kernel parameter or template argument.