@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/operator-routing/elementwise/ROUTING.md

@@ -1,178 +0,0 @@
----
-title: Elementwise Skill Routing
-pattern_class: cuda-core
-op: elementwise
-id: routing-elementwise-ROUTING
-type: operator-routing
-vendor: nvidia
-operator: elementwise
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- register-budgeting
-- data-reuse
-- kernel-fusion
-- loop-unrolling
-- shared-memory-optimization
-- software-exp
-kernel_types:
-- fused-kernel
-- attention
-confidence: inferred
-evidence_level: inferred
-tags:
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- register-budgeting
-- data-reuse
-- kernel-fusion
-- loop-unrolling
-- shared-memory-optimization
-- software-exp
-- fused-kernel
-- attention
-- cuda-cpp
----
-# Elementwise -- Skill Whitelist
-
-This file lists the optimization skills applicable to custom elementwise kernels, in priority order. Only skills that exist under `wiki/nvidia/foundations/` are referenced.
-
----
-
-## Priority 1: coalescing (critical)
-
-- **Skill**: `wiki/nvidia/foundations/memory/coalescing/`
-- **Why**: Elementwise kernels are purely memory-bound. Coalescing is the single most important optimization. A stride-1 access pattern ensures that each warp's 32 loads/stores are serviced in the minimum number of 32-byte transactions.
-- **When**: Always. Every elementwise kernel must verify that its thread indexing produces coalesced access. The canonical pattern is `tid = blockIdx.x * blockDim.x + threadIdx.x; data[tid]`.
-- **Measured impact**: On H200, stride-32 (non-coalesced) access is 2.85x slower than stride-1 (coalesced) access (see coalescing skill measured characteristics).
-
----
-
-## Priority 2: bank-conflict (conditional)
-
-- **Skill**: `wiki/nvidia/foundations/memory/bank-conflict/`
-- **Why**: If an elementwise kernel stages data through `__shared__` memory (for example, to reorganize a non-contiguous tensor into a contiguous layout before applying the pointwise function), bank conflicts in the shared memory access can degrade performance.
-- **When**: Only when the kernel uses `__shared__` memory. Most pure elementwise kernels do NOT use shared memory (there is no data reuse across threads), so this skill is rarely needed. Apply it only if profiling or code inspection reveals shared memory staging.
-- **Measured impact**: On H200, 32-way bank conflicts cause 3.03x latency increase on shared memory loads (see bank-conflict skill measured characteristics).
-
----
-
-## Priority 3: warp-primitives (rare)
-
-- **Skill**: `wiki/nvidia/foundations/compute/warp-primitives/`
-- **Why**: Pure elementwise kernels have no intra-warp communication needs -- each thread operates independently. However, if an elementwise kernel includes a reduction epilogue (e.g., computing a norm or a sum alongside the elementwise operation), warp shuffles can accelerate that reduction phase.
-- **When**: Only when the kernel fuses an elementwise operation with a partial reduction or when a warp-level predicate check (`__any_sync`, `__all_sync`) is used for early-exit logic (e.g., checking for NaN values across a warp).
-- **Measured impact**: Warp butterfly reduction takes ~145 cycles for 5 steps on H200 (see warp-primitives skill measured characteristics).
-
----
-
-### Priority 4: Layout Transform (pre-kernel fix when the input is stride-unfriendly)
-
-- **Skill**: `wiki/nvidia/foundations/memory/layout-transform/`
-- **Why**: coalescing (Priority 1) is a kernel-side fix for the stride-1 case. When the input is a struct-of-fields (AoS) and the elementwise kernel touches only a subset of fields, or when a 2-D tensor's row width is not a multiple of 128 B, the kernel cannot *become* stride-1 from inside — the layout must change before the kernel runs. Sub-skill S1 (AoS → SoA) is the primary relevance; S2 (`cudaMallocPitch`) applies when 2-D rows are misaligned.
-- **When**: the upstream data producer supplies AoS but the elementwise kernel reads a single field (or a subset of fields with low locality to each other). For an N-field struct where each elementwise kernel uses k fields, break-even is roughly `k/N × downstream_kernel_count` calls. Measured on H200 (24-B `Particle` struct, kernel reads `.x` only): AoS→SoA conversion amortizes at **3.65 downstream calls** (probe: `sources/experience/hw-probes/aos-vs-soa/`).
-- **When NOT**: all downstream kernels touch all struct fields (AoS and SoA are equivalent at the HBM level and SoA can be worse via cache fragmentation); single-use data that sees only one downstream kernel; rows already 128-B-aligned (skip S2).
-- **Measured impact**: SoA read-one-field is **1.99×** faster than AoS read-one-field on H200; SoA + float4 vectorization is **2.99×** total. The `aos_to_soa_convert` kernel itself runs at 81 % HBM SoL.
-
-### Priority 5: Vectorized Access
-
-- **Skill**: `wiki/nvidia/foundations/memory/vectorized-access/`
-- **Why**: float4 vectorized loads achieve 2.59× bandwidth over scalar loads on H200 (3784 vs 1463 GB/s). For memory-bound elementwise kernels this is the single biggest optimization after coalescing.
-- **When**: input arrays are 16-byte aligned and N is divisible by 4. Use `reinterpret_cast<float4*>` or native vector types.
-
-### Priority 6: Warp Divergence (relevant for fused activations / masked writes)
-
-- **Skill**: `wiki/nvidia/foundations/compute/warp-divergence/`
-- **Why it matters for elementwise**: fused kernels that combine a pointwise activation with a conditional branch (ReLU, GELU-select, masked write, clipped gradient) can hit divergence cost if the body is too large for the compiler to predicate. On a compute-bound fused-activation kernel on H200, divergent branching measured **1.82× slower** at any p ∈ (0, 1) vs uniform warps; short-body predicated rewrite is **flat** across p at ~1.07× — elementwise pipelines should always prefer ternary/`fmax`/predicated forms.
-- **When to apply**:
-  - Rewrite ReLU-style gates as `fmaxf(x, 0.f)` (branchless).
-  - Use ternary for small conditional writes: `out[i] = cond ? v : out[i]`.
-  - For longer conditional bodies (inline function with > ~7 instructions, or any function call), either split into two kernels by input mask, or accept the predicated 2× cost if p is unpredictable per-lane.
-- **When NOT to apply**: elementwise kernels that are memory-bound (the common case: H200 HBM-bound copy-with-activation). On a memory-bound kernel, divergence cost is hidden behind the memory wait — do not churn the code for a metric that won't move. Measured: same kernel shape at N=16M was flat across p on all three variants.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — "Compute-bound elementwise with divergent conditional on hot path."
-
-### Priority 7: Fast Math Intrinsics
-
-- **Skill**: `wiki/nvidia/foundations/compute/fast-math/`
-- **Why**: `__expf` is 1.27× faster than `expf` on H200 (3318 vs 2604 Gop/s). Relevant for activation functions (gelu, silu, sigmoid) and normalization (rsqrtf).
-- **When**: compute involves transcendentals AND relaxed precision is acceptable (not for loss/gradient computation).
-
-### Priority 8: Instruction-Level Parallelism (ILP)
-
-- **Skill**: `wiki/nvidia/foundations/compute/ilp/`
-- **Why**: processing multiple elements per thread via grid-stride loop with unrolling exposes ILP, hiding FMA pipeline latency (4 cycles on H200). 4-acc chains saturate the pipeline.
-- **When**: kernel is latency-bound (low occupancy or few instructions per element). Memory-bound elementwise kernels gain less from ILP.
-
-### Priority 9: Compiler Hints
-
-- **Skill**: `wiki/nvidia/foundations/compute/compiler-hints/`
-- **Why**: `__launch_bounds__`, `__restrict__`, `#pragma unroll` guide register allocation and code generation. `__restrict__` enables the compiler to auto-route loads through the read-only cache.
-- **When**: tuning phase after functional correctness.
-
-### Priority 10: Register Pressure Management
-
-- **Skill**: `wiki/nvidia/foundations/memory/register-pressure/`
-- **Why**: vectorized access + ILP increase register usage. Forced reduction (--maxrregcount=32) caused 4.84× slowdown from spilling on H200. Monitor with `-Xptxas=-v`.
-- **When**: after applying skills 4-6, if occupancy drops below 50%.
-
-### Priority 11: Async Copy (LDGSTS Prefetching)
-
-- **Skill**: `wiki/nvidia/foundations/memory/async-copy/`
-- **Why**: for compute-heavy fused elementwise chains (e.g., bias + gelu + scale), LDGSTS async prefetching to shared memory increases bytes-in-flight without consuming registers.
-- **When**: ONLY when the elementwise computation is non-trivial (involves multiple transcendentals or long dependency chains). GTC25-S72683 demonstrated that trivial `a*b` elementwise gets NO benefit (actually 10% regression on H200). The "Stall Long Scoreboard" metric in ncu must be the dominant bottleneck.
-- **Caveat**: most pure elementwise kernels are already bandwidth-saturated with coalescing + vectorized access. Async copy adds code complexity for marginal gain in the common case.
-
-### Priority 12: Cache Load Hints (mostly a null on H200; avoid the traps)
-
-- **Skill**: `wiki/nvidia/foundations/memory/cache-load-hints/`
-- **Why it matters for elementwise**: elementwise kernels are the canonical "read-only input, stream through once" shape — the exact pattern the legacy KB suggested `__ldg` for. Measured on H200 sm_9.0a: at DRAM-bound elementwise scale (buffer > 60 MiB L2, single pass), **all six cache-hint variants are within 0.5%** of default. The hint choice is free and irrelevant; time spent picking one is wasted.
-- **When to apply**:
-  - Use `const __restrict__` on the input pointers and the compiler auto-emits `ld.global.nc` — same result as explicit `__ldg` with less code.
-  - For chained / fused elementwise that re-reads the input across multiple passes in one kernel (rare but occurs in e.g. residual-add-then-gelu-then-scale), default-load. **Measured 2.26× slowdown** if `__ldcg` or `__ldcv` is used on L2-resident data with SM-local reuse (skill §S3).
-- **When NOT to apply**:
-  - Never use `__ldcg` / `__ldcv` on elementwise inputs. The 2.26× L2-reuse penalty hits any chained elementwise kernel where the input is touched by more than one pass.
-  - `__ldcs` (streaming / evict-first) is a measured null in un-contended L2 — elementwise MVPs have no competing workload, so no benefit. Default-load.
-- **Measured impact on H200**: none in single-kernel elementwise. Benefit only materializes in contended-L2 scenarios not reached by MVP harnesses; see `sources/experience/hw-probes/cache-hint.md`.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — agent reports "tried `__ldcg` and slower". Correct response: revert to default load and fix pointer qualifiers (`const __restrict__`).
-
-### Priority 13: Half-Precision Math (fp16 / bf16 scalar + packed)
-
-- **Skill**: `wiki/nvidia/foundations/compute/half-precision-math/`
-- **⚠ PRECISION IMPACT — read before applying**: fp16 max finite ≈ **6.55e4** and ULP at 1.0 ≈ **4.88e-4** (~3 decimal digits); bf16 has fp32's range but ULP at 1.0 ≈ **7.81e-3** (~2 decimal digits). Choosing fp16/bf16 is a **correctness decision**, not just a performance decision — see skill's §Precision. Rule: format choice first (range vs precision), packing second.
-- **Why it matters for elementwise**: elementwise is the canonical shape that benefits most from half precision — the kernel is memory-bound, so halving the byte footprint gives an immediate **2× effective bandwidth**. The compute-throughput gain (1.58×–1.84× vs fp32, measured) is secondary but real for fused transcendental chains (gelu, silu, activation+scale+add).
-- **When to apply**:
-  - **Pure elementwise (bandwidth-bound)**: load/store in fp16 or bf16 — halves DRAM traffic. Do the fused compute in whatever precision preserves correctness (skill §S1): if the op is a straight map with no sum (e.g. `out = a * scale + bias`), compute in fp16/bf16 directly. If the op accumulates (attention softmax numerator, reduction-in-elementwise-disguise), **promote to fp32** for the accumulator (skill §S3).
-  - **Compute-heavy fused chains (gelu, silu, bias+act+scale+add)**: consider packed `__hfma2`. Measured on H200: only **1.16× over scalar `__hfma`** (not 2× as folklore claims; legacy P14 upgraded to measured). Worth the complexity only when the chain has ≥ 50 FMAs per pack and the alignment is natural.
-  - **Precision-sensitive operations**: if any intermediate is compared against a small threshold or fed into a transcendental near a discontinuity, use fp32 regardless of memory cost. See elementwise INDEX.md new dtype decision step.
-- **When NOT to apply**:
-  - **Any long-running accumulation without fp32 promotion** (softmax denominator over H=4096, dot products, running averages). fp16 overflows at ~6e4; bf16 mantissa error ~0.8% per add. Skill P1, P2.
-  - **Pure-streaming memory-bound kernel where packing adds alignment fragility**. The 1.16× compute gain is invisible behind memory stalls (P11). Use scalar `__hadd` / `__hfma`.
-  - **Framework-built CUDA extensions** without explicit `__hadd` / `__hgt` calls may define `-D__CUDA_NO_HALF_OPERATORS__` (P13).
-- **Measured impact on H200 sm_9.0a**:
-  - fp16 scalar = 1.58× fp32 scalar throughput (45 vs 29 TFLOPS).
-  - fp16 packed (`__hfma2`) = 1.84× fp32 scalar (1.16× over fp16 scalar).
-  - bf16 packed = 12% slower than fp16 packed (46 vs 53 TFLOPS).
-  - See `sources/experience/hw-probes/half2-throughput.md`.
-- **Relevance to bottleneck triage**: Q3 in `reasoning/bottleneck-triage.md` — elementwise kernel with fp16 input and suspected overflow in a fused reduction-like step. Correct response: promote accumulator to fp32, cast at store.
-
-### Priority 14: Branch Elimination (use intrinsics; do NOT simulate select with arithmetic)
-
-- **Skill**: `wiki/nvidia/foundations/compute/branch-elimination/`
-- **Why it matters for elementwise**: many elementwise kernels contain simple conditionals — ReLU (`max(x, 0)`), clamp, abs, guarded writes (`if (mask[i]) dst[i] = v`). On H200 sm_9.0a the `if/else` and ternary forms of these simple-body conditionals already lower to `FSEL` (SASS predicated-select) — they are branchless at the hardware level. The real win comes from using the **single-op hardware intrinsic** where one exists: `fmaxf` / `fabsf` / `fminf` emit 1 SASS op instead of the 2 ops a hand-written `if/else` produces.
-- **When to apply**:
-  - Rewrite ReLU as `fmaxf(x, 0.0f)` — measured **1.56× faster** than `(x < 0) ? 0 : x` (one `FMNMX` vs two-op `FSETP + FSEL`).
-  - Rewrite `if (x<0) x=-x` as `fabsf(x)` — measured **2.36× faster** (one `LOP3.LUT` vs two-to-three-op branch form). Also faster than the explicit bit-trick `b & 0x7FFFFFFF` (1.51×).
-  - Keep `if (cond) acc = v;` and `acc = cond ? v : acc;` as-is for register-level conditional assignment — both emit `FSEL` (1 op) and read more clearly than any rewrite.
-- **When NOT to apply**:
-  - Do NOT rewrite `if/else` as `a * (1-c) + b * c` — measured **1.27× slower** on H200 (4 FP ops vs 1 FSEL). This is the most common anti-pattern.
-  - Do NOT use explicit bit tricks for abs / relu — the compiler synthesises `fabsf` to one `LOP3.LUT` but hand bit tricks emit two LOP3s because the int↔float round-trip prevents fusion.
-  - When the `if/else` body is non-trivial (function call, memory store, long expression), a real branch (`BRA`) is emitted and divergence cost becomes real — that is the `warp-divergence` skill's regime, not this one.
-- **Measured impact on H200 sm_9.0a**: 1.56× speedup (ReLU via `fmaxf`), 2.36× (abs via `fabsf`), 0.79× (if you arithmetic-simulate a select — slower). See `sources/experience/hw-probes/branchless-patterns.md`.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — elementwise kernel whose hot path includes `(x < 0) ? 0 : x` or `if (cond) dst[i] = ...`. Check SASS: if it is already `FSEL`, the branch doesn't exist; focus elsewhere. If it is `BRA`, simplify the body so predication kicks in.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/operator-routing/gemm/INDEX.md

@@ -1,161 +0,0 @@
----
-title: Tensor-core GEMM Pattern -- Decision Tree
-pattern_class: tensor-core
-op: gemm
-status: draft
-hardware:
-  device: H200
-  sm: 9.0a
-source:
-- path: wiki/nvidia/techniques/warp-specialization.md
-  anchor: Warp-specialized GEMM mainloop (algorithm skeleton)
-  excerpt: Plain WS / Pingpong / Cooperative variants; producer/consumer split with mbarrier full/empty pair
-- path: wiki/nvidia/foundations/compute/gemm-ptx.md
-  anchor: Hopper GEMM via raw PTX (cutlass-free)
-  excerpt: TMA-PTX + wgmma-PTX composed into a single GEMM kernel; cutlass-free preprocessor + linked-binary gates
-- path: wiki/nvidia/foundations/compute/gemm.md
-  anchor: Aligned GEMM on Hopper via cutlass cooperative warp-specialized kernel
-  excerpt: Cutlass-API track; aligned (M,N,K) divisible by wgmma atom; fast-path mainloop
-- path: sources/experience/kernel-records/2026-04-29-gemm-ws-ptx.md
-  anchor: Cutlass-free warp-specialized GEMM (record)
-  excerpt: Producer warp + consumer warpgroup composed from tma-ptx + wgmma-ptx + warp-specialization skill
-id: routing-gemm-INDEX
-type: operator-routing
-vendor: nvidia
-operator: gemm
-architectures:
-- sm90
-- sm90a
-- sm100
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-- tma
-- mbarrier
-- cluster
-- fp8
-- nvfp4
-techniques:
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- kernel-fusion
-- shared-memory-optimization
-- tma-multicast
-- software-exp
-kernel_types:
-- gemm
-- attention
-- fused-kernel
-- quantization
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- tma
-- mbarrier
-- cluster
-- fp8
-- nvfp4
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- kernel-fusion
-- shared-memory-optimization
-- tma-multicast
-- software-exp
-- gemm
-- attention
-- fused-kernel
-- quantization
-- ptx
-- cuda-cpp
-- cute-dsl
----
-# Tensor-core GEMM Pattern -- Decision Tree
-
-This document guides the kernel-writing agent through a Hopper tensor-core GEMM task that requires a dedicated kernel implementation. It picks between the cutlass-API track (`wiki/nvidia/foundations/compute/gemm/aligned`) and the cutlass-free PTX track (`wiki/nvidia/foundations/compute/gemm-ptx`).
-
-## Scope
-
-This decision tree covers custom-kernel implementation choices only. It starts after the task has been classified as requiring a dedicated kernel implementation.
-
-## Step 1 -- Choose the custom track
-
-Two tracks live under this pattern, distinguished by the dependency policy:
-
-```
-Q4. Is cutlass + cute available as a build-time and link-time dependency?
-    YES --> CUTLASS-API track. See `wiki/nvidia/foundations/compute/gemm.md`
-            (aligned shapes) or `wiki/nvidia/foundations/compute/gemm/non-aligned-tail/skill.md`
-            (M / N not multiples of the wgmma atom -- adds a tail-handling block).
-            For fused epilogues that need extension beyond the cutlass catalogue,
-            see `wiki/nvidia/foundations/compute/gemm-fused/cutlass-epilogue-prologue/skill.md`.
-            Continue to Step 2 to pick the warp-spec variant.
-
-    NO  --> CUTLASS-FREE PTX track. The kernel must contain zero
-            `cutlass::` / `cute::` symbols at both the preprocessed source
-            and linked-binary level. See `wiki/nvidia/foundations/compute/gemm-ptx.md`
-            for the single-tile composition; see
-            `sources/experience/kernel-records/2026-04-29-gemm-ws-ptx.md`
-            for a working warp-specialized extension that adds a producer-warp
-            + consumer-warpgroup split and a multi-K-tile pipeline.
-            Continue to Step 2 to pick the warp-spec variant -- the
-            algorithm skeleton in `wiki/nvidia/techniques/warp-specialization`
-            is dependency-agnostic.
-```
-
-The cutlass-free track exists for one specific reason: **the kernel must be auditable without pulling cute headers** (compliance / supply-chain audit / building a kernel inside a project that cannot accept cutlass as a dependency). Any time cutlass is allowed, the cutlass-API track is the engineering default -- it saves thousands of lines and gets the cooperative + cluster-multicast variants for free.
-
-## Step 2 -- Pick the warp-specialization variant
-
-The Hopper GEMM mainloop is a producer/consumer pipeline regardless of which track. Three variants from `wiki/nvidia/techniques/warp-specialization.md`:
-
-```
-Q5. How large is the M*N grid relative to the H200 SM count (132)?
-
-    Small (problem fits in <= a few SM-fulls of CTAs):
-        --> Plain WS. 1 producer warp + 1 consumer warpgroup per CTA.
-            Cluster `<1,1,1>`, no persistence. Lowest coordination cost
-            and the leader at small problems (measured 2048^3 H200:
-            195.4 TFLOPS plain WS vs 186.9 TFLOPS cooperative).
-
-    Medium (uniform per-tile K, room to alternate work):
-        --> Pingpong. 1 producer + 2 alternating consumer warpgroups.
-            Persistent. Even-k tiles to consumer-A, odd-k to consumer-B.
-            Requires UNIFORM K across all tiles (WS pitfall #4).
-
-    Large (problem is many cluster-grids worth; per-tile work is heavy):
-        --> Cooperative + cluster-multicast.
-            1 producer + 2 cooperating consumers; each consumer takes
-            half the wgmma rows of the same tile. REQUIRES cluster
-            >= <2,1,1> AND the multicast variant of the bulk-tensor
-            mnemonic (`cp.async.bulk.tensor.shared::cluster.global
-            .tile.bulk_group.cta_group::2`). Without multicast,
-            cooperative is strictly worse than plain WS (pitfall #5).
-            Cooperative wins at 8192^3 (292.7 TFLOPS measured).
-```
-
-The minimum-viable variant is **plain WS**. The cutlass-free working artifact in `sources/experience/kernel-records/2026-04-29-gemm-ws-ptx/` realizes plain WS at one CTA + 5 warps (4 consumer + 1 producer); pingpong / cooperative are mechanical extensions of that scaffolding.
-
-## Step 3 -- Optimization via ROUTING.md skills
-
-After the basic kernel is correct (cutlass-free preprocessor + linked-binary gate, OR task numeric gate), apply the optimization skills from `ROUTING.md` in priority order. The hot levers for tensor-core GEMM are:
-
-1. **TMA pipeline depth** -- producer's stage count. 2 is the minimum that overlaps; 3-4 the H200 sweet spot; >4 rarely justified.
-2. **Multi-warpgroup consumers** -- pingpong / cooperative variants for problems that amortize their setup.
-3. **Cluster-multicast TMA** -- cooperative-only; broadcasts one TMA load to two CTAs.
-4. **Persistent kernel scheduling** -- pair with pingpong / cooperative; one CTA processes many output tiles serially, hiding launch overhead.
-
-After each optimization, re-benchmark against the task-provided baseline and follow the bottleneck triage in `reasoning/bottleneck-triage.md`.
-
-## Cross-references
-
-- **Skill whitelist for this pattern**: `ROUTING.md`
-- **Task packet template**: `TASK-PACKET.md`
-- **Bottleneck triage after benchmarking**: `reasoning/bottleneck-triage.md`
-- **Algorithm skeleton (variant-independent)**: `wiki/nvidia/techniques/warp-specialization.md`
-- **Cutlass-free working kernel record**: `sources/experience/kernel-records/2026-04-29-gemm-ws-ptx/`

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/operator-routing/gemm/ROUTING.md

@@ -1,194 +0,0 @@
----
-title: Tensor-core GEMM Pattern -- Skill Routing
-pattern_class: tensor-core
-op: gemm
-status: draft
-id: routing-gemm-ROUTING
-type: operator-routing
-vendor: nvidia
-operator: gemm
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-- tma
-- mbarrier
-- cluster
-techniques:
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- vectorized-loads
-- register-budgeting
-- kernel-fusion
-- shared-memory-optimization
-- swizzling
-- tma-multicast
-kernel_types:
-- gemm
-- fused-kernel
-- quantization
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- tma
-- mbarrier
-- cluster
-- warp-specialization
-- persistent-kernel
-- pipeline-stages
-- vectorized-loads
-- register-budgeting
-- kernel-fusion
-- shared-memory-optimization
-- swizzling
-- tma-multicast
-- gemm
-- fused-kernel
-- quantization
-- ptx
-- cuda-cpp
-- cute-dsl
----
-# Tensor-core GEMM Pattern -- Skill Whitelist
-
-This file lists the optimization skills applicable to a Hopper tensor-core GEMM kernel, in recommended application order. Only skills with a completed `skill.md` under `wiki/nvidia/foundations/`, `wiki/nvidia/hardware/`, or `wiki/nvidia/techniques/` are listed.
-
-The pattern has two implementation tracks (see `INDEX.md` Step 1):
-
-- **Cutlass-API track** -- consumes cutlass + cute as a header dependency.
-- **Cutlass-free PTX track** -- contains zero `cutlass::` / `cute::` symbols at both the preprocessor and linked-binary level.
-
-Each entry below marks which track(s) it applies to. Track-agnostic skills (most algorithm-level patterns) apply to both.
-
-## Applicable skills
-
-### 1. Warp-specialization mainloop (primary, track-agnostic)
-
-- **Skill path**: `wiki/nvidia/techniques/warp-specialization/`
-- **Why it matters for GEMM**: hides TMA latency behind wgmma compute by pinning the TMA issuer to a separate warp / warpgroup. The producer/consumer split is the canonical Hopper GEMM mainloop; serial (non-WS) interleaving stalls every iteration on the next TMA tile. Cutlass realizes this as `MainloopSm90TmaGmmaWarpSpecialized*`; the cutlass-free realization composes the producer side from skill 6 and the consumer side from skill 7.
-- **When to apply**: every Hopper GEMM. Variant choice (plain / pingpong / cooperative) follows `INDEX.md` Step 2; plain is the safe default below ~a few cluster-grids of CTAs in M*N.
-- **Tracks**: both.
-- **Measured on H200**: +9.2 % TFLOPS at 2048^3 vs serial baseline (178.9 -> 195.4 TFLOPS), bit-identical correctness; cooperative + multicast climbs to 292.7 TFLOPS at 8192^3.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "wgmma issue port stalls every iteration on TMA load completion."
-
-### 2. Persistent kernel scheduling (primary, pair with WS)
-
-- **Skill path**: `wiki/nvidia/techniques/persistent-kernel/`
-- **Why it matters for GEMM**: a persistent CTA processes many output tiles serially, amortizing kernel-launch cost and stabilizing wgmma issue-port utilization across tile boundaries. Pingpong and cooperative warp-spec variants are persistent by design.
-- **When to apply**: pair with pingpong / cooperative (skill 1's medium / large branches). Plain WS does not persist; for small problems the launch-overhead amortization is the wrong trade.
-- **Tracks**: both.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Per-tile launch overhead visible in nsys timeline as repeated short kernels."
-
-### 3. Aligned GEMM (primary, cutlass-API baseline)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/gemm/aligned/`
-- **Why it matters for GEMM**: the cutlass-API canonical kernel for shapes where (M, N, K) are exact multiples of the wgmma atom (M=64, N in {8, 16, 32, 64, 128, 256}, K=8 / 16 / 32 by dtype). No tail / no predicate / no padding -- the fast-path mainloop runs every iteration.
-- **When to apply**: every cutlass-API GEMM with aligned shapes; the starting point before reaching for the more specialized skills below.
-- **Tracks**: cutlass-API.
-- **Relevance to bottleneck triage**: not a triage-driven skill; this is the *baseline* the triage tree's tensor-core branch returns to first.
-
-### 4. GEMM with non-aligned tail (primary, cutlass-API)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/gemm/non-aligned-tail/`
-- **Why it matters for GEMM**: extends skill 3 with a tail-handling block for shapes where M or N is not a multiple of the wgmma atom. Adds masking + a smaller atom for the boundary tile so the fast path still runs on the bulk of the grid.
-- **When to apply**: cutlass-API GEMM where the caller cannot pad to the alignment boundary upstream (memory budget, layout shared with another op, etc.).
-- **Tracks**: cutlass-API.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "M or N not divisible by atom; current path either pads (wastes bandwidth) or falls back to scalar tail (slow)."
-
-### 5. Cutlass-free GEMM via raw PTX (primary, cutlass-free)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/gemm-ptx/`
-- **Why it matters for GEMM**: composes skill 6 (`tma-ptx`) and skill 7 (`wgmma-ptx`) into a single GEMM kernel that contains zero `cutlass::` / `cute::` symbols. Verified by `nvcc -E | grep` and `cuobjdump --dump-elf-symbols | grep`. The single-tile starting point for the cutlass-free track; the warp-specialized extension lives at `sources/experience/kernel-records/2026-04-29-gemm-ws-ptx/`.
-- **When to apply**: cutlass-free track only. Required when the project cannot accept cutlass as a build-time or link-time dependency.
-- **Tracks**: cutlass-free.
-- **Status as of writing**: structural cutlass-free gates pass (0 `cutlass::` / `cute::` symbols); numeric correctness against the task reference at the smallest shape has a residual 497-501 / 512 mismatches traced to the per-thread fragment-store mapping (skill's `pitfalls.md` #1). The WS extension inherits the same residual.
-- **Relevance to bottleneck triage**: not a triage-driven skill; this is the *baseline* the triage tree's cutlass-free branch returns to first.
-
-### 6. TMA producer (primary)
-
-- **Skill paths**:
-  - cutlass-API: `wiki/nvidia/hardware/tma/`
-  - cutlass-free: `wiki/nvidia/hardware/tma-ptx/`
-- **Why it matters for GEMM**: the producer warp's load mechanism. `cuTensorMapEncodeTiled` builds the descriptor on the host; `cp.async.bulk.tensor.2d.shared::cluster.global.tile.mbarrier::complete_tx::bytes` issues the load on the device; the mbarrier protocol synchronizes producer with consumer. Without TMA, the producer side of skill 1 has nothing to issue and the WS pattern does not apply.
-- **When to apply**: every Hopper GEMM; the producer side of skill 1.
-- **Tracks**: both -- pick the matching path by track.
-- **Specific guidance for GEMM**:
-  - Feeding wgmma: `SWIZZLE_128B`, fast-axis = 64 bf16 / 32 fp32 (= 128 B), `box_rows >= 64`, pipeline depth >= 4.
-  - mbarrier `expected_tx` must equal `(BOX_M*BOX_K + BOX_K*BOX_N) * sizeof(elem)` (the *sum* of A and B tile bytes for one stage), not just one of the two -- WS pitfall #2.
-  - Phase tracking on each mbarrier alternates 0,1,0,1,... per stage cycle: consumer wait phase = `(k / STAGES) & 1`; producer wait phase on bar_empty = `((k - STAGES) / STAGES) & 1` -- WS pitfall #3.
-- **Measured on H200** (cutlass-free path): peak 3.72 TB/s = 77.5 % of HBM3e datasheet peak at SWIZZLE_128B + depth=4 + 16 KiB tiles.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Stall on `cp.async.bulk.tensor` not overlapped with wgmma compute."
-
-### 7. wgmma consumer (primary)
-
-- **Skill paths**:
-  - cutlass-API: `wiki/nvidia/hardware/wgmma/`
-  - cutlass-free: `wiki/nvidia/hardware/wgmma-ptx/`
-- **Why it matters for GEMM**: the consumer warpgroup's compute mechanism. `wgmma.mma_async.sync.aligned.m64nNkK.<dtype>` is the GEMM-K accumulator step, issued by exactly 128 threads. Without wgmma the consumer side of skill 1 has nothing to issue and the kernel falls back to Ampere mma.sync at far lower throughput.
-- **When to apply**: every Hopper GEMM; the consumer side of skill 1.
-- **Tracks**: both.
-- **Specific guidance for GEMM**:
-  - Atom selection: `N=128` maximizes FLOPs per serialized issue (5.32 TFLOPS single-warpgroup); `N=64` minimizes register pressure (4.70 TFLOPS, half the accumulator regs). Dtype is precision-only, not throughput.
-  - `wgmma.fence` + `wgmma.commit_group` + `wgmma.wait_group 0` are non-optional in every issue sequence -- wgmma-ptx pitfall #4.
-  - Smem descriptor must be 16-byte aligned -- wgmma-ptx pitfall #3.
-  - Per-thread fragment count is atom-shape-dependent (4 floats per thread for m64n8k16; 64 for m64n128k8 tf32). Get the accumulator clobber list wrong and ptxas truncates silently -- wgmma-ptx pitfall #5.
-  - PTX immediate count varies by dtype family (5 for bf16/fp16, 3 for tf32, 1 for s8) -- wgmma-ptx pitfall #8.
-- **Measured on H200**: single-warpgroup serialized plateau ~5.6 TFLOPS bf16/fp16 at any N>=128 (wgmma-ptx pitfall #11). Real GEMM throughput requires multi-warpgroup-per-CTA + multi-accumulator pipelining + multi-CTA scaling, which composes with skills 1, 2, 10, and 11.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "wgmma single-warpgroup serialized issue plateau visible in ncu as low `sm__inst_executed_pipe_tensor.sustained`."
-
-### 8. Fused epilogue / prologue (secondary, cutlass-API)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/gemm-fused/cutlass-epilogue-prologue/`
-- **Why it matters for GEMM**: extends the cutlass GEMM with a custom epilogue (e.g. fused activation, scale, quantize) or prologue (e.g. dequantize from a scaled buffer) when the required epilogue/prologue fusion is outside the preselected implementation track. Avoids the global-memory round-trip of a separate post-GEMM kernel.
-- **When to apply**: cutlass-API GEMM where the fused pattern must be implemented inside the custom epilogue/prologue.
-- **Tracks**: cutlass-API.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Post-GEMM elementwise kernel re-reads D from DRAM."
-
-### 9. Compiler hints for register / occupancy (secondary)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/compiler-hints/`
-- **Why it matters for GEMM**: `__launch_bounds__` controls the register-allocation ceiling, which interacts with wgmma fragment register count and TMA pipeline depth (deeper pipelines = more in-flight smem barriers + descriptors). Setting it incorrectly drops occupancy and serializes wgmma issue.
-- **When to apply**: tuning the final kernel after correctness is established. The cutlass-free WS record uses `__launch_bounds__(160, 1)` to bind producer + consumer thread count.
-- **Tracks**: both.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "ncu `achieved_occupancy` low and `register_pressure` high; verify `__launch_bounds__` matches the actual CTA size."
-
-### 10. Instruction-level parallelism via multi-accumulator (secondary)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/ilp/`
-- **Why it matters for GEMM**: a single accumulator chain serializes wgmma issues at ~5.6 TFLOPS (wgmma-ptx pitfall #11). Multi-accumulator pipelining splits the K-loop into 2-4 independent chains so consecutive wgmmas do not depend on each other; the issue port stays warm.
-- **When to apply**: after warp-spec is in place. Pair with pingpong / cooperative -- the shape that benefits is many-K-tile mainloops where latency hiding inside the K loop matters.
-- **Tracks**: both.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "wgmma issue port underutilized; consecutive `wgmma.wait_group 0` flush-stalls visible in nsys."
-
-### 11. Code-extraction discipline (meta)
-
-- **Skill path**: `wiki/nvidia/foundations/meta/code-extraction/`
-- **Why it matters for GEMM**: the cutlass-free track exists by extracting cutlass's PTX wrappers into hand-rolled inline asm. The meta-skill codifies how to cleanly extract a primitive without inheriting the surrounding template machinery.
-- **When to apply**: starting any new cutlass-free kernel or porting a cutlass primitive.
-- **Tracks**: cutlass-free (primary); cutlass-API (when extending the cutlass codebase, secondary).
-- **Relevance to bottleneck triage**: not a triage-driven skill; this is a discipline meta-skill applied at kernel-start time, not at performance-tuning time.
-
-## Skills NOT applicable to tensor-core GEMM
-
-The following skills exist in the KB but are generally not relevant for Hopper tensor-core GEMM kernels:
-
-- **Warp primitives** (`wiki/nvidia/foundations/compute/warp-primitives/`): wgmma is itself the cross-thread reduction primitive for the GEMM K-loop. Adding `__shfl_*` on top of the wgmma fragment is redundant and the m64nNk16 fragment layout is already cross-warp (4 warps) -- a shuffle-based reduce would fight the layout.
-
-- **Reduction skills** (`wiki/nvidia/foundations/sync/atomic-reduction/`): a GEMM kernel's output is a tile, not a scalar; the global-write phase is one store per output element with no contention. Atomic reduction applies to split-K GEMMs (where partial sums from different CTAs are atomically combined), but the split-K combine is itself usually a separate kernel and falls under the reduction pattern, not this one.
-
-- **Memory coalescing** (`wiki/nvidia/foundations/memory/coalescing/`): TMA is the load mechanism on Hopper; coalescing is enforced by the descriptor and the box shape, not by the thread mapping. Skill 6's "fast-axis = 64 bf16" guidance is the equivalent prescription expressed in TMA terms.
-
-- **Bank-conflict avoidance** (`wiki/nvidia/foundations/memory/bank-conflict/`): wgmma's smem descriptor encodes its own swizzle (`SWIZZLE_NONE / 32B / 64B / 128B`); paired with the TMA producer's matching swizzle mode, banks are conflict-free by construction. Hand-rolled `+1` padding does not apply -- wgmma will read the swizzled layout, not a `[N][N+1]` tile.
-
-- **Vectorized access** (`wiki/nvidia/foundations/memory/vectorized-access/`): TMA already issues 128-bit-aligned bulk loads (`cp.async.bulk.tensor`); adding `float4` reinterpretation on top would conflict with the descriptor's element type. Vectorization here is intrinsic to the instruction, not a kernel-level rewrite.
-
-- **Async copy** (`wiki/nvidia/foundations/memory/async-copy/`): `cp.async` (the Ampere primitive) is superseded by TMA on Hopper for tensor-shaped loads. The TMA path is async by design and is what skill 6 covers.
-
-- **Fast math** (`wiki/nvidia/foundations/compute/fast-math/`): GEMM uses the tensor-core MMA path (wgmma); the IEEE-754 vs fast-math distinction does not apply to FMA inside a tensor core. For the epilogue (skill 8) where elementwise activations are added, fast-math may apply -- in that case it is the epilogue kernel's lever, not the GEMM mainloop's.