@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/code-walkthroughs/cutlass-cute/wgmma-atom-decoding/README.md

@@ -1,143 +0,0 @@
----
-id: code-cutlass-cute-wgmma-atom-decoding-README
-type: code-walkthrough
-vendor: nvidia
-title: Readme
-upstream_repo: NVIDIA/cutlass-cute
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-- fp8
-techniques:
-- warp-specialization
-- shared-memory-optimization
-kernel_types:
-- gemm
-- quantization
-- decode
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- fp8
-- warp-specialization
-- shared-memory-optimization
-- gemm
-- quantization
-- decode
-- ptx
-- cuda-cpp
-- cute-dsl
----
-# wiki/nvidia/code-walkthroughs/cutlass-cute/wgmma-atom-decoding — understanding CUTLASS WGMMA atoms
-
-This directory is a **CUTLASS reading aid**. Its purpose is to explain how CUTLASS / CuTe represents Hopper WGMMA instructions in C++ types, especially in `48_hopper_warp_specialized_gemm.cu`.
-
-It is not a handwritten-kernel tuning guide. Performance rules, atom recommendations, and register/smem budgeting belong outside this folder.
-
-The canonical runnable artifact for the CUTLASS path lives at:
-
-- `artifacts/experience/api-probes/gemm/gemm_aligned.cu`
-
-## Reader Question
-
-This folder answers one question:
-
-> In CUTLASS / CuTe, how do I describe the single hardware WGMMA operation that the consumer warpgroup will repeatedly issue?
-
-That single operation is called an **atom**. The surrounding producer / consumer mainloop is tracked separately under `../example48-hopper-warp-specialized-gemm/mainloop_skeleton.md`. No buildable code lives here.
-
-## Use This For
-
-Use this directory when you are reading CUTLASS code and need to decode:
-
-- what `MMA_64x128x8_F32TF32TF32_SS_TN` means
-- why that type is wrapped in `MMA_Atom<...>`
-- how `TiledMMA` repeats the atom across a CTA tile
-- why `cute::gemm(tiled_mma, sA, sB, accum)` lowers to `wgmma.mma_async`
-- how to read atom names in profiler/kernel-signature output
-
-Do not use this directory to understand the full GEMM schedule. That belongs to `../example48-hopper-warp-specialized-gemm/`.
-
-## What Atom Means Here
-
-An atom is CuTe's type-level description of **one architecture instruction family instance**.
-
-For WGMMA on Hopper, one atom answers:
-
-| Question | Example answer |
-|---|---|
-| Which instruction shape? | `64x128x8` |
-| Which accumulator / input types? | `F32 <- TF32 x TF32` |
-| Where do operands come from? | `SS` = A and B both from shared memory |
-| Which logical layouts? | `TN` = A row-major-ish, B column-major-ish for this atom |
-| Which PTX instruction is emitted? | `wgmma.mma_async.sync.aligned.m64n128k8.f32.tf32.tf32.f32` |
-
-So:
-
-```cpp
-MMA_Atom<MMA_64x128x8_F32TF32TF32_SS_TN<1, 1>>
-```
-
-means "the smallest WGMMA unit CuTe will tile with is a Hopper warpgroup MMA instruction that computes a 64 x 128 output slice over K=8 using TF32 inputs and F32 accumulation."
-
-It is **not** the whole GEMM. It is also **not** the whole CTA tile. A CTA tile such as `Shape<_128,_128,_32>` is built by repeating this atom across M and K:
-
-```text
-CTA tile:       128 x 128 x 32
-WGMMA atom:      64 x 128 x  8
-Needed repeats:   2 x   1 x  4 atom issues
-```
-
-`TiledMMA` is the CuTe object that says how to repeat the atom. `cute::gemm(tiled_mma, sA, sB, accum)` is the call that emits those repeated atom instructions.
-
-## Name Decoding
-
-The naming pattern is:
-
-```text
-MMA_<M>x<N>x<K>_<Acc><A><B>_<OperandSource>_<Layout>
-```
-
-For the production TF32 atom:
-
-```text
-MMA_64x128x8_F32TF32TF32_SS_TN
-    64x128x8      one hardware WGMMA instruction shape
-    F32           accumulator type
-    TF32TF32      A and B input types
-    SS            A and B are read from shared memory
-    TN            A/B logical layout variant
-```
-
-## Atom Name Catalogue
-
-CuTe exposes one atom type per `(M, N, K, accumulator-dtype, A-dtype, B-dtype, layout, A-source)` combination. This table is for decoding CUTLASS symbols, not for recommending which atom to choose.
-
-The naming pattern is `MMA_<M>x<N>x<K>_<accD><dtA><dtB>_<SS|RS>_<TN|NT|NN|TT>` where `SS` = both A and B in smem, `RS` = A in registers + B in smem.
-
-| Axis | Hopper-supported values | Notes |
-|---|---|---|
-| M | 64 | wgmma's M is fixed at 64 (= 4 warps × 16 rows) |
-| N | 8, 16, 32, 64, 96, 128, 192, 256 | larger N → more FLOPs / issue, more accumulator regs |
-| K | 8 (tf32) / 16 (bf16, fp16) / 32 (s8/u8/fp8) | dtype-dependent |
-| Acc / A / B dtype | f32/tf32/tf32 · f32/bf16/bf16 · f32/fp16/fp16 · f16/fp16/fp16 · s32/s8/s8 · s32/u8/u8 · f32/fp8/fp8 | tf32 supports only `_TN`; the others support all four trans variants |
-| Layout | `_SS_TN`, `_SS_NT`, `_SS_NN`, `_SS_TT` | bf16 / fp16 / int / fp8 atoms expose all four; tf32 only `_TN` |
-| A-source | `_SS_*` (smem) / `_RS_*` (registers) | RS skips A's smem footprint at the cost of an explicit register pack |
-
-## Cross-references
-
-- Skill: `wiki/nvidia/hardware/wgmma/skill.md`
-- Cutlass-free counterpart (raw PTX, same instructions): `wiki/nvidia/hardware/wgmma/skill-wgmma-ptx.md`
-- Pitfalls: `wiki/nvidia/hardware/wgmma/pitfalls.md`
-- Canonical artifacts: `artifacts/experience/api-probes/gemm/gemm_aligned.cu`
-- Atom skeleton: `wgmma_skeleton.md`
-- Surrounding WS mainloop: `../example48-hopper-warp-specialized-gemm/mainloop_skeleton.md`
-- Atom-shape sweep: `sources/experience/api-probes/gemm.md`
-- Counter measurements: `sources/experience/api-probes/gemm.md`

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/code-walkthroughs/cutlass-cute/wgmma-atom-decoding/wgmma_skeleton.md

@@ -1,144 +0,0 @@
----
-id: code-cutlass-cute-wgmma_skeleton
-type: code-walkthrough
-vendor: nvidia
-title: Wgmma_Skeleton
-upstream_repo: NVIDIA/cutlass-cute
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-- cluster
-- fp8
-techniques:
-- warp-specialization
-- shared-memory-optimization
-- swizzling
-kernel_types:
-- gemm
-- quantization
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- cluster
-- fp8
-- warp-specialization
-- shared-memory-optimization
-- swizzling
-- gemm
-- quantization
-- ptx
-- cuda-cpp
-- cute-dsl
----
-# CUTLASS WGMMA Atom Skeleton (example 48, commit `f74fea9c`)
-
-This is a reading guide for the CUTLASS / CuTe WGMMA atom path in `48_hopper_warp_specialized_gemm.cu`. The same source file is represented by the runnable repro under `artifacts/experience/api-probes/gemm/gemm_aligned.cu`; here we annotate only the CUTLASS symbols needed to understand which WGMMA instruction is being described.
-
-The warp-specialized producer / consumer mainloop that surrounds these calls lives in `../example48-hopper-warp-specialized-gemm/mainloop_skeleton.md`.
-
-## Mental Model
-
-Read "atom" as "one hardware WGMMA instruction described as a C++ type."
-
-The path is:
-
-```text
-MMA_64x128x8_F32TF32TF32_SS_TN   // atom type: one instruction shape
-  -> MMA_Atom<...>               // CuTe wrapper around that atom
-  -> TiledMMA                    // how this atom is repeated across a CTA tile
-  -> cute::gemm(...)             // emits repeated wgmma.mma_async instructions
-```
-
-For example 48:
-
-```text
-CTA tile:       128 x 128 x 32
-selected atom:   64 x 128 x  8
-repeat pattern:   2 x   1 x  4 atom issues
-```
-
-So the atom is smaller than the CTA tile. It is the instruction-level building block that CuTe repeats. This file stops at that code-reading boundary; it does not try to recommend an atom for a new handwritten kernel.
-
-## What cute emits
-
-```
-[device, consumer wg]   wgmma.fence.sync.aligned                    // make smem operands visible to the wgmma issue group
-[device, consumer wg]   wgmma.mma_async.sync.aligned.m64nNk8.f32.tf32.tf32.f32   // the actual MMA
-[device, consumer wg]   wgmma.commit_group.sync.aligned             // mark issue boundary
-[device, consumer wg]   wgmma.wait_group.sync.aligned 0             // block until N-1 groups have completed
-```
-
-cutlass-cute hides every step behind `cute::gemm(tiled_mma, sA, sB, accumulator)`. The `tiled_mma` object contains the atom (`MMA_64xNxK_F32TF32TF32_SS_TN<...>`) plus the repeat layout.
-
-## Critical wgmma fragments
-
-### A. Atom selection (compile-time)
-
-Lines 110-134 of the vendored example 48 variant:
-
-```cpp
-using TileShape    = Shape<_128, _128, _32>;     // Threadblock-level tile (M, N, K)
-using ClusterShape = Shape<_4,   _2,  _1>;       // Cluster of CTAs
-
-using CollectiveMainloop = typename cutlass::gemm::collective::CollectiveBuilder<
-    ArchTag, OperatorClass,
-    ElementA, LayoutA, AlignmentA,
-    ElementB, LayoutB, AlignmentB,
-    ElementAccumulator,
-    TileShape, ClusterShape,
-    ...,
-    cutlass::gemm::collective::KernelScheduleAuto         // picks Cooperative / Pingpong / Multistage
-  >::CollectiveOp;
-```
-
-`CollectiveBuilder` auto-selects the wgmma atom from `TileShape`, dtype, layouts, and operand source. The N dimension of `TileShape` controls the wgmma N, and therefore which atom in the `MMA_64xNxK` family is issued. For `TileShape = <_128,_128,_32>` and TF32 input, cute picks `MMA_64x128x8_F32TF32TF32_SS_TN<1, 1>`. Smaller TileShape N -> smaller atom; larger TileShape N -> larger atom. The atom-shape sweep exercises this knob.
-
-`KernelScheduleAuto` is shown here only because it appears in the same template block. Its producer / consumer schedule choice belongs to the warp-specialization track.
-
-### B. Issue + wait (runtime)
-
-Inside the consumer warpgroup, the wgmma part of the loop is:
-
-```cpp
-warpgroup_fence_operand(accum);                       // wgmma.fence
-cute::gemm(tiled_mma, sA, sB, accum);                 // emits N wgmma.mma_async issues
-warpgroup_commit_batch();                             // wgmma.commit_group
-warpgroup_wait<0>();                                  // wgmma.wait_group 0 — block until previous batch lands
-```
-
-`warpgroup_fence_operand` / `warpgroup_commit_batch` / `warpgroup_wait` are inline helpers in `include/cute/arch/mma_sm90.hpp` that wrap `__nvvm_intrinsic`-style PTX `wgmma.fence`/`wgmma.commit_group`/`wgmma.wait_group`.
-
-### C. Pre-conditions (from the cute traits headers)
-
-Decoded from the vendored example's compiled kernel signature:
-
-- `MMA_Atom<MMA_64x128x8_F32TF32TF32_SS_TN<1,1>>` — atom is 64M × 128N × 8K, TF32 inputs, F32 accumulator, SS-form (both A and B from smem), TN layout (A row-major, B column-major).
-- `Layout<tuple<C<2>, C<1>, C<1>>, ...>` — the atom is repeated 2× along M, so the consumer warpgroup covers 128 = 2 × 64 rows for this CTA tile.
-- Smem layout uses `Swizzle<3,4,3>` over a 32-bit unit (canonical bank-conflict-free pattern for SS wgmma).
-
-## Atom Family Names Seen In CUTLASS
-
-| Family | M | N | K | Notes |
-|---|---|---|---|---|
-| F32 ← BF16 × BF16 | 64 | 8/16/32/64/96/128/192/256 | 16 | |
-| F32 ← FP16 × FP16 | 64 | 8/16/32/64/96/128/192/256 | 16 | |
-| F32 ← TF32 × TF32 | 64 | 8/16/32/64/96/128/192/256 | 8 | Example 48 uses this family. |
-| F32 ← FP8 × FP8 | 64 | 8/16/32/64/96/128/192/256 | 32 | Need `e4m3` or `e5m2` dtype. |
-
-## Cross-references
-
-- Skill: `wiki/nvidia/hardware/wgmma/skill.md`
-- Pitfalls: `wiki/nvidia/hardware/wgmma/pitfalls.md`
-- Single-atom probe: `sources/experience/api-probes/gemm.md`
-- Atom-shape sweep: `sources/experience/api-probes/gemm.md`
-- Reproducible bundle: `artifacts/experience/api-probes/gemm/`
-- Surrounding WS mainloop: `../example48-hopper-warp-specialized-gemm/mainloop_skeleton.md`
-- Upstream (pinned): `cutlass@f74fea9c` (`{{CUTLASS_REPO_REF}}`); see `examples/48_hopper_warp_specialized_gemm/` and `include/cute/atom/mma_traits_sm90_gmma.hpp`.
-- Authoritative blog: `corpus/nvidia/blogs/colfax/cutlass-tutorial-fast-matrix-multiplication-with-wgmma-on-nvidia-hopper-gpus`.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/code-walkthroughs/flash-attention-v3/README.md

@@ -1,127 +0,0 @@
----
-id: code-flash-attention-v3-README
-type: code-walkthrough
-vendor: nvidia
-title: Readme
-upstream_repo: NVIDIA/flash-attention-v3
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-- python
-hardware_features:
-- wgmma
-- tma
-- fp8
-techniques:
-- pipeline-stages
-- kernel-fusion
-- shared-memory-optimization
-- software-exp
-- conditional-rescaling
-kernel_types:
-- gemm
-- attention
-- flash-attention
-- fused-kernel
-- quantization
-- mla
-- decode
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- tma
-- fp8
-- pipeline-stages
-- kernel-fusion
-- shared-memory-optimization
-- software-exp
-- conditional-rescaling
-- gemm
-- attention
-- flash-attention
-- fused-kernel
-- quantization
-- mla
-- decode
-- ptx
-- cuda-cpp
-- cute-dsl
-- python
----
-# FlashAttention v3 (Tri Dao et al.)
-
-## Source
-
-- Repository: https://github.com/Dao-AILab/flash-attention
-- Local clone: `{{FLASH_ATTENTION_REPO_REF}}`
-- Commit: `bbda031f1cd1adf1a57a3b3e8cdc3db2b54d3994`
-- License: BSD-3-Clause
-- MANIFEST registration: `corpus/MANIFEST.yaml` → `source-code/flash-attention`
-
-## What it is
-
-FlashAttention v3 is Tri Dao's reference implementation of fused multi-head attention with online softmax (the "FlashAttention" algorithm). It supports:
-- Forward and backward passes
-- Hopper (sm_90a) optimized kernels using wgmma and TMA
-- Ampere (sm_80) fallback kernels using mma.sync
-- FP16, BF16, and FP8 input types
-- Causal masking, variable-length (varlen), split-KV, and MLA (Multi-head Latent Attention) decode
-- Multi-head, grouped-query, and multi-query attention patterns
-
-## Key source files
-
-| File | Role |
-|---|---|
-| `hopper/flash_fwd_kernel_sm90.h` | Hopper forward kernel (wgmma-based) |
-| `hopper/flash_bwd_kernel_sm90.h` | Hopper backward kernel |
-| `hopper/flash_api.cpp` | C++ API entry point |
-| `hopper/block.h` | Block/tile configuration |
-| `flash_attn/flash_attn_interface.py` | Python API |
-| `hopper/epilogue_fwd.hpp` | Forward epilogue (writeback + rescale) |
-| `hopper/epilogue_bwd.hpp` | Backward epilogue |
-
-## Algorithmic approach
-
-FlashAttention v3 implements the same core algorithm as FlashAttention-2:
-
-1. **Tiled online softmax**: Process KV tiles sequentially, maintaining a running max and running sum per Q-row. Rescale the output accumulator whenever the max updates.
-2. **IO-aware tiling**: Tile sizes are chosen to minimize HBM reads by keeping Q/O tiles in SRAM (registers + smem) while streaming K/V tiles from HBM.
-3. **Fused kernel**: The entire attention computation (Q@K^T, softmax, P@V) runs in a single kernel launch, avoiding materialization of the S or P matrices in HBM.
-
-### Comparison with cutlass FMHA (`examples/88_hopper_fmha`)
-
-| Aspect | FlashAttention v3 | cutlass FMHA |
-|---|---|---|
-| Origin | Tri Dao (academic/industry) | NVIDIA (in-tree cutlass) |
-| Template system | Hand-rolled C++ templates | cutlass `CollectiveBuilder` + `TiledMMA` |
-| Hopper path | Direct wgmma/TMA inline PTX via cutlass cute atoms | Full cutlass collective/kernel/device stack |
-| Ampere fallback | Yes (sm_80 kernels) | No (Hopper only in example 88) |
-| Variable-length | Yes (varlen API) | Partial (batch-only) |
-| FP8 | Yes | Yes |
-| MLA decode | Yes (specialized kernel) | No |
-| Backward | Yes | Yes |
-
-### Comparison with MVP minimal attention (`wiki/nvidia/foundations/compute/attention/mvp-minimal`)
-
-| Aspect | FlashAttention v3 | MVP minimal |
-|---|---|---|
-| Purpose | Production performance | Correctness reference + learning |
-| Code complexity | ~10K lines | ~200 lines |
-| Tiling | Multiple tuned tile sizes | Fixed BLOCK_M=64, BLOCK_N=64 |
-| Pipeline | Multi-stage TMA pipeline | Single-buffer |
-| Performance | Optimized for production (no KB measurement yet) | Correctness reference |
-| Dependencies | cutlass cute atoms | None (raw CUDA) |
-
-## Tuning parameters
-
-See `tuning.md` for the parameter space.
-
-## Status
-
-- Registered in `corpus/MANIFEST.yaml` with commit hash and local path.
-- No measured artifacts from this KB's H200 infrastructure yet. If measured, artifacts would follow the standard layout at `artifacts/experience/api-probes/attention/`.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/code-walkthroughs/flash-attention-v3/tuning.md

@@ -1,70 +0,0 @@
----
-id: code-flash-attention-v3-tuning
-type: code-walkthrough
-vendor: nvidia
-title: Tuning
-upstream_repo: NVIDIA/flash-attention-v3
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-- python
-hardware_features:
-- wgmma
-techniques:
-- pipeline-stages
-- shared-memory-optimization
-kernel_types:
-- gemm
-- attention
-- flash-attention
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- pipeline-stages
-- shared-memory-optimization
-- gemm
-- attention
-- flash-attention
-- cuda-cpp
-- python
----
-# FlashAttention v3 Tuning Parameters
-
-## Compile-time parameters (from `hopper/block.h`)
-
-| Knob | Typical values | Effect |
-|---|---|---|
-| kBlockM | 64, 128, 192 | Q-tile row count. Must be a multiple of wgmma M=64. Larger kBlockM increases compute per KV iteration. |
-| kBlockN | 64, 128, 176, 256 | KV-tile sequence chunk. Larger kBlockN reduces number of KV iterations but increases smem. |
-| kHeadDim | 64, 128, 192, 256 | Head dimension. Fixed per model. Affects the K-dimension of both Q@K^T and P@V tiles. |
-| kNWarps | 8, 12, 16 | Warps per CTA. More warps enable more warpgroups (wgmma consumers). Typical: 8 = 2 warpgroups, 12 = 3 warpgroups. |
-| kStages | 1, 2 | smem pipeline stages for KV tiles. |
-| Is_causal | true, false | Enables triangular causal mask. |
-| Is_local | true, false | Enables sliding window / local attention. |
-
-## Runtime parameters (from Python/C++ API)
-
-| Parameter | Type | Effect |
-|---|---|---|
-| batch_size | int | Number of sequences. |
-| seqlen_q, seqlen_k | int | Query and Key/Value sequence lengths. |
-| num_heads, num_heads_k | int | Number of Q heads and KV heads (for GQA). |
-| head_dim | int | Must match kHeadDim. |
-| softmax_scale | float | Typically 1/sqrt(head_dim). |
-| window_size_left, window_size_right | int | For local attention. |
-| num_splits | int | Split-KV parallelism along sequence dimension. |
-
-## Key differences from cutlass FMHA tuning
-
-- FlashAttention v3 uses a fixed set of pre-tuned tile configurations selected via head_dim and dtype, rather than cutlass's fully template-parameterized builder.
-- The `num_splits` parameter enables split-KV parallelism (analogous to split-K in GEMM), which cutlass FMHA does not expose.
-- kNWarps directly maps to warpgroup count, whereas cutlass FMHA uses `kNumMmaWarpGroups` as a separate option.
-
-## Source reference
-
-Tile configurations: `hopper/block.h`
-Python API: `flash_attn/flash_attn_interface.py`
-C++ dispatch: `hopper/flash_api.cpp`

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/advanced/INDEX.md

@@ -1,19 +0,0 @@
----
-id: skill-advanced-index
-title: 'Index: Foundations Advanced'
-type: skill
-vendor: nvidia
-architectures:
-- '*'
-tags:
-- cuda-cpp
-evidence_level: spec
-applies_to:
-- general
-source:
-- path: spec
-  anchor: CUDA reference
-languages:
-- cuda-cpp
-confidence: source-reported
----

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/compute/attention/pitfalls.md

@@ -1,75 +0,0 @@
----
-id: pitfall-attention
-type: pitfall
-vendor: nvidia
-title: Pitfalls
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-hardware_features:
-- wgmma
-- tma
-techniques:
-- pipeline-stages
-- shared-memory-optimization
-- software-exp
-- conditional-rescaling
-kernel_types:
-- attention
-- flash-attention
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- tma
-- pipeline-stages
-- shared-memory-optimization
-- software-exp
-- conditional-rescaling
-- attention
-- flash-attention
-- cuda-cpp
----
-# MVP Minimal Flash-Attention Pitfalls
-
-## 1. Online softmax numerical stability
-
-**Problem**: The online softmax algorithm requires tracking a running maximum `m_i` per row. If `m_i` is not updated before computing `exp(S - m_new)`, the exponential overflows for large logits, producing `inf` values that corrupt the output.
-
-**Trigger**: Processing long sequences (S > 2048) with large attention scores, especially with fp16 inputs where the dynamic range is limited.
-
-**Mitigation**: Always update `m_new = max(m_i, rowmax(S))` before computing `exp(S - m_new)`. The rescaling factor `exp(m_i - m_new)` is always <= 1.0, so it never overflows. Use f32 accumulators for the softmax computation even when inputs are fp16.
-
-## 2. QK^T vs Q@K layout confusion
-
-**Problem**: The attention score `S = Q @ K^T` requires K to be transposed. If K is loaded in the same layout as Q (row-major with head_dim as the fast dimension), the wgmma instruction computes Q @ K instead of Q @ K^T, producing a (BLOCK_M x BLOCK_N) result from (BLOCK_M x D) @ (D x BLOCK_N) only if K is transposed.
-
-**Trigger**: Loading K tile with the same TMA descriptor as Q without accounting for the transpose.
-
-**Mitigation**: Either transpose K explicitly in smem before the wgmma, or use a different wgmma layout variant (e.g., `.TN` with K loaded in N-major layout). The cutlass FMHA example uses the latter approach.
-
-## 3. Accumulator rescaling precision
-
-**Problem**: Each KV-tile iteration rescales the running output accumulator by `exp(m_old - m_new)`. Over many iterations, this multiplicative rescaling can accumulate rounding error, especially in fp16.
-
-**Trigger**: Very long sequences (S > 8192) with many KV-tile iterations.
-
-**Mitigation**: Keep the output accumulator in f32 throughout the loop. Only convert to fp16 at the final writeback after the `O /= l_i` normalization.
-
-## 4. Shared memory capacity for Q + K + V tiles
-
-**Problem**: The kernel needs smem for Q (BLOCK_M x D), K (BLOCK_N x D), and V (BLOCK_N x D) tiles simultaneously. For BLOCK_M=128, BLOCK_N=128, D=128 in fp16, this is 128x128x2 + 128x128x2 + 128x128x2 = 96 KB — close to the H200 per-SM smem limit of 228 KB but feasible. With multi-stage pipelining, the requirement multiplies.
-
-**Trigger**: Large tile sizes with multiple pipeline stages exceeding available smem.
-
-**Mitigation**: Size tiles to fit within the target SM's smem budget. H200 supports up to 228 KB dynamic smem per CTA (with `cudaFuncSetAttribute`). Use 2-stage pipelining for large tiles or reduce BLOCK_N.
-
-## 5. wgmma atom shape mismatch with attention dimensions
-
-**Problem**: wgmma atoms have M=64 fixed. If BLOCK_M < 64 or BLOCK_M is not a multiple of 64, the kernel cannot directly use wgmma for the Q @ K^T tile.
-
-**Trigger**: Choosing BLOCK_M=32 for small sequence lengths.
-
-**Mitigation**: Always set BLOCK_M to a multiple of 64. For small sequences, use BLOCK_M=64 as the minimum.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/compute/attention/tuning.md

@@ -1,99 +0,0 @@
----
-id: skill-attention-tuning
-type: skill
-vendor: nvidia
-title: Tuning
-tags:
-- cuda-cpp
-- wgmma
-- tma
-- mbarrier
-- pipeline-stages
-- double-buffering
-- shared-memory-optimization
-- swizzling
-- software-exp
-- conditional-rescaling
-- attention
-- flash-attention
-applies_to:
-- general
-source:
-- path: spec
-  anchor: CUDA Programming Guide
-evidence_level: spec
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-hardware_features:
-- wgmma
-- tma
-- mbarrier
-techniques:
-- pipeline-stages
-- double-buffering
-- shared-memory-optimization
-- swizzling
-- software-exp
-- conditional-rescaling
-kernel_types:
-- attention
-- flash-attention
-confidence: source-reported
----
-# MVP Minimal Flash-Attention Tuning Parameters
-
-## Primary kernel: fixed-config TMA+wgmma
-
-The primary artifact (`flash_attn_tma_wgmma.cu`) uses fixed constants — not template parameters:
-
-| Constant | Value | Code location | Notes |
-|---|---|---|---|
-| BM | 64 | `static constexpr int BM = 64` | Matches wgmma M=64. Not tunable without recompilation. |
-| BN | 64 | `static constexpr int BN = 64` | KV-tile width. Fixed for this MVP. |
-| HD | 64 | `static constexpr int HD = 64` | Head dimension. Fixed per model. |
-| NTH | 128 | `static constexpr int NTH = 128` | 1 warpgroup (4 warps × 32 threads). |
-
-Future tuning axes (not yet explored):
-- Pipeline depth (single-buffer → double-buffer TMA)
-- Smem swizzle mode (SWIZZLE_128B for bank-conflict-free access)
-- Larger N per wgmma (m64n32k16 or m64n64k16 for fewer wgmma calls)
-
-## Runtime parameters
-
-| Parameter | Effect |
-|---|---|
-| B (batch) | Number of independent sequences. Grid Y dimension. |
-| H (heads) | Number of attention heads. Grid Y dimension (B×H). |
-| S (seq_len) | Sequence length. Must be a multiple of 64 (BM=BN=64). Total work is O(S² × D) per batch-head. |
-| D (head_dim) | Must equal 64 (HD constant). |
-| scale | Softmax temperature, typically 1/sqrt(D). |
-
-## Smem budget (primary kernel)
-
-Total dynamic smem = ~40.8 KB (fits in default 48 KB):
-- Q tile: 64×64 fp16 = 8 KB
-- KV tile: 64×64 fp16 = 8 KB (shared for K then V)
-- S matrix: 64×64 f32 = 16 KB (wgmma output / softmax workspace)
-- P matrix: 64×64 fp16 = 8 KB (softmax output for second wgmma)
-- Per-row state: 3 × 64 × f32 = 768 B (max, sum, rescale)
-- mbarrier: 8 B
-
-## Secondary reference: thread-level tuning sweep
-
-The secondary reference kernel (`flash_attn_minimal.cu`) is templatized on BLOCK_M × BLOCK_N and has a measured tile-shape sweep on H200. This data characterizes thread-level scalar performance, not the primary TMA+wgmma kernel.
-
-H200-SXM, sm_90a, cuda 12.9.86. Fixed workload: B=1, H=2, S=256, D=64. CSV at `artifacts/experience/api-probes/attention/attention-sweep.csv`. Full analysis at `sources/experience/hw-probes/attention-tuning.md`.
-
-| BLOCK_M | BLOCK_N | kernel_ms | throughput_gflops | Disposition |
-|---|---|---|---|---|
-| 32 | 64 | **0.151** | **221.77** | Passed |
-| 32 | 32 | 0.166 | 201.77 | Passed |
-| 64 | 64 | 0.298 | 112.67 | Passed |
-| 64 | 128 | 0.310 | 108.38 | Passed |
-| 64 | 32 | 0.341 | 98.49 | Passed |
-| 128 | 64 | 0.502 | 66.89 | Passed |
-
-BLOCK_M is the dominant knob (2x cost per 2x BLOCK_M). These results characterize thread-level scalar performance, not tensor-core capability.