@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/hardware/wgmma/atomic_skeleton.md

@@ -1,129 +0,0 @@
----
-id: skill-wgmma-ptx-atomic_skeleton
-type: skill
-vendor: nvidia
-title: Atomic_Skeleton
-tags:
-- cuda-cpp
-- wgmma
-- shared-memory-optimization
-- swizzling
-- gemm
-- ptx
-- cute-dsl
-evidence_level: spec
-applies_to:
-- gemm
-source:
-- path: spec
-  anchor: WGMMA PTX ISA
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-techniques:
-- shared-memory-optimization
-- swizzling
-kernel_types:
-- gemm
-confidence: source-reported
----
-# wgmma Atomic Skeleton (PTX, cutlass-free)
-
-This document extracts the minimal runnable wgmma kernel from `artifacts/experience/hw-probes/wgmma-ptx/wgmma_hello.cu`. The kernel issues a single `wgmma.mma_async.sync.aligned.m64n8k16.f32.bf16.bf16` instruction with all-ones inputs and verifies that every output equals K=16.
-
-## Build and run
-
-```bash
-nvcc -std=c++17 -O3 -gencode=arch=compute_90a,code=sm_90a \
-     -lineinfo wgmma_hello.cu -o wgmma_hello
-./wgmma_hello
-# Expected: D entries matching K=16: 512 / 512
-```
-
-Cutlass-free verification:
-
-```bash
-nvcc -std=c++17 -O3 -gencode=arch=compute_90a,code=sm_90a -E wgmma_hello.cu \
-    | grep -cE 'cutlass::|cute::'
-# Expected: 0
-
-cuobjdump --dump-ptx wgmma_hello | grep -c 'wgmma.mma_async'
-# Expected: >= 1
-```
-
-## Kernel structure
-
-```
-1. Cooperative load: 128 threads load A (64x16 bf16 = 2 KB) and B (16x8 bf16 = 256 B) into smem
-2. Build smem descriptors: make_smem_desc(ptr, ld_bytes, sd_bytes, swizzle)
-3. wgmma.fence.sync.aligned        -- barrier between prior SASS and wgmma issue
-4. wgmma.mma_async.sync.aligned    -- the MMA instruction
-5. wgmma.commit_group.sync.aligned -- close the issue group
-6. wgmma.wait_group.sync.aligned 0 -- wait for retirement
-7. Scatter accumulator fragments back to global memory
-```
-
-## Smem descriptor construction
-
-```cpp
-__device__ __forceinline__ uint64_t make_smem_desc(
-    const void* smem_ptr, uint32_t ld_bytes, uint32_t sd_bytes, uint32_t swizzle = 0) {
-    uint32_t smem_int = static_cast<uint32_t>(__cvta_generic_to_shared(smem_ptr));
-    uint64_t desc = 0;
-    desc |= ((uint64_t)(smem_int >> 4)) & 0x3FFF;
-    desc |= ((uint64_t)(ld_bytes >> 4) & 0xFFFF) << 14;
-    desc |= ((uint64_t)(sd_bytes >> 4) & 0xFFFF) << 30;
-    desc |= ((uint64_t)(swizzle & 0x3)) << 62;
-    return desc;
-}
-```
-
-Bit layout: `[0:13]` = smem byte address >> 4; `[14:29]` = leading-dim offset >> 4; `[30:45]` = stride-dim offset >> 4; `[46:48]` = base offset (0 for no swizzle); `[62:63]` = swizzle mode.
-
-## PTX inline assembly (the core 4-instruction sequence)
-
-```cpp
-float d0 = 0.f, d1 = 0.f, d2 = 0.f, d3 = 0.f;  // 4 f32 accumulators per thread
-uint64_t descA = make_smem_desc(smem_a, 256, 256, 0);
-uint64_t descB = make_smem_desc(smem_b, 256, 256, 0);
-
-asm volatile("wgmma.fence.sync.aligned;\n");
-asm volatile(
-    "wgmma.mma_async.sync.aligned.m64n8k16.f32.bf16.bf16 "
-    "{%0, %1, %2, %3}, %4, %5, 1, 1, 1, 0, 0;\n"
-    : "+f"(d0), "+f"(d1), "+f"(d2), "+f"(d3)
-    : "l"(descA), "l"(descB));
-asm volatile("wgmma.commit_group.sync.aligned;\n");
-asm volatile("wgmma.wait_group.sync.aligned 0;\n");
-```
-
-Immediate arguments for `m64n8k16.f32.bf16.bf16`: `scaleD, scaleA, scaleB, transA, transB` (5 values). For tf32 atoms: 3 values (no trans flags). For s8 atoms: 1 value (scaleD only).
-
-## Accumulator fragment layout
-
-For `m64n8k16`: each of the 128 threads in the warpgroup owns 4 f32 values. The output matrix D is 64x8 = 512 elements. Fragment deposition follows the standard m64xN mapping:
-- `row_base = warp_id * 16 + (lane_id >> 2)`
-- `col_base = (lane_id & 0x3) * 2`
-- Thread writes `d0, d1` at `(row_base, col_base)` and `(row_base, col_base+1)`, and `d2, d3` at `(row_base+8, col_base)` and `(row_base+8, col_base+1)`.
-
-## Measured result
-
-H200-SXM, sm_90a, cuda 12.9.86. All-ones input:
-- `D entries matching K=16: 512 / 512` (correctness gate pass)
-- Single-warpgroup serialized throughput: 0.74 TFLOPS (see zoo config `ss_tn_bf16_n8`)
-
-## Adapting to other atoms
-
-To switch to `m64n64k16.f32.bf16.bf16`, change the PTX mnemonic and increase the accumulator fragment to 32 floats per thread (64*64/128). To switch to tf32, use `m64n64k8.f32.tf32.tf32` and drop the `transA, transB` immediates.
-
-## Source
-
-Full runnable code: `artifacts/experience/hw-probes/wgmma-ptx/wgmma_hello.cu`
-Build script: `artifacts/experience/hw-probes/wgmma-ptx/build.sh`
-Measured record: `sources/experience/hw-probes/wgmma-ptx.md`

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/hardware/wgmma/pitfalls-wgmma-ptx.md

@@ -1,90 +0,0 @@
----
-id: pitfall-wgmma-ptx
-type: pitfall
-vendor: nvidia
-title: Pitfalls
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-- tma
-techniques:
-- shared-memory-optimization
-- swizzling
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- tma
-- shared-memory-optimization
-- swizzling
-- ptx
-- cuda-cpp
-- cute-dsl
----
-# Hopper wgmma via raw PTX — pitfalls
-
-## 1. `-arch=sm_90a` alone is insufficient for wgmma inline PTX
-
-nvcc 12.9 with `-arch=sm_90a` generates an intermediate `compute_90` (NO `a` suffix) PTX file that ptxas rejects with `Instruction 'wgmma.fence' not supported on .target 'sm_90'`. The fix is to use the explicit `-gencode=arch=compute_90a,code=sm_90a` form, which forces both the intermediate compute target AND the SM target to the `a` (architecture-specific) variant. See `build.sh` in the artifacts/ directory for the exact flags.
-
-## 2. wgmma is a warpgroup instruction — issue from exactly 128 threads
-
-`wgmma.mma_async.sync.aligned.*` is documented as a warpgroup instruction: it must be issued by all 128 threads (4 warps) of a warpgroup simultaneously. Issuing it from a single warp, or from 64 / 192 / 256 threads, is undefined behavior. The hello-world kernel uses `<<<1, 128, smem_bytes>>>` exactly for this reason.
-
-## 3. Smem descriptor must be 16-byte aligned
-
-The descriptor's address bits store `smem_addr >> 4`, so the smem allocation start must be 16-byte aligned. `extern __shared__ uint8_t smem[]` is naturally 16-byte aligned by CUDA's smem allocator, but if you carve up the buffer with non-16-byte-aligned offsets (e.g. an arbitrary uint8_t offset), the descriptor will silently encode the wrong start address.
-
-## 4. wgmma.fence + wgmma.wait_group are not optional
-
-Without the `wgmma.fence.sync.aligned` before the issue, the wgmma may race with previous shared-memory stores; without `wgmma.wait_group.sync.aligned 0` after, the per-thread fragment registers are not yet committed when downstream code reads them. Both fences MUST appear in the issue sequence, in the order shown in `wgmma_hello.cu`.
-
-## 5. The atom shape `m64nNkK` constrains the per-thread fragment count
-
-For `m64n8k16` bf16: 64×8 = 512 outputs, distributed across 128 threads = 4 floats per thread, packed as 2 register pairs. For `m64n128k8` tf32 (the cutlass-API reference): 64×128 = 8192 outputs, 64 floats per thread, 32 register pairs. The accumulator clobber list in the inline asm must enumerate all of them. Get the count wrong and ptxas will silently truncate or the kernel will hang at the wgmma issue.
-
-## 6. The swizzle mode interacts with leading-dim / stride-dim encoding
-
-For non-swizzled smem layouts, `swizzle = 0` and the leading-dim / stride-dim fields encode the actual byte stride. For 128B-swizzle layouts (`swizzle = 1`), the leading-dim field's interpretation changes — see PTX ISA spec section "Hopper Tensor Core Operations" for the table. This skill's hello-world uses no swizzle; production kernels typically use 128B swizzle for higher TMA bandwidth.
-
-## 7. Numerically equivalent to cutlass's wgmma at the same atom
-
-The cutlass `cute::SM90::GMMA::MMA_64x128x8_F32TF32TF32_SS_TN` atom compiles to the same `wgmma.mma_async.sync.aligned.m64n128k8.f32.tf32.tf32` instruction this skill issues by hand. The expected output is bit-identical for the same A/B inputs. If they diverge, the descriptor encoding (or per-thread fragment layout) is wrong; verify by setting up an identical input pair and comparing element-by-element.
-
-## 8. PTX immediate count varies by dtype family
-
-The `wgmma.mma_async` instruction takes a different number of trailing immediates depending on the dtype suffix. Get this wrong and ptxas rejects with `Arguments mismatch for instruction 'wgmma.mma_async'`:
-
-| Dtype suffix | Immediates after `descA, descB` | Form |
-|---|---|---|
-| `.f32.bf16.bf16`, `.f32.f16.f16`, `.f16.f16.f16` | 5 | `scaleD, scaleA, scaleB, transA, transB` |
-| `.f32.tf32.tf32` | 3 | `scaleD, scaleA, scaleB` (A and B are fixed K-major; no trans) |
-| `.s32.s8.s8`, `.s32.u8.u8`, mixed-sign | 1 | `scaleD` only (no scaleA / scaleB / trans) |
-
-This is easy to miss because the ptxas error message points only at the line number. The codegen at `artifacts/experience/hw-probes/wgmma-ptx/gen_wgmma_zoo.py` enumerates the cases.
-
-## 9. TF32 wgmma supports only the TN format
-
-For `.f32.tf32.tf32`, A and B are *spec-fixed* to K-major; there is no `transA`/`transB` immediate (and consequently no SS_NT / SS_NN / SS_TT variant). cute reflects this — only `MMA_64xNx8_F32TF32TF32_SS_TN` exists in `mma_traits_sm90_gmma.hpp`, with no NT/NN/TT siblings. By contrast, bf16/fp16 atoms exist in all four layouts.
-
-## 10. RS variant: A is always K-major
-
-For `wgmma_RS_<...>` (A in registers, B in smem), only `transB` is a free immediate; A's layout is fixed by the in-register fragment format. The PTX form is:
-```
-wgmma.mma_async.sync.aligned.m64nNk16.f32.bf16.bf16 {acc...}, {a0,a1,a2,a3}, descB, scaleD, scaleA, scaleB, transB;
-```
-For bf16 / fp16 m64xK16, each thread holds 4 × 32-bit registers (= 8 fp16/bf16 elements per K-row slice). Where the bytes for those registers come from is *not* dictated by the wgmma instruction — it is the kernel's job to arrange them per the per-thread fragment layout in PTX ISA section "Asynchronous Warpgroup-Level Matrix Instructions / Matrix Fragments". Get the load wrong and the result is *layout-permuted but not catastrophic*; an all-ones-input correctness gate cannot detect this. Use non-uniform inputs to expose the bug.
-
-## 11. Single-warpgroup serialized wgmma plateaus at ~5.6 TFLOPS on H200
-
-A naive one-warpgroup-per-CTA kernel with a single accumulator chain caps near 5.6 TFLOPS regardless of N — the wgmma engine is throughput-bound at the issue port. The path to higher throughput on H200 is *not* a larger atom; it is **multi-warpgroup-per-CTA** (cutlass cooperative scheme uses 2 producer + 2 consumer warpgroups), **multi-accumulator pipelining** (typically 2-4 chains so consecutive wgmmas don't depend on each other), and **multi-CTA scaling** (132 SMs). These three multipliers compose to lift the floor by ~1000× and reach the ~990 TFLOPS device-peak figure. Treat the 5.6 TFLOPS plateau as a baseline / single-instance characterization, not a target.
-
-## 12. Dtype is a precision/range knob, not a throughput knob
-
-At fixed atom shape (N=64, SS_TN), bf16 / fp16 / tf32 / s8 all issue at the same per-instance rate. The TFLOPS-difference reported in benchmarks is purely from the per-instance MAC count (= K's worth of scalar MACs). Choose dtype on the precision × dynamic-range trade alone — there is no "faster" dtype on Hopper wgmma at the same K.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/hardware/wgmma/pitfalls-wgmma.md

@@ -1,79 +0,0 @@
----
-id: pitfall-wgmma
-type: pitfall
-vendor: nvidia
-title: Pitfalls
-architectures:
-- sm90
-- sm90a
-- sm100
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-hardware_features:
-- wgmma
-- mbarrier
-techniques:
-- cache-policy
-- shared-memory-optimization
-- swizzling
-kernel_types:
-- gemm
-confidence: inferred
-evidence_level: inferred
-tags:
-- wgmma
-- mbarrier
-- cache-policy
-- shared-memory-optimization
-- swizzling
-- gemm
-- ptx
-- cuda-cpp
-- cute-dsl
----
-# wgmma — pitfalls
-
-## 1. `-arch=sm_90a`, not `-arch=sm_90`
-
-wgmma encodings live in the architecture-specific extension `sm_90a`. Plain `sm_90` will not emit wgmma PTX even if the cutlass templates instantiate it; you'll get either a `ptxas` error or, worse, a silently downgraded mma.sync path on some toolchain combinations. The canonical build script at `artifacts/experience/api-probes/gemm/build.sh` pins `-arch=sm_90a`.
-
-## 2. **Do not** copy Ampere `mma.sync` / `wmma` experience verbatim
-
-On Ampere a single warp issues `mma.sync.aligned.m16n8k16` (16 threads cooperate). On Hopper a wgmma issue is at the **warpgroup** granularity (128 threads = 4 warps cooperate on a 64-row tile). Carrying Ampere assumptions about register layouts, sync semantics, or accumulator placement to wgmma produces correct-but-slow kernels (or worse, race-condition surprises). Specifically:
-
-- Ampere: `__syncthreads()` between consecutive mmas; Hopper: `wgmma.fence` + `wgmma.commit_group` + `wgmma.wait_group<N>` decouples issue from completion.
-- Ampere: accumulator can sit in registers OR shared memory; Hopper: accumulator must be in **registers** of the consumer warpgroup throughout.
-- Ampere: warps are independent; Hopper: a warpgroup is the unit of issue and **all 4 warps must stay live** until `wgmma.wait_group` clears.
-
-## 3. SmemLayoutAtom must be wgmma-compatible
-
-A wrong smem layout gives one of two failure modes:
-
-- **Build-time failure**: cutlass's `static_assert` chain rejects the layout (most common; the error message is verbose but pointable).
-- **Runtime silent corruption**: the kernel runs, but A or B operands are read from the wrong banks. The cutlass example uses `Swizzle<3,4,3>` over a 32-bit unit; deviating without understanding the SS-form access pattern is a foot-gun. Use `cute::GMMA::ss_op_selector` or the high-level `GMMA::CtaTileMNK` helpers.
-
-## 4. Block size must cover ≥1 warpgroup
-
-128 threads = 1 warpgroup. The cooperative kernel uses 3 warpgroups (1 producer + 2 consumers). A 64-thread block cannot issue wgmma at all; cutlass will refuse to instantiate.
-
-## 5. F32 accumulator only (on Hopper sm_90a)
-
-The `wgmma.mma_async.sync.aligned.m64nNkK.f32.f16.f16.f32` and similar mnemonics all return **F32** accumulator regardless of input dtype on sm_90a. Do not try to declare a `cute::Tensor<float16_t, …>` accumulator and expect wgmma to emit the f16 form — it doesn't on Hopper. (Blackwell sm_100 has different rules; not relevant here.)
-
-## 6. Kernel-name template-mangling explosion at compile time
-
-The mangled name of the cutlass cooperative kernel is several KB of template parameters. nvcc's symbol table can compile them in 30-60 s for a single kernel, and bypassing build caches recompiles them every iteration. Set up `ccache` or use the cutlass profiler to amortize. The single-kernel build in our `build.sh` takes ~25 s on H200.
-
-## 7. ncu's `wgmma`-specific counter is unavailable on driver 570.124.06
-
-`smsp__inst_executed_pipe_wgmma.sum` returns `n/a` on this stack. Use `sm__inst_executed_pipe_tensor_op_hmma.sum` as the proxy — it correctly counts wgmma issues on Hopper. (The "hmma" name is a holdover from Ampere; on Hopper this counter folds wgmma in.)
-
-## 8. Profiler attach overhead distorts kernel time
-
-Cutlass's own timer reports ~0.608 ms for the example GEMM unprofiled, ~0.799 ms under ncu instrumentation (~30 % overhead). When comparing wgmma shape variants, take both numbers from the same instrumentation level — never compare `--metrics …` runs against unprofiled runs.
-
-## 9. Producer/consumer warpgroup deadlocks
-
-If a producer warpgroup increments an mbarrier the consumer is waiting on by **fewer** than the consumer's expected arrival count, the entire CTA hangs forever (no timeout, no error). Mirror cutlass's `Sm90Pipeline` arrival counts exactly; do not "simplify" them.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/hardware/wgmma/skill-wgmma-ptx.md

@@ -1,252 +0,0 @@
----
-title: Hopper wgmma via raw PTX (cutlass-free)
-status: verified
-evidence_level: measured
-applies_to_pattern_class:
-- tensor-core
-- tensor-core/gemm
-applies_to_ops:
-- gemm
-- mma
-requires_sm: '>=9.0a'
-requires_features:
-- wgmma
-single_kernel_useful: true
-cuda_version_tested: 12.9.86
-driver_version_tested: 570.124.06
-toolchain: nvcc 12.9 + ptxas 12.9
-measured_on: H200-SXM | sm_90a | cuda 12.9.86 | driver 570.124.06
-source:
-- path: sources/experience/hw-probes/wgmma-ptx.md
-  anchor: cutlass-free hello-world (single m64n8k16 bf16 atom)
-- path: sources/experience/hw-probes/wgmma-ptx.md
-  anchor: cutlass-free zoo (11 configs across N-shape, dtype, layout, A-source)
-artifacts:
-  code: artifacts/experience/hw-probes/wgmma-ptx/wgmma_hello.cu
-  build: artifacts/experience/hw-probes/wgmma-ptx/build.sh
-  zoo_code: artifacts/experience/hw-probes/wgmma-ptx/wgmma_zoo.cu
-  zoo_codegen: artifacts/experience/hw-probes/wgmma-ptx/gen_wgmma_zoo.py
-  zoo_build: artifacts/experience/hw-probes/wgmma-ptx/build_zoo.sh
-  zoo_run: artifacts/experience/hw-probes/wgmma-ptx/run_zoo.sh
-  profile: artifacts/experience/hw-probes/wgmma-ptx/2026-04-29-wgmma-zoo.csv
-related_apis: []
-related_skills:
-- wgmma
-- tma-ptx
-- gemm-ptx
-id: skill-wgmma-ptx
-type: skill
-vendor: nvidia
-tags:
-- cuda-cpp
-- wgmma
-- tma
-- pipeline-stages
-- register-budgeting
-- shared-memory-optimization
-- swizzling
-- tile-scheduling
-- gemm
-- ptx
-- cute-dsl
-- python
-applies_to:
-- general
-source_refs:
-- source_id: source-code/cutlass
-  path: include/cute/arch/mma_sm90_gmma.hpp
-  anchor: cutlass's wgmma inline-PTX wrapper (used as a reference for descriptor construction; not included in our binary)
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/parallel-thread-execution/cuda_parallel-thread-execution_index.html.md
-  anchor: L28659-L28675
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/parallel-thread-execution/cuda_parallel-thread-execution_index.html.md
-  anchor: L28451-L28465
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/parallel-thread-execution/cuda_parallel-thread-execution_index.html.md
-  anchor: L28674-L28695
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/parallel-thread-execution/cuda_parallel-thread-execution_index.html.md
-  anchor: L29622-L29640
-- source_id: blogs/colfax
-  path: cutlass-tutorial-fast-matrix-multiplication-with-wgmma-on-nvidia-hopper-gpus
-  anchor: Hopper wgmma walkthrough — descriptor format + sync semantics
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-- cute-dsl
-- python
-hardware_features:
-- wgmma
-- tma
-techniques:
-- pipeline-stages
-- register-budgeting
-- shared-memory-optimization
-- swizzling
-- tile-scheduling
-kernel_types:
-- gemm
-confidence: experimental
-artifact_dir: artifacts/experience/hw-probes/wgmma-ptx
----
-# Hopper wgmma via raw PTX (cutlass-free)
-
-## What it is
-
-A minimal implementation of Hopper's `wgmma.mma_async.sync.aligned.m64n8k16.f32.bf16.bf16` instruction issued from inline PTX in a plain CUDA kernel that **does NOT include any cutlass or cute headers**. This is the cutlass-free wgmma primitive layer: it proves the agent can drive Hopper wgmma without depending on cutlass/cute, which is the prerequisite for a cutlass-free GEMM.
-
-Compared to the cutlass-API track (which uses cutlass's `MMA_64x128x8_F32TF32TF32_SS_TN` atom via cute), this skill:
-
-- Constructs the wgmma smem descriptor by hand (bit-layout per the PTX ISA spec).
-- Issues `wgmma.fence`, `wgmma.mma_async`, `wgmma.commit_group`, `wgmma.wait_group` via inline PTX.
-- Allocates fragments as plain C++ floats; no `cute::Tensor`, no `cute::Layout`, no `cutlass::Array`.
-- Verifies the binary has zero `cutlass::` / `cute::` symbols at the preprocessor level (`nvcc -E | grep`) AND at the linked-symbol level (`cuobjdump --dump-elf-symbols | grep`).
-
-## When to use it
-
-- Building cutlass-free kernels (prerequisite for the cutlass-free GEMM).
-- Auditing what cutlass's wgmma wrapper (`cute/arch/mma_sm90_gmma.hpp`) does at the PTX level — the wrapper itself is a thin inline-asm layer over the same instructions.
-- Educational: understanding how the wgmma descriptor encoding works.
-
-## When NOT to use it
-
-- Production GEMM kernels where cutlass's tile / pipeline / scheduler abstractions would save thousands of lines of code. Cutlass exists for a reason; the cutlass-free track is "extract and rewrite", not "abandon cutlass for production".
-- A wgmma issue rate above the **single-warpgroup serialized ~5.6 TFLOPS** ceiling. Real GEMM throughput requires multi-warpgroup-per-CTA + multi-accumulator pipelining + multi-CTA scaling — none of which this skill's primitives provide alone.
-
-## PTX mnemonic inventory (this skill)
-
-| Mnemonic | Purpose |
-|---|---|
-| `wgmma.fence.sync.aligned` | Barrier between non-wgmma SASS preceding the wgmma issue and the wgmma instructions; flushes outstanding dependent instructions. |
-| `wgmma.mma_async.sync.aligned.m64n8k16.f32.bf16.bf16 {acc}, descA, descB, scaleD, scaleA, scaleB, transA, transB` | The wgmma instruction itself. Reads matrices A/B from smem (via descriptors), accumulates into per-thread register fragment `{acc}`. |
-| `wgmma.commit_group.sync.aligned` | Closes the current wgmma issue group. |
-| `wgmma.wait_group.sync.aligned 0` | Waits for the group to retire so the accumulator fragments are stable for downstream stores. |
-
-For the m64n128k8 tf32 atom (the cutlass-API reference), substitute `m64n128k8.f32.tf32.tf32` and adjust the per-thread fragment layout (64 floats per thread instead of 4).
-
-## Smem descriptor format (uint64_t)
-
-| Bits | Field | Notes |
-|---|---|---|
-| 0..13 | smem byte address >> 4 | The matrix start address in shared memory, divided by 16 (wgmma requires 16-byte alignment) |
-| 14..29 | leading-dim byte offset >> 4 | Stride between rows (or cols, depending on layout) of the matrix, divided by 16 |
-| 30..45 | stride-dim byte offset >> 4 | Stride between consecutive 8-row groups (the wgmma fragment unit), divided by 16 |
-| 46..48 | matrix base offset | Used for 128B swizzle (set to 0 for no swizzle) |
-| 49..51 | reserved | Must be 0 |
-| 52..61 | fixed offset | Set to 0 |
-| 62..63 | swizzle mode | 00=none, 01=128B, 10=64B, 11=32B |
-
-Construction helper (see `wgmma_hello.cu`):
-
-```cpp
-__device__ __forceinline__ uint64_t make_smem_desc(
-    const void* smem_ptr, uint32_t ld_bytes, uint32_t sd_bytes, uint32_t swizzle = 0) {
-    uint32_t smem_int = static_cast<uint32_t>(__cvta_generic_to_shared(smem_ptr));
-    uint64_t desc = 0;
-    desc |= ((uint64_t)(smem_int >> 4)) & 0x3FFF;
-    desc |= ((uint64_t)(ld_bytes >> 4) & 0xFFFF) << 14;
-    desc |= ((uint64_t)(sd_bytes >> 4) & 0xFFFF) << 30;
-    desc |= ((uint64_t)(swizzle & 0x3)) << 62;
-    return desc;
-}
-```
-
-## Atom-shape and dtype family covered
-
-The cutlass-free PTX path covers the full Hopper wgmma family. The 11-config zoo at `sources/experience/hw-probes/wgmma-ptx.md` exercises four orthogonal axes — *N-shape*, *element dtype*, *AB layout*, *A-source location* — all verified cutlass-free.
-
-| Axis | Values measured | Notes |
-|---|---|---|
-| N-shape (bf16 SS_TN K=16) | 8, 16, 32, 64, 128, 256 | M=64 fixed for all bf16/fp16 atoms |
-| dtype (N=64 SS_TN) | bf16 (K=16), fp16 (K=16), tf32 (K=8), s8 (K=32) | f32-acc for floats; s32-acc for int8 |
-| layout | SS_TN (transA=0, transB=0), SS_NT (transA=1, transB=1) | NN / TT supported by spec but not measured |
-| A-source | SS (smem descriptor), RS (4 × 32-bit regs / thread for K=16) | RS skips A's smem footprint |
-
-PTX immediates count differs by dtype family — get this wrong and ptxas rejects with `Arguments mismatch for instruction 'wgmma.mma_async'`:
-
-| Dtype | Immediates after descA, descB | Form |
-|---|---|---|
-| `.f32.bf16.bf16`, `.f32.f16.f16`, `.f16.f16.f16` | 5 | `scaleD, scaleA, scaleB, transA, transB` |
-| `.f32.tf32.tf32` | 3 | `scaleD, scaleA, scaleB` (A and B are fixed K-major; no trans flags) |
-| `.s32.s8.s8`, `.s32.u8.u8` | 1 | `scaleD` only (no scaleA/scaleB; no trans) |
-
-## Measured Characteristics
-
-### Correctness (hello-world)
-
-H200-SXM, sm_90a. The single-instance hello-world kernel issues one m64n8k16 bf16 wgmma with all-1.0 inputs and verifies all 512 outputs equal K = 16 (matched / total = 512 / 512).
-
-### Single-warpgroup throughput (11-config zoo)
-
-H200-SXM, sm_90a. 1 CTA × 128 threads × N_INNER=1024 serialized wgmma issues per launch (accumulator-chained — every iteration depends on the previous, the worst-case pipeline serialization). All-ones inputs, every output = K × N_INNER. 5 warmup + 20 timed launches, median ms. **Every config passes the matched/total correctness gate.** Source `2026-04-29-wgmma-zoo.csv`.
-
-| config            | N | K | TFLOPS (single-warpgroup serialized) |
-|-------------------|--:|--:|-------:|
-| ss_tn_bf16_n8     |   8 | 16 |   0.74 |
-| ss_tn_bf16_n16    |  16 | 16 |   1.44 |
-| ss_tn_bf16_n32    |  32 | 16 |   2.77 |
-| ss_tn_bf16_n64    |  64 | 16 |   4.70 |
-| ss_tn_bf16_n128   | 128 | 16 |   5.32 |
-| ss_tn_bf16_n256   | 256 | 16 | **5.63** |
-| ss_tn_fp16_n64    |  64 | 16 |   4.69 |
-| ss_tn_tf32_n64    |  64 |  8 |   2.35 |
-| ss_tn_s8_n64      |  64 | 32 | **9.55** |
-| ss_nt_bf16_n64    |  64 | 16 |   4.70 |
-| rs_tn_bf16_n64    |  64 | 16 |   4.69 |
-
-**Important scale caveat**: these TFLOPS are *single CTA × 1 warpgroup × full accumulator-dependency chain*. Device peak (~990 TFLOPS bf16 on H200) requires multiplying by ~2 warpgroups × ~4 accumulator chains × 132 SMs ≈ 1000×. The numbers here characterize the per-instance issue cost, not engineering performance.
-
-## Throughput shape — three independent multipliers
-
-1. **N-shape**: doubling N below N=64 ≈ doubles TFLOPS (issue latency floor dominated by per-instance overhead). Above N=64 the instance itself takes more cycles per FMA, so TFLOPS plateaus near 5.6 — adding N just spends more cycles per issue.
-2. **Dtype**: at N=64 SS_TN, normalized TFLOPS (`measured × 16 / K`) is ≈ 4.70 for all four dtypes. **Dtype is a precision/range knob, not a throughput knob.** s8 reports 9.55 only because K=32 (2× MACs/instance vs bf16 K=16); tf32 reports 2.35 only because K=8 (½× MACs).
-3. **Layout & A-source**: SS_TN, SS_NT, RS_TN are within 0.3 % of each other at N=64 bf16. Trans flags reinterpret the descriptor / register pack; A in registers vs smem changes feed but not issue rate.
-
-## Recommended atom selection (engineering rule of thumb)
-
-| Constraint | Choose |
-|---|---|
-| Maximize FLOPs per warpgroup serialized issue | **N=128** for bf16/fp16 (5.32 TFLOPS — only 6 % below N=256, half the registers) |
-| Minimize register pressure | N=64 (4.70 TFLOPS, 32 fp32 accum / thread vs 64 at N=128) |
-| Match cutlass production atom (`MMA_64x128x8_F32TF32TF32_SS_TN`) | tf32 N=128 K=8 (not in the zoo; same instruction family) |
-| Minimize accumulator dtype size | s32 + s8 inputs (K=32) — 2× the FLOP density of bf16/fp16 |
-
-## Atomic Usage (PTX)
-
-The raw PTX template for a single wgmma atom (bf16 SS_TN example):
-
-```asm
-// Fence: ensure all prior register/smem writes are visible to wgmma
-wgmma.fence.sync.aligned;
-
-// Issue: M=64, N=8, K=16, f32 accumulator, bf16 A and B from smem descriptors
-wgmma.mma_async.sync.aligned.m64n8k16.f32.bf16.bf16
-    {%f0, %f1, %f2, %f3},     // 4 f32 accumulators per thread
-    descA,                      // uint64_t descriptor for A matrix in smem
-    descB,                      // uint64_t descriptor for B matrix in smem
-    1,                          // scaleD: 1 = accumulate, 0 = overwrite
-    1, 1,                       // scaleA, scaleB: 1 = +1.0
-    0, 0;                       // transA=0 (K-major), transB=0 (K-major)
-
-// Commit the issue group
-wgmma.commit_group.sync.aligned;
-
-// Wait for the group to retire
-wgmma.wait_group.sync.aligned 0;
-```
-
-The number of accumulator registers per thread scales with N: N=8 gives 4 floats, N=64 gives 32, N=128 gives 64, N=256 gives 128. The immediate arguments also vary by dtype family (see the table in "Atom-shape and dtype family covered" above).
-
-A complete self-contained atomic skeleton is at [wiki/nvidia/hardware/wgmma-ptx/atomic_skeleton.md](atomic_skeleton.md). The skeleton compiles and runs on H200 as a single m64n8k16 bf16 atom, producing the expected output (all 512 elements = K = 16). Build command: `nvcc -gencode=arch=compute_90a,code=sm_90a -o wgmma_hello wgmma_hello.cu` (see `artifacts/experience/hw-probes/wgmma-ptx/build.sh`).
-
-## Cross-references
-
-- wgmma reference (cutlass-API path): `wiki/nvidia/hardware/wgmma/skill.md` + `sources/experience/api-probes/gemm.md`.
-- 11-config zoo with full sweep + open questions: `sources/experience/hw-probes/wgmma-ptx.md`.
-- TMA-PTX sibling (also cutlass-free): `wiki/nvidia/hardware/tma/skill-tma-ptx.md`.
-- Cutlass-free GEMM (composes both PTX primitives): `wiki/nvidia/foundations/compute/gemm-ptx.md`.
-- Failure modes: `wiki/nvidia/hardware/wgmma/pitfalls-wgmma-ptx.md`.