@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/memory/async-copy/pitfalls.md

@@ -1,207 +0,0 @@
----
-title: Asynchronous Data Copies - Pitfalls
-status: draft
-evidence_level: spec
-applies_to_pattern_class:
-- cuda-core
-applies_to_ops:
-- elementwise
-- reduction
-- normalization
-- stencil
-- scan
-requires_sm: '>=8.0'
-single_kernel_useful: true
-source:
-- path: spec
-  anchor: ldgsts-apis-and-modes
-  excerpt: L1 BYPASS -- sizeof(datatype) and alignment must be 16 bytes. producer_acquire() and producer_commit() must be called from converged code.
-id: pitfall-async-copy
-type: pitfall
-vendor: nvidia
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L11266-L11269
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L11267-L11268
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L11269
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-hardware_features:
-- tma
-techniques:
-- pipeline-stages
-- cache-policy
-- shared-memory-optimization
-kernel_types:
-- fused-kernel
-confidence: source-reported
-tags:
-- tma
-- pipeline-stages
-- cache-policy
-- shared-memory-optimization
-- fused-kernel
-- cuda-cpp
----
-## P1: Alignment requirements for L1 BYPASS mode
-
-**Symptom**: LDGSTS falls back to L1 ACCESS mode (4-byte loads through L1), adding unnecessary L1 cache pollution and missing the potential bandwidth benefit of L1 BYPASS.
-
-**Cause**: L1 BYPASS mode requires all of the following (programming guide L11266, GTC25-S72683 slide interval_0093):
-- Copy size is exactly 16 bytes.
-- Both source (global memory) and destination (shared memory) pointers are 16-byte aligned.
-- The copy is expressed through `cuda::aligned_size_t<16>(size)` or an equivalent 16-byte type.
-
-If any requirement is violated, the hardware silently falls back to L1 ACCESS mode.
-
-**Example**:
-```cuda
-// BAD: 4-byte copy -- always L1 ACCESS mode
-cuda::memcpy_async(&smem[tid], &gmem[idx], sizeof(float), pipe);
-
-// GOOD: 16-byte copy with alignment hint -- L1 BYPASS mode
-// Only BLOCK_DIM/4 threads each copy 16 bytes
-if (tid < BLOCK_DIM / 4) {
-    cuda::memcpy_async(&smem[tid * 4], &gmem[offset + tid * 4],
-                       cuda::aligned_size_t<16>(4 * sizeof(float)), pipe);
-}
-```
-
-**Fix**: Restructure the copy so that each participating thread copies 16 bytes. Use `memcpy_threads = BLOCK_DIM / 4` threads, each copying 4 floats (16 bytes). Ensure global memory allocations are 128-byte aligned (cudaMalloc guarantees at least 256-byte alignment). Ensure shared memory arrays start at 16-byte boundaries.
-
-**Detection**: Check the SASS disassembly for `LDGSTS.BYPASS` vs `LDGSTS` instructions. Alternatively, compare L1 hit rates in Nsight Compute: L1 BYPASS mode should show no L1 hits for the async copy traffic.
-
-## P2: `producer_commit` from diverged code
-
-**Symptom**: Excessive `LDGDEPBAR` (dependency barrier) instructions in the compiled SASS, or pipeline stages that complete out of order.
-
-**Cause**: `pipe.producer_commit()` (and `__pipeline_commit()`) is a warp-level instruction. If it is called inside a divergent branch, the compiler emits one commit per conditional path, because it cannot prove that all threads in the warp take the same path (GTC25-S72683, slide interval_0119).
-
-**Example**:
-```cuda
-// BAD: commit inside conditional code
-if (idx < N) {
-    cuda::memcpy_async(&smem[tid], &gmem[idx], sizeof(float), pipe);
-    pipe.producer_commit();  // compiler may emit multiple commits
-}
-
-// GOOD: commit in converged code outside the conditional
-if (idx < N) {
-    cuda::memcpy_async(&smem[tid], &gmem[idx], sizeof(float), pipe);
-}
-pipe.producer_commit();  // single commit for the entire warp
-```
-
-**Fix**: Always place `producer_acquire()` and `producer_commit()` (or `__pipeline_commit()`) in code paths where the entire warp converges. Move them outside of `if` statements.
-
-## P3: NUM_STAGES not a compile-time constant
-
-**Symptom**: Pipeline operations compile but performance is poor; the compiler generates unnecessary bookkeeping instructions.
-
-**Cause**: `cuda::pipeline_consumer_wait_prior<N>(pipe)` requires `N` to be a compile-time constant (template parameter). If `NUM_STAGES` is a runtime variable, the template cannot be instantiated and the code either fails to compile or falls back to a less efficient wait mechanism (GTC25-S72683, slide interval_0130).
-
-**Example**:
-```cuda
-// BAD: runtime stage count
-int num_stages = compute_stages();
-cuda::pipeline_consumer_wait_prior<num_stages - 1>(pipe);  // compile error
-
-// GOOD: compile-time constant
-constexpr int NUM_STAGES = 2;
-cuda::pipeline_consumer_wait_prior<NUM_STAGES - 1>(pipe);  // OK
-```
-
-**Fix**: Define `NUM_STAGES` as a `constexpr` or template parameter. If different stage counts are needed for different shapes, use a `switch`-`case` or `if constexpr` dispatch over a small set of values.
-
-## P4: Missing `__syncthreads()` in producer-consumer pattern
-
-**Symptom**: Threads read stale or partially-written data from shared memory. Correctness failures that are non-deterministic.
-
-**Cause**: When only a subset of threads performs the async copies (e.g., `BLOCK_DIM/4` threads for 16-byte L1 BYPASS copies) but all threads consume the data, the pipeline wait only guarantees that the *copying threads* see the data as ready. Other threads in the block need an explicit `__syncthreads()` after the pipeline wait to ensure visibility (programming guide L11269, GTC25-S72683 slide interval_0142).
-
-**Example**:
-```cuda
-const int memcpy_threads = BLOCK_DIM / 4;
-
-// Producer: only first memcpy_threads threads copy
-if (tid < memcpy_threads) {
-    cuda::memcpy_async(&smem[tid * 4], &gmem[offset + tid * 4],
-                       cuda::aligned_size_t<16>(4 * sizeof(float)), pipe);
-}
-pipe.producer_commit();
-
-// BAD: missing __syncthreads() -- non-copying threads may see stale data
-cuda::pipeline_consumer_wait_prior<NUM_STAGES - 1>(pipe);
-float val = smem[tid];  // thread tid >= memcpy_threads reads stale data
-
-// GOOD: add __syncthreads() after the wait
-cuda::pipeline_consumer_wait_prior<NUM_STAGES - 1>(pipe);
-__syncthreads();
-float val = smem[tid];  // all threads see the correct data
-```
-
-**Fix**: Insert `__syncthreads()` between the pipeline wait and the compute step whenever a **subset** of threads performs the copies. If all threads copy their own element (the simpler pattern), this is not needed because each thread only reads data it copied.
-
-## P5: TMA peeling loop without `invoke_one`
-
-**Symptom**: TMA instructions execute sequentially across all active threads in a warp instead of once, wasting bandwidth and cycles.
-
-**Cause**: TMA is warp-uniform: it should be issued by exactly one thread per warp. When wrapped in `if (threadIdx.x == 0) { ... }`, the compiler does not know that only one thread enters the branch, so it emits a **peeling loop** that iterates the TMA instruction for each active lane (GTC25-S72683, slides interval_0193 and interval_0196).
-
-**Example**:
-```cuda
-// BAD: compiler emits peeling loop
-if (threadIdx.x == 0) {
-    cuda::memcpy_async(smem_ptr, global_ptr,
-                       cuda::aligned_size_t<16>(num_bytes), bar);
-}
-
-// GOOD: invoke_one tells the compiler only one thread executes
-namespace cg = cooperative_groups;
-cg::invoke_one(cg::coalesced_threads(), [&] {
-    cuda::memcpy_async(smem_ptr, global_ptr,
-                       cuda::aligned_size_t<16>(num_bytes), bar);
-});
-```
-
-**Fix**: Use `cooperative_groups::invoke_one(cg::coalesced_threads(), ...)` to wrap TMA instructions. This is a Hopper+ concern and applies when using TMA 1D/ND, not LDGSTS.
-
-## P6: Applying async copies to trivially simple kernels
-
-**Symptom**: The async-copy version of a kernel is **slower** than the vanilla version.
-
-**Cause**: For very simple compute (e.g., `c[i] = a[i] * b[i]`), the overhead of shared memory staging -- `commit`, `wait_prior`, `release` instructions plus SMEM reads -- exceeds the latency-hiding benefit. Our H200 probe confirmed this: the 2-stage LDGSTS kernel was ~10% slower than vanilla for `a*b` compute.
-
-GTC25-S72683 explicitly states: "kernels that read GMEM -> write SMEM -> compute -> write back see only a minor improvement just by switching to async copies -- unless they can batch loads. The big wins are on iterative kernels that can prefetch data for future iterations, especially low-occupancy compute-heavy kernels."
-
-**Fix**: Before adding async copies, check:
-1. Is the kernel actually bottlenecked by "Stall Long Scoreboard" in Nsight Compute warp-state statistics? If not, async copies will not help.
-2. Is the compute intensity high enough to amortize the staging overhead? GTC data shows that `sqrt` chains give 1.3x uplift, but trivial `a*b` gives no benefit.
-3. Can the kernel genuinely prefetch -- i.e., does it have at least 2 iterations where future data can be loaded while current data is consumed?
-
-## P7: Forgetting to drain the pipeline epilogue
-
-**Symptom**: The last 1-2 stages' worth of data is never consumed, producing incorrect results for the tail elements.
-
-**Cause**: The prologue fills `NUM_STAGES` stages, and the main loop prefetches one stage ahead while consuming one. At the end of the input, there are `NUM_STAGES - 1` stages still in flight. If the loop terminates without waiting for and consuming these remaining stages, the tail data is lost.
-
-**Fix**: After the main loop, add an epilogue that waits for and consumes the remaining stages:
-
-```cuda
-// Epilogue: drain remaining stages
-for (int s = 0; s < NUM_STAGES - 1 && remaining > 0; ++s) {
-    __pipeline_wait_prior(0);  // wait for all
-    // consume stage
-    __pipeline_commit();       // keep pipeline counter balanced
-}
-```
-
-Alternatively, structure the main loop to include both the prologue and epilogue stages in its iteration count.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/memory/async-copy.md

@@ -1,317 +0,0 @@
----
-title: Asynchronous Data Copies (LDGSTS)
-status: draft
-evidence_level: measured
-applies_to_pattern_class:
-- cuda-core
-applies_to_ops:
-- elementwise
-- reduction
-- normalization
-- stencil
-- scan
-requires_sm: '>=8.0'
-requires_features:
-- cp.async
-single_kernel_useful: true
-cuda_version_tested: '12.9'
-driver_version_tested: 570.124.06
-toolchain: nvcc 12.9 + ptxas 12.9
-measured_on: H200-SXM
-source:
-- path: spec
-  anchor: part-1-maximizing-memory-bandwidth
-  excerpt: 'Asynchronous copies let the data go directly from global memory to shared memory, skipping the register file. Benefits: free up registers for compute, reduce L1 traffic, reduce MIO pressure.'
-artifacts:
-  code: artifacts/experience/hw-probes/async-copy/async_copy_probe.cu
-  build: nvcc -arch=sm_90a -O3 -std=c++17 -lineinfo -o async_copy_probe async_copy_probe.cu
-  introspection: ''
-  profile: ''
-related_apis:
-- __pipeline_memcpy_async
-- __pipeline_commit
-- __pipeline_wait_prior
-- cuda::memcpy_async
-- cuda::pipeline
-- cuda::barrier
-- cuda::aligned_size_t
-- cooperative_groups::memcpy_async
-related_skills:
-- ilp
-- vectorized-access
-- register-pressure
-- coalescing
-id: skill-async-copy
-type: skill
-vendor: nvidia
-tags:
-- cuda-cpp
-- tma
-- mbarrier
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- register-budgeting
-- loop-unrolling
-- shared-memory-optimization
-- fused-kernel
-applies_to:
-- general
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L3937-L3944
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L11259-L11269
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L940-L999
-architectures:
-- sm90
-- sm90a
-- sm100
-languages:
-- cuda-cpp
-hardware_features:
-- tma
-- mbarrier
-techniques:
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- register-budgeting
-- loop-unrolling
-- shared-memory-optimization
-kernel_types:
-- fused-kernel
-confidence: experimental
-artifact_dir: artifacts/experience/hw-probes/async-copy
----
-## What
-
-Asynchronous data copies (LDGSTS) are a hardware-accelerated mechanism (Ampere / cc 8.0+) that transfers data **directly from global memory to shared memory**, bypassing the register file. Traditional synchronous copies require a two-step `LDG` (global to register) + `STS` (register to shared) sequence; LDGSTS fuses this into a single asynchronous instruction that the hardware executes in the background while the thread continues computing.
-
-LDGSTS supports copying 4, 8, or 16 bytes per transfer. It has two modes (programming guide L11266):
-- **L1 ACCESS** (4-byte or 8-byte copies): data is also cached in L1.
-- **L1 BYPASS** (16-byte copies, 16-byte aligned): L1 is not polluted. Best performance when source and destination are 128-byte aligned.
-
-The completion of async copies is signaled through either a **pipeline** (`cuda::pipeline` or `__pipeline_*` primitives) or a **shared memory barrier** (`cuda::barrier`). The choice determines the synchronization granularity: pipelines are thread-scoped (cheaper, no inter-thread sync needed for the copy itself), while barriers are block-scoped (required when a subset of threads does the copying for the whole block).
-
-## Why: Little's Law and Bytes-in-Flight
-
-The fundamental motivation comes from **Little's Law** applied to GPU memory:
-
-```
-bytes-in-flight = bandwidth x mean_latency
-```
-
-Bandwidth and latency are fixed by hardware; **bytes-in-flight is the only knob** the programmer controls. To achieve high DRAM bandwidth utilization, the software must keep enough bytes in-flight per SM.
-
-!BW trends across generations
-
-DRAM bandwidth per SM is increasing roughly 2x per generation, while SM count grows slowly. A simple `c[i] = a[i] + b[i]` kernel with 2 loads/thread, 4 bytes/load, 256 threads/block, and 8 blocks/SM produces only 16 KiB/SM of bytes-in-flight. The experimental targets for >90% BW utilization are (GTC25-S72683):
-
-!Little's Law bytes-in-flight scaling
-
-- **H100**: ~32 KiB/SM
-- **H200**: ~64 KiB/SM
-- **Blackwell (GB200-NVL)**: ~40 KiB/SM
-
-There are three techniques to grow bytes-in-flight:
-1. **ILP via unrolling** -- more independent loads per thread (see `ilp` skill).
-2. **DLP via vectorized loads** -- wider loads per instruction (see `vectorized-access` skill).
-3. **Asynchronous copies** -- LDGSTS and TMA (**this skill**).
-
-The first two increase bytes-in-flight at the cost of **register pressure**:
-
-!Register pressure vs BW utilization
-
-Asynchronous copies avoid this tradeoff by routing data through shared memory instead of registers.
-
-## How It Works: APIs and Programming Patterns
-
-### Three API Layers
-
-CUDA exposes LDGSTS through three API layers (GTC25-S72683, programming guide L3879-L3934):
-
-1. **Primitives API** (`<cuda_pipeline.h>`):
-   - `__pipeline_memcpy_async(dst, src, size)` -- initiate async copy
-   - `__pipeline_commit()` -- associate a barrier with pending copies
-   - `__pipeline_wait_prior(N)` -- wait until at most N stages are pending
-
-2. **libcudacxx** (`<cuda/pipeline>`, `<cuda/barrier>`):
-   - `cuda::memcpy_async(dst, src, size, pipe)` -- with pipeline completion
-   - `cuda::memcpy_async(dst, src, aligned_size_t<16>(size), barrier)` -- with barrier completion
-   - `cuda::pipeline<cuda::thread_scope_thread>` for thread-local staging
-   - `cuda::barrier<cuda::thread_scope_block>` for block-wide sync
-
-3. **Cooperative Groups** (`<cooperative_groups/memcpy_async.h>`):
-   - `cooperative_groups::memcpy_async(group, dst, src, size)` -- group-wide
-   - `cooperative_groups::wait(group)` or `wait_prior<N>(group)`
-
-**Recommendation** (GTC25-S72683): use the Primitives API or libcudacxx for fine-grained control. Cooperative groups is convenient for large block-wide copies but provides less synchronization granularity.
-
-### LDGSTS vs TMA Cheat-Sheet
-
-!LDGSTS / TMA matrix
-
-| Method | Alignment | When to use |
-| --- | --- | --- |
-| LDGSTS | 4, 8, or 16 bytes | Element-wise copies, tiles < 1 KiB |
-| TMA 1D | 16 bytes, size multiple of 16 | Bulk copies >= 2 KiB (Hopper+) |
-| TMA ND | SMEM 128B, GMEM 16B | Multi-dimensional tiles (Hopper+) |
-
-**Rule of thumb**: LDGSTS for small or irregular transfers; TMA for large contiguous blocks. TMA instructions have higher latency so they need more data to amortize their cost.
-
-### Basic LDGSTS Pattern (Primitives API)
-
-The simplest usage replaces `smem[tid] = gmem[idx]` with async copies:
-
-```cuda
-__shared__ float a_buf[BLOCK_DIM];
-__shared__ float b_buf[BLOCK_DIM];
-
-// Fire async copies
-__pipeline_memcpy_async(&a_buf[tid], &a[i], sizeof(float));
-__pipeline_memcpy_async(&b_buf[tid], &b[i], sizeof(float));
-
-// Commit and wait
-__pipeline_commit();
-__pipeline_wait_prior(0);  // wait for ALL pending copies
-
-// Compute from shared memory
-c[i] = a_buf[tid] * b_buf[tid];
-```
-
-### Multi-Stage Prefetching with `cuda::pipeline`
-
-The real power of LDGSTS comes from **multi-stage prefetching**: while the current iteration's data is being consumed from shared memory, the next iteration's data is being loaded in the background.
-
-```cuda
-__shared__ float a_buf[NUM_STAGES][BLOCK_DIM];
-__shared__ float b_buf[NUM_STAGES][BLOCK_DIM];
-
-cuda::pipeline<cuda::thread_scope_thread> pipe = cuda::make_pipeline();
-
-// Prologue: fill NUM_STAGES stages
-for (int s = 0; s < NUM_STAGES; ++s) {
-    pipe.producer_acquire();
-    cuda::memcpy_async(&a_buf[s][tid], &a[offset + s*stride], sizeof(float), pipe);
-    cuda::memcpy_async(&b_buf[s][tid], &b[offset + s*stride], sizeof(float), pipe);
-    pipe.producer_commit();
-}
-
-// Main loop
-int stage = 0;
-for (int i = 0; i < iterations; ++i) {
-    // Prefetch NUM_STAGES ahead
-    pipe.producer_acquire();
-    cuda::memcpy_async(&a_buf[stage][tid], &a[next_offset], sizeof(float), pipe);
-    pipe.producer_commit();
-
-    stage ^= 1;  // flip for 2-stage
-
-    // Wait for the current stage
-    cuda::pipeline_consumer_wait_prior<NUM_STAGES - 1>(pipe);
-
-    // Compute
-    c[ci] = a_buf[stage][tid] * b_buf[stage][tid];
-
-    // Release
-    pipe.consumer_release();
-}
-```
-
-**Critical**: `NUM_STAGES` must be a **compile-time constant** so the pipeline's internal bookkeeping instructions can be optimized away (GTC25-S72683, slide interval_0130).
-
-### Stage-Count Formula
-
-From the bytes-in-flight equation:
-
-```
-bytes-in-flight/SM = blocks/SM x threads/block x loads/thread x bytes/load
-```
-
-With async copies, `loads/thread = 2 x num_stages`. Solving for the target:
-- **H200**: target ~64 KiB/SM. With 8 blocks/SM, 256 threads/block, 4 bytes/load: 2 stages gives 32 KiB, 4 stages gives 64 KiB. **Minimum 2 stages recommended; 4 stages to fully saturate.**
-- **H100**: 2 stages suffices (32 KiB target).
-- **Blackwell**: 3 stages recommended (40 KiB target).
-
-### L1 BYPASS Mode
-
-To enable L1 BYPASS, each thread must copy exactly 16 bytes with 16-byte alignment. The common pattern reduces the number of copying threads:
-
-```cuda
-const int memcpy_threads = BLOCK_DIM / 4;
-if (tid < memcpy_threads) {
-    cuda::memcpy_async(&a_buf[stage][tid * 4],
-                       &a[offset + tid * 4],
-                       cuda::aligned_size_t<16>(4 * sizeof(float)),
-                       pipe);
-}
-// __syncthreads() is REQUIRED after pipeline wait
-// because only a subset of threads did the copies
-```
-
-This creates a **producer-consumer pattern** where `memcpy_threads` act as producers and all threads consume. The `__syncthreads()` after the pipeline wait is mandatory.
-
-## When to Use
-
-Async copies are most beneficial in these scenarios:
-
-- **Iterative kernels with prefetch opportunity**: the classic use case. If a kernel loops over tiles of global data, each tile can be prefetched while the previous tile is being computed.
-
-- **Compute-heavy kernels at low occupancy**: when the compute per element is expensive (sqrt chains, trig functions, etc.), async copies hide the memory latency that would otherwise dominate. GTC25-S72683 showed a **1.305x speedup** on H100 for `sqrt(sqrt(a)/sqrt(b))` vs vanilla.
-
-!2-stage benchmark a*b
-
-!2-stage benchmark sqrt
-
-- **Register-pressure-sensitive kernels**: when ILP/DLP techniques would push register usage too high and cause spilling or low occupancy, async copies provide bytes-in-flight without consuming registers.
-
-- **Stencil kernels with halo loads**: conditional code paths (loading left halo, center, right halo) prevent the compiler from batching loads optimally. LDGSTS forces all loads to be in-flight regardless of control flow (programming guide L11324-L11350).
-
-## When NOT to Use
-
-- **Simple elementwise kernels with low compute intensity**: as our H200 probe confirms, for trivial `c[i] = a[i] * b[i]`, the overhead of shared memory staging exceeds the latency-hiding benefit. The vanilla kernel already achieves 75% BW utilization at 16 KiB bytes-in-flight.
-
-- **Kernels already achieving >90% BW utilization**: if ILP/DLP techniques already saturate bandwidth, adding LDGSTS staging will only add overhead.
-
-- **Kernels with very small working sets**: if the working set fits in L1/L2 cache, the memory latency is already low and async copies add unnecessary SMEM traffic.
-
-- **Single-pass kernels without iteration**: the prefetch benefit requires at least 2 iterations to amortize the prologue cost.
-
-## Optimization Decision Flow
-
-!Optimization flow
-
-The GTC25-S72683 flowchart for increasing bytes-in-flight:
-
-```
-Enough bytes-in-flight? --YES--> Do nothing
-  | NO
-  v
-Prefetch in...?
-  |-- REG  --> Unroll / vectorize (ilp, vectorized-access skills)
-  |-- SMEM --> Aligned to...?
-              |-- 4, 8 bytes --> LDGSTS (this skill)
-              |-- 16 bytes   --> Size of tile
-                                 |-- < 1 KiB        --> LDGSTS
-                                 |-- 1 KiB - 2 KiB  --> LDGSTS or TMA (experiment)
-                                 |-- > 2 KiB        --> TMA
-```
-
-## TMA 1D: Brief Overview (Hopper+)
-
-TMA (Tensor Memory Accelerator) is Hopper's bulk async copy mechanism. **Full TMA coverage is deferred** to the `40-hardware-feature` layer (out of MVP scope); this section provides only the essential context.
-
-Key differences from LDGSTS:
-- TMA 1D requires 16-byte alignment and copy size as a multiple of 16 bytes.
-- Programming model is **warp-uniform**: issue from one thread per warp.
-- `cuda::memcpy_async` auto-uses TMA when alignment/size requirements are met (GTC25-S72683 slide interval_0177).
-- Use `cooperative_groups::invoke_one` to eliminate the compiler's peeling loop when issuing TMA from within an `if (threadIdx.x == 0)` block.
-
-## Measured Characteristics
-
-- async-copy-2stage bandwidth probe: On H200 (sm_90a, CUDA 12.9), a **vanilla elementwise kernel** (`c[i] = a[i] * b[i]`, N=256M floats) achieved **3654 GB/s** (median 0.8815 ms). A **2-stage LDGSTS prefetch version** using `__pipeline_memcpy_async` achieved **3295 GB/s** (median 0.9775 ms), approximately **10% slower**. This confirms the GTC guidance: for trivially simple compute, the shared memory staging overhead outweighs the latency-hiding benefit. The async approach delivers significant uplift only with heavier compute intensity or iterative kernels with genuine prefetch opportunities.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/memory/bank-conflict/pitfalls.md

@@ -1,83 +0,0 @@
----
-title: Bank Conflict Avoidance -- Pitfalls
-parent_skill: bank-conflict
-id: pitfall-bank-conflict
-type: pitfall
-vendor: nvidia
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- shared-memory-optimization
-- swizzling
-confidence: inferred
-evidence_level: inferred
-tags:
-- shared-memory-optimization
-- swizzling
-- cuda-cpp
----
-## Pitfall 1: Invisible conflicts in column-wise 2D array access
-
-**Symptom**: a kernel using `__shared__ float tile[32][32]` shows unexpectedly low shared-memory throughput or high cycle counts in NCU, even though every load/store appears to use a simple index expression.
-
-**Root cause**: C++ arrays are row-major.  When consecutive threads (varying `threadIdx.x`) access `tile[threadIdx.x][fixed_col]`, the stride between adjacent threads' addresses is 32 elements = 128 bytes.  Since `(128 / 4) % 32 = 0`, all 32 threads hit the same bank, causing a 32-way conflict.  This is easy to miss because the indexing looks simple and correct.
-
-**Detection**: in NCU, check `l1tex__data_bank_conflicts_pipe_lsu_mem_shared_op_ld.sum` and `l1tex__data_bank_conflicts_pipe_lsu_mem_shared_op_st.sum`.  Non-zero values indicate bank conflicts.
-
-**Fix**: add +1 padding to the inner dimension: `__shared__ float tile[32][33];`.  This changes the stride to 33, and `33 % 32 = 1`, eliminating the conflict entirely.
-
-## Pitfall 2: Padding wastes shared memory at large tile sizes
-
-**Symptom**: after adding +1 padding to a large 2D shared array (e.g., `tile[128][129]` instead of `tile[128][128]`), the kernel fails to launch because the shared memory allocation exceeds the per-SM limit (228 KB on H200 with opt-in dynamic shared memory).
-
-**Root cause**: padding adds one element per row.  For an array of R rows and C columns of `float`, padding adds `R * 4` bytes.  For large R this can push the total beyond the shared memory budget, especially when multiple such arrays are declared.
-
-**Fix**: consider index swizzling (`CU_TENSOR_MAP_SWIZZLE_128B` or manual XOR-based swizzle) instead of padding.  Swizzling remaps addresses without using extra memory.  Alternatively, reduce tile size to stay within budget.
-
-## Pitfall 3: Confusing broadcast with conflict
-
-**Symptom**: a developer adds padding or swizzle to an access pattern where all threads read the **same** shared-memory address, expecting a speedup. No improvement is observed.
-
-**Root cause**: when all threads in a warp read the same address in the same bank, the hardware broadcasts the value to all threads in a single transaction.  This is NOT a bank conflict.  Conflicts only occur when multiple threads access **different addresses** in the **same bank**.
-
-**Detection**: the NCU bank-conflict counters will already show zero for broadcast patterns.  If conflicts are zero, do not apply padding -- the problem is elsewhere.
-
-**Fix**: profile before optimizing.  Only apply bank-conflict fixes when the bank-conflict counters show non-zero values.
-
-## Pitfall 4: 64-bit types double the conflict factor
-
-**Symptom**: a kernel accessing `__shared__ double arr[32]` with stride-1 (`arr[threadIdx.x]`) shows 2-way bank conflicts even though the pattern looks conflict-free.
-
-**Root cause**: each `double` occupies 8 bytes = 2 consecutive 32-bit words.  Thread 0 reads banks 0-1, thread 1 reads banks 2-3, ... thread 15 reads banks 30-31, thread 16 wraps around to banks 0-1.  So threads 0 and 16 conflict on bank 0 (and bank 1), creating a 2-way conflict.
-
-**Detection**: NCU bank-conflict counters show non-zero values.
-
-**Fix**: for 64-bit types, the padding trick requires adding 1 element (8 bytes) per row, which shifts each row by 2 banks.  Alternatively, redesign the access pattern to avoid stride-16 conflicts (e.g., use a different tile layout).
-
-## Pitfall 5: Padding trick does not help stride-2 conflicts much
-
-**Symptom**: a kernel with stride-2 shared-memory access shows 2-way bank conflicts; developer applies +1 padding but sees little or no improvement.
-
-**Root cause**: on H200, 2-way bank conflicts have negligible overhead (measured 1.03x vs conflict-free in our probe).  The padding adds complexity and wastes memory for no practical benefit.
-
-**Detection**: measure before and after.  If the before measurement shows <1.1x overhead from conflicts, padding is not worth the added complexity.
-
-**Fix**: only optimize bank conflicts that are high-degree (16-way or 32-way) where the measured penalty is significant.  Low-degree conflicts on H200 are effectively free.
-
-## Pitfall 6: Dynamic shared memory base address alignment
-
-**Symptom**: a kernel using `extern __shared__` dynamic shared memory with manually computed offsets introduces bank conflicts that were not present with statically declared shared arrays.
-
-**Root cause**: when multiple arrays are packed into a single `extern __shared__` region, incorrect offset alignment can cause the second array's base address to start at a bank offset that produces conflicts.  For example, if array A uses 128 bytes (banks 0-31) and array B starts immediately after at byte 128, B's first element is at bank 0 -- fine.  But if A uses 130 bytes and B starts at byte 130, B's first element is at bank 0 but offset by 2 bytes, causing misalignment and potential conflicts.
-
-**Fix**: align each sub-array's base address to 128 bytes (32 banks x 4 bytes) to ensure the bank mapping is predictable.  Use the standard pattern:
-
-```cuda
-extern __shared__ char smem_raw[];
-float* arrA = (float*)smem_raw;
-// Align next array to 128-byte boundary
-float* arrB = (float*)(smem_raw + ((sizeA + 127) & ~127));
-```