@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/operator-routing/reduction/INDEX.md

@@ -1,126 +0,0 @@
----
-title: Reduction Pattern -- Decision Tree
-pattern_class: cuda-core
-op: reduction
-status: draft
-hardware:
-  device: H200
-  sm: 9.0a
-source:
-- path: spec
-  anchor: Reference
-id: routing-reduction-INDEX
-type: operator-routing
-vendor: nvidia
-operator: reduction
-source_refs:
-- source_id: source-code/cuda-samples
-  path: Samples/2_Concepts_and_Techniques/reduction/reduction_kernel.cu
-  anchor: L75-L91
-- source_id: source-code/cuda-samples
-  path: Samples/2_Concepts_and_Techniques/reduction/reduction.cpp
-  anchor: L28-L62
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA API References/cub/cub.md
-  anchor: L26607-L26630
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- pipeline-stages
-- kernel-fusion
-- shared-memory-optimization
-kernel_types:
-- fused-kernel
-confidence: inferred
-evidence_level: inferred
-tags:
-- pipeline-stages
-- kernel-fusion
-- shared-memory-optimization
-- fused-kernel
-- cuda-cpp
----
-# Reduction Pattern -- Decision Tree
-
-This document guides the kernel-writing agent through a reduction task from a custom-kernel requirement to a working, optimized kernel.
-
-## Scope
-
-This decision tree covers custom-kernel implementation choices only. It starts after the task has been classified as requiring a dedicated kernel implementation.
-
-## Step 1 -- Choose the custom reduction strategy
-
-```
-Q4. How many elements per reduction instance?
-    <=32 (single warp)
-        --> Warp-level butterfly reduction using __shfl_xor_sync.
-            Each warp independently reduces its 32 (or fewer) elements
-            in log2(32) = 5 shuffle steps. No shared memory needed.
-            See wiki/nvidia/foundations/compute/warp-primitives/skill.md.
-
-    33..1024 (single block)
-        --> Block-level reduction:
-            Phase 1: each warp reduces its 32 elements via __shfl_xor_sync.
-            Phase 2: warp leaders write partial sums to shared memory.
-            Phase 3: a single warp reduces the partial sums (<=32 values).
-            Requires shared memory (see bank-conflict avoidance below).
-
-    >1024 (multi-block / grid-level)
-        --> Grid-level two-pass reduction:
-            Pass 1: launch many blocks, each reduces a tile of the input
-            to a single partial sum (using block-level reduction above).
-            Write partial sums to a global buffer.
-            Pass 2: reduce the partial sums (a second kernel launch, or
-            ONE atomicAdd per block if combining into a scalar).
-            Alternative: threadFenceReduction pattern (single-kernel with
-            __threadfence() and atomicInc to detect the last block).
-            See cuda-samples/threadFenceReduction/ for the canonical example.
-
-            CRITICAL: the atomic tail must issue ONE atomic per block,
-            NOT one per thread. A naive "every-thread atomicAdd to one
-            scalar" pattern is both 248x slower AND numerically wrong
-            for FP32 sums with N > ~1e5 (the running sum saturates the
-            24-bit mantissa). Use skill wiki/nvidia/foundations/sync/atomic-reduction/
-            (S1 hierarchical) for the grid-level combining step. On H200,
-            S1 + grid-stride loop achieves 1498x speedup over naive and
-            75% of HBM peak bandwidth for sum-reduction.
-```
-
-## Step 2 -- Optimization via ROUTING.md skills
-
-After the basic custom kernel is working and correct, apply optimization skills from ROUTING.md in priority order:
-
-1. **Coalescing** (wiki/nvidia/foundations/memory/coalescing/) -- ensure the input load phase uses coalesced (stride-1) global memory access. This is the single most impactful optimization for memory-bound reduction kernels.
-
-2. **Warp primitives** (wiki/nvidia/foundations/compute/warp-primitives/) -- replace shared-memory tree reduction within a warp with register-based `__shfl_xor_sync` / `__shfl_down_sync` butterfly reduction. Eliminates shared-memory round-trips for the final 32-element reduce stage.
-
-3. **Bank-conflict avoidance** (wiki/nvidia/foundations/memory/bank-conflict/) -- if the kernel uses shared memory for the block-level reduction tree, ensure the layout avoids bank conflicts. For a standard tree reduction, sequential addressing (reduce2 pattern from cuda-samples) is conflict-free; but interleaved addressing (reduce1 pattern) causes conflicts.
-
-4. **Atomic reduction contention control** (wiki/nvidia/foundations/sync/atomic-reduction/) -- for the grid-level combining step in multi-block reductions, the atomic tail must issue **one atomic per block**, not one per thread. The S1 hierarchical pattern (warp shuffle -> shmem -> block-level atomicAdd) is required when the kernel's final stage is a global atomicAdd. Pair with skill 2 (warp-primitives, for the inner shuffle) and skill 9 (memory-ordering, for scope / ordering correctness). Measured on H200: 247.8x speedup over the naive per-thread atomic pattern; 1498x with grid-stride loop on top of S1.
-
-After each skill application, re-benchmark against the task-provided baseline and follow the bottleneck-triage procedure in reasoning/bottleneck-triage.md.
-
-## Step 3 -- Advanced techniques (Brent's theorem)
-
-The cuda-samples reduce6 kernel demonstrates **Brent's theorem optimization**: each thread serially accumulates multiple elements from global memory before participating in the parallel reduction tree.  This increases arithmetic intensity and reduces the total number of threads (and blocks) needed, which can improve occupancy and reduce launch overhead for very large arrays (>1M elements).
-
-```cuda
-// Each thread accumulates multiple elements (grid-stride loop)
-float mySum = 0;
-unsigned int i = blockIdx.x * blockSize + threadIdx.x;
-unsigned int gridSize = blockSize * gridDim.x;
-while (i < n) {
-    mySum += g_idata[i];
-    i += gridSize;
-}
-// Then proceed with block-level reduction of mySum
-```
-
-## Cross-references
-
-- **Skill whitelist for this pattern**: `ROUTING.md`
-- **Task packet template**: `TASK-PACKET.md`
-- **Bottleneck triage after benchmarking**: `reasoning/bottleneck-triage.md`

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/operator-routing/reduction/ROUTING.md

@@ -1,186 +0,0 @@
----
-title: Reduction Pattern -- Skill Routing
-pattern_class: cuda-core
-op: reduction
-status: draft
-id: routing-reduction-ROUTING
-type: operator-routing
-vendor: nvidia
-operator: reduction
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-hardware_features:
-- mbarrier
-techniques:
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- register-budgeting
-- data-reuse
-- kernel-fusion
-- loop-unrolling
-- shared-memory-optimization
-- software-exp
-- communication-overlap
-kernel_types:
-- fused-kernel
-- attention
-- quantization
-confidence: inferred
-evidence_level: inferred
-tags:
-- mbarrier
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- register-budgeting
-- data-reuse
-- kernel-fusion
-- loop-unrolling
-- shared-memory-optimization
-- software-exp
-- communication-overlap
-- fused-kernel
-- attention
-- quantization
-- cuda-cpp
----
-# Reduction Pattern -- Skill Whitelist
-
-This file lists the optimization skills applicable to a custom reduction kernel, in recommended application order. Only skills that currently exist under `wiki/nvidia/foundations/` (with a completed `skill.md`) are listed.
-
-Skills that are conceptually relevant but not yet built (e.g., vectorized-access, ilp, register-pressure) are NOT included. Do not reference them until their `skill.md` files exist.
-
-## Applicable skills
-
-### 1. Global Memory Coalescing
-
-- **Skill path**: `wiki/nvidia/foundations/memory/coalescing/`
-- **Why it matters for reduction**: the input-load phase of any reduction kernel reads the full input array from global memory. Coalesced (stride-1) loads are essential for achieving peak memory bandwidth. On H200, non-coalesced (stride-32) access is 2.85x slower than coalesced access (measured).
-- **When to apply**: always. Every reduction kernel's load phase should be verified for coalescing. The standard pattern `tid = blockIdx.x * blockDim.x + threadIdx.x; val = input[tid]` is coalesced. Watch for multi-dimensional reductions where the reduction axis may not be the contiguous dimension.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Reads/writes memory with stride > 1 per thread."
-
-### 2. Warp Primitives (Shuffle Reduction)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/warp-primitives/`
-- **Why it matters for reduction**: warp-level butterfly reduction via `__shfl_xor_sync` (or `__shfl_down_sync`) replaces shared-memory tree reduction for the final 32-element stage. On H200, a full 5-step butterfly reduction takes ~145 cycles (measured), eliminating shared-memory round-trips entirely within a warp.
-- **When to apply**: always for the intra-warp reduction stage. The canonical block-level reduction pattern (from cuda-samples reduce4+) uses shared memory for inter-warp communication but warp shuffles for the intra-warp stage.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Thread 0 (or lane 0) does a serial reduction."
-
-### 3. Shared Memory Cache (prerequisite for the block-level tree)
-
-- **Skill path**: `wiki/nvidia/foundations/memory/shared-memory-cache/`
-- **Why it matters for reduction**: the block-level reduction phase (§1 Q4 "single block" branch in `INDEX.md`) uses shared memory as the inter-warp staging area — warp leaders write their partial sums to `smem[warpId]`, then a final warp reduces those 32 partials via shuffles. For grid-level reductions, each block also optionally uses a shared-memory tile to cache strides of the input before reducing. The sibling skill's sub-skill S1 (tile reuse) + S2 (coalesce-then-reorder) both apply.
-- **When to apply**: always, for reductions over > 32 elements (i.e., anything that leaves a single warp). For pure warp-level reductions (`<= 32` elements), smem is not needed — the final shuffle suffices.
-- **Specific guidance for reduction**:
-  - The classic `__shared__ float smem[32];` for warp-leader partials is bank-conflict-free on stride-1 lane access; no padding required.
-  - When caching a multi-element-per-thread tile before reducing (Brent's theorem pattern), declare `__shared__ float tile[TILE][TILE+1]` if any column access is planned — measured on H200: unpadded `[32][32]` → 16.3 M bank conflicts, padded → 33 K (488× drop; see `sources/experience/hw-probes/smem-tile-reuse/`).
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Block-level reduction without smem staging / with redundant global loads."
-
-### 4. Barrier Optimization (scope narrowing + async-barrier overlap)
-
-- **Skill path**: `wiki/nvidia/foundations/sync/barrier-optimization/`
-- **Why it matters for reduction**: the block-level reduction tree uses `__syncthreads()` between warp-leader writes to smem and the final warp's read of those partials. On H200 sm_90a, each `__syncthreads()` costs ~31–62 ns depending on block size (measured: `sources/experience/hw-probes/barrier-cost/`). For reductions that loop over many tiles with a `__syncthreads()` per tile, this cost compounds.
-- **When to apply**:
-  - **S1 narrow scope**: if the reduction is warp-local (N ≤ 32 per reduction instance), drop `__syncthreads()` entirely — the `_sync` shuffle intrinsics embed their own warp sync. Saves ~2× the per-barrier cost per iteration.
-  - **S1 narrow scope**: if the reduction is block-local but only a subset of warps need to agree (e.g. after warp 0 has written the final smem slot, other warps don't need to wait), use a named `bar.sync` with subset count or `cuda::barrier` with `expected_count < blockDim.x`.
-  - **S2 arrive/wait split (async barrier)**: only beneficial if the post-barrier consumer warp has independent work to do BEFORE needing the barrier result. For a standard block-level reduction the consumer depends immediately on the barrier, so S2 does NOT help here — pitfall P6 shows bare mbarrier cost is 1.6–2.2× a `__syncthreads`.
-- **When NOT to apply**: grid-stride reduction kernels with just one `__syncthreads()` per block are already barrier-light — the next optimization target is elsewhere (NCU `stall_barrier` < 5 %).
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — "Reduction kernel with `stall_barrier` above ~15 %."
-
-### 5. Bank-Conflict Avoidance
-
-- **Skill path**: `wiki/nvidia/foundations/memory/bank-conflict/`
-- **Why it matters for reduction**: block-level reduction kernels that use shared memory (skill 3 above) for the tree-reduction phase are susceptible to bank conflicts. Interleaved addressing (cuda-samples reduce1) creates conflicts; sequential addressing (reduce2) avoids them. When warp leaders write partial sums to `smem[warpId]`, 32-way conflicts can occur if all warps' lane-0 threads hit bank 0.
-- **When to apply**: whenever the kernel uses `__shared__` memory for inter-warp communication or the reduction tree. Check the access pattern: if consecutive active threads access addresses with a stride that is a multiple of 32, apply the +1 padding trick or redesign the index mapping.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Narrow reduction into smem with heavy bank contention."
-
-### 6. Vectorized Access
-
-- **Skill path**: `wiki/nvidia/foundations/memory/vectorized-access/`
-- **Why**: loading float4 in the input phase increases bytes-in-flight per thread. On H200, float4 vs scalar shows 2.59× bandwidth gain.
-- **When**: input array is 16-byte aligned and N is divisible by 4.
-
-### 7. Warp Divergence (reduction-tail `if (tid < offset)` shapes)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/warp-divergence/`
-- **Why it matters for reduction**: the classical tree-reduction tail has a stride halving loop with `if (tid < offset)` guards. Unguarded, the active lane count halves each iteration — and the inactive lanes still occupy the warp. Fortunately nvcc typically predicates the body (short: one load + one add + one store to smem), so the divergence is a non-issue for stride-halving tails. It becomes an issue only when the per-lane body grows (e.g. reduction with fused fmad + reciprocal, or reduction over struct fields) and falls outside nvcc's predication threshold.
-- **When to apply**:
-  - Keep the reduction-tail body short and arithmetic-only so nvcc predicates it; let the last few iterations converge within a single warp using shuffles instead of smem (skill #2 warp-primitives already replaces the tail with branch-free shuffles).
-  - For reductions with a fused compute (e.g. softmax's exp + sum-reduction + divide-by-sum), the reduction kernel is typically compute-bound at the compute step and memory-bound at the tail — measured divergence cost on the tail is usually invisible. Verify before rewriting.
-- **When NOT to apply**: pure sum-reduction kernels using shuffle- based warp reduction (the canonical H200 pattern from `atomic-reduction` skill). These have no divergent tail — the warp-reduction folds 32 elements into lane 0 with five shuffles and zero divergence.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — "Reduction-tail body too large for predication."
-
-### 8. Instruction-Level Parallelism (ILP)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/ilp/`
-- **Why**: multiple independent accumulator chains hide FMA pipeline latency (4 cycles on H200). 4-acc achieves 3.97× throughput vs 1-acc.
-- **When**: the reduction accumulation loop processes one element at a time; unrolling with multiple accumulators helps.
-
-### 9. Register Pressure Management
-
-- **Skill path**: `wiki/nvidia/foundations/memory/register-pressure/`
-- **Why**: vectorized access + ILP increase register usage, potentially reducing occupancy. On H200, forcing 32 regs caused 4.84× slowdown from spilling.
-- **When**: after applying skills 4-5, check regs/thread via `-Xptxas=-v`.
-
-### 10. Compiler Hints
-
-- **Skill path**: `wiki/nvidia/foundations/compute/compiler-hints/`
-- **Why**: `__launch_bounds__` controls register allocation ceiling; `#pragma unroll` exposes ILP.
-- **When**: tuning the final kernel after functional correctness is established.
-
-### 11. Async Copy (LDGSTS Prefetching)
-
-- **Skill path**: `wiki/nvidia/foundations/memory/async-copy/`
-- **Why**: for iterative / compute-heavy reduction kernels, LDGSTS async copies to shared memory increase bytes-in-flight without consuming registers. Little's Law: H200 needs ~64 KiB/SM for >90% BW.
-- **When**: reduction kernel is iterative (processes multiple tiles per block) AND compute per element is non-trivial (e.g., involves sqrt/exp). Simple sum/max over a flat array does NOT benefit (GTC25-S72683 showed 10% regression for trivial `a*b`).
-- **Caveat**: evaluate carefully — async copies add code complexity and only help when "Stall Long Scoreboard" is the dominant bottleneck in ncu.
-
-### 12. Memory Ordering (correctness)
-
-- **Skill path**: `wiki/nvidia/foundations/sync/memory-ordering/`
-- **Why**: grid-level reductions with `atomicAdd` or `__threadfence` patterns require correct memory ordering to avoid data races.
-- **When**: multi-block reduction that communicates via global memory atomics or flags.
-
-### 13. Atomic Reduction Contention Control (critical at grid level)
-
-- **Skill path**: `wiki/nvidia/foundations/sync/atomic-reduction/`
-- **Why it matters for reduction**: once the block-level reduction produces a partial sum, the grid-level combining step uses global atomics. A naive kernel that has **every thread** atomicAdd into a single global scalar is both contention-pathological and numerically wrong (on H200, 33M-element FP32 reduction measured 248× slower AND `rel_err = 5.03e-02` vs double reference -- the running sum saturates the FP32 mantissa, see skill's pitfall P7). The S1 hierarchical pattern commits **one atomic per block** after in-warp shuffle + shared-memory fan-in; S1 + grid-stride loop (fewer, fatter blocks) lifts this to 1498× and 75% DRAM SoL on H200.
-- **When to apply**: every multi-block (`>1024` elements per instance) reduction that terminates with a global atomic. Required for the `>1024` branch of Q4 in `INDEX.md`. Pair with skill #9 (memory-ordering) for scope / ordering correctness, and with skill #2 (warp-primitives) since S1's inner loop uses `__shfl_down_sync`.
-- **Measured impact**: 247.8× speedup of S1 over naive; 1498× for S1+grid-stride. See `sources/experience/hw-probes/atomic-reduction-contention.md`.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "many threads contending on the same atomic address" / high `stall_long_scoreboard` on `RED`/`ATOM`.
-
-### 14. L2 Access Policy (hitRatio-tuned persisting window)
-
-- **Skill path**: `wiki/nvidia/foundations/memory/l2-access-policy/`
-- **Why it matters for reduction**: multi-stage reduction pipelines that re-read a hot buffer across kernels (e.g. grid-level combine that touches the per-block partials buffer twice, or a softmax sum-then-divide shape that re-reads the input between the two passes) can pin that buffer into H200's 37.5 MiB L2 set-aside via `cudaStreamAttributeAccessPolicyWindow`. Measured on H200 at WS = 80 MiB / set_aside = 37.5 MiB: `hitRatio = set_aside / WS` recovers **+17.7%** effective BW; `hitRatio = 1.0` silently degenerates to no-op.
-- **When to apply**:
-  - The hot buffer is repeatedly read **across more than one kernel launch** on the same stream. A single kernel's grid-stride pass does not trigger eviction pressure worth pinning against (measured: WS = 40 MiB single-kernel soft-null).
-  - WS approaches or exceeds the H200 set-aside cap of 37.5 MiB. Below that, normal LRU keeps the buffer hot for free.
-  - Set the `hitRatio` formula: `hitRatio = min(1.0f, (float)set_aside / (float)ws_bytes)`. Do **not** use `hitRatio = 1.0` when WS > set_aside — measured as a silent null on H200.
-- **When NOT to apply**:
-  - Single-kernel MVPs with no competing workload on the L2 (skill's P8 directly applies to reduction microbenchmarks).
-  - MIG partitions (set-aside reservation is a silent no-op; skill P3).
-  - Buffers already below set-aside with no concurrent pressure — policy is overhead without benefit (measured neutral at WS = 4 / 40 MiB).
-- **Measured impact**: +17.7% effective BW at WS = 80 MiB on H200 when `hitRatio` is tuned. See `sources/experience/hw-probes/l2-residency.md`.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — multi-pass reductions where the second kernel's DRAM throughput stays at 100% Roofline even though the same buffer was just fully read by the first kernel (no L2 reuse).
-
-### 15. Branch Elimination (max/min reductions; tail already predicated)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/branch-elimination/`
-- **Why it matters for reduction**:
-  - **Max-reduction / min-reduction / argmax / argmin** kernels have a comparator at every reduction step. Using `fmaxf(a, b)` / `fminf(a, b)` in place of `(a > b) ? a : b` saves **one SASS op per comparison** on H200 (FMNMX vs FSETP+FSEL). Over a log2(N)-depth tree this compounds.
-  - **Tree-reduction tail** (`for (offset = blockDim/2; offset > 0; offset >>= 1) if (tid < offset) smem[tid] += smem[tid + offset];`) contains an `if (tid < offset)` guard on each step. nvcc 12.9 already predicates this simple-body guard to `FSEL`; `cuobjdump --dump-sass` confirms no inner-loop `BRA`. No rewrite needed.
-  - **Clamped-sum reductions** (e.g. `sum(max(x - threshold, 0))` as in quantile estimators or hinge loss): use `fmaxf` for the clamp — 1 SASS op vs 2 for the equivalent `if/else`.
-- **When to apply**:
-  - Max/min-reduction: prefer `fmaxf`/`fminf` over explicit ternary. Measured saving 1.56× per comparison on H200.
-  - Argmax: use `fmaxf` for the value comparison but still need an `if`/`selp` to carry the index — compiler predicates the index select automatically. Do NOT rewrite as arithmetic.
-  - Sum-of-clamped: `sum += fmaxf(x - t, 0.0f)` — one FMNMX per element vs two-op guarded add.
-- **When NOT to apply**:
-  - Do NOT rewrite `if (tid < offset) smem[tid] += smem[tid + offset]` as arithmetic — it is already predicated by nvcc, and the arithmetic form would add FADD/FMUL ops (measured 1.27× slower in the branchless-patterns probe cond-arith variant).
-  - Do NOT hand-code bit tricks for argmax sign handling — `copysignf` or `fabsf` + a predicated index carry is cleaner.
-- **Measured on H200**: 1.56× per comparison from switching `ternary` to `fmaxf` in a max-reduction tree (branchless-patterns probe Group A).
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — max/min reduction kernel where hot path shows `FSETP + FSEL` pairs instead of `FMNMX`. Correct response: replace ternary with `fmaxf`/`fminf`.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/operator-routing/scan-cumulative/INDEX.md

@@ -1,145 +0,0 @@
----
-title: Scan (Cumulative) Pattern -- Decision Tree
-pattern_class: cuda-core
-op: scan-cumulative
-status: draft
-hardware:
-  device: H200
-  sm: 9.0a
-source:
-- path: spec
-  anchor: Reference
-id: routing-scan-cumulative-INDEX
-type: operator-routing
-vendor: nvidia
-operator: scan-cumulative
-source_refs:
-- source_id: source-code/cuda-samples
-  path: Samples/2_Concepts_and_Techniques/shfl_scan/shfl_scan.cu
-  anchor: L55-L131
-- source_id: source-code/cuda-samples
-  path: Samples/2_Concepts_and_Techniques/scan/scan.cu
-  anchor: L43-L62
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA API References/cub/cub.md
-  anchor: L28474-L28508
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- pipeline-stages
-- vectorized-loads
-- kernel-fusion
-- shared-memory-optimization
-- software-exp
-kernel_types:
-- fused-kernel
-- attention
-confidence: inferred
-evidence_level: inferred
-tags:
-- pipeline-stages
-- vectorized-loads
-- kernel-fusion
-- shared-memory-optimization
-- software-exp
-- fused-kernel
-- attention
-- cuda-cpp
----
-# Scan (Cumulative) Pattern -- Decision Tree
-
-This document guides the kernel-writing agent through a scan (prefix sum / cumulative sum) task from a custom-kernel requirement to a working, optimized kernel.
-
-Scan operators covered: inclusive scan, exclusive scan, cumulative sum (cumsum), prefix sum, prefix max/min, and segmented variants.
-
-Key distinction from reduction: a scan produces an output element for every input element (output length == input length), whereas a reduction collapses the input to a single value (or one value per reduction axis).
-
-## Scope
-
-This decision tree covers custom-kernel implementation choices only. It starts after the task has been classified as requiring a dedicated kernel implementation.
-
-## Step 1 -- Choose the custom scan strategy
-
-```
-Q4. How many elements per scan instance?
-    <=32 (single warp)
-        --> Warp-level Hillis-Steele scan using __shfl_up_sync.
-            Each warp independently computes an inclusive prefix sum
-            in log2(32) = 5 shuffle steps. No shared memory needed.
-            See wiki/nvidia/foundations/compute/warp-primitives/skill.md.
-
-            For exclusive scan: shift the inclusive result right by 1
-            lane and insert the identity element at lane 0.
-
-    33..1024 (single block)
-        --> Block-level scan:
-            Phase 1: each warp computes an inclusive warp scan via
-            __shfl_up_sync.
-            Phase 2: the last lane of each warp writes its total to
-            shared memory.
-            Phase 3: warp 0 scans the warp totals (<=32 values) via
-            __shfl_up_sync.
-            Phase 4: every thread (except those in warp 0) adds the
-            scanned total of the preceding warp to its own value.
-            Requires shared memory (see bank-conflict avoidance).
-            See cuda-samples/shfl_scan/shfl_scan.cu for the canonical
-            example.
-
-    >1024 (multi-block / grid-level)
-        --> Grid-level multi-pass scan:
-            Option A -- Three-kernel approach (Blelloch-style):
-              Pass 1: each block scans its tile, writes block total.
-              Pass 2: scan the block totals (a second kernel launch).
-              Pass 3: uniform_add -- each block adds its prefix to
-              all of its elements.
-              Work-efficient (O(n)), but requires 3 kernel launches.
-              See cuda-samples/shfl_scan/ for a 2-kernel variant
-              (the third pass is uniform_add).
-
-            Option B -- Single-pass decoupled lookback:
-              Each block scans its tile locally, then propagates its
-              partial aggregate to subsequent blocks via global memory
-              using atomics and a status flag (X/P/A protocol).
-              Single kernel launch, O(2n) data movement.
-              This is the algorithm used by CUB internally.
-              Reference: Merrill & Garland, "Single-pass Parallel
-              Prefix Scan with Decoupled Look-back", NVR-2016-002.
-              Custom implementation is complex; prefer CUB unless
-              fusion requires it.
-```
-
-## Step 2 -- Optimization via ROUTING.md skills
-
-After the basic custom kernel is working and correct, apply optimization skills from ROUTING.md in priority order:
-
-1. **Warp primitives** (wiki/nvidia/foundations/compute/warp-primitives/) -- replace shared-memory tree scan within a warp with register-based `__shfl_up_sync` Hillis-Steele scan. This is the single most impactful optimization for scan kernels since the intra-warp scan is on the critical path of every element.
-
-2. **Coalescing** (wiki/nvidia/foundations/memory/coalescing/) -- ensure the input load and output store phases use coalesced (stride-1) global memory access. Scan kernels read and write every element, so both load and store phases matter (unlike reduction, which only loads).
-
-3. **Bank-conflict avoidance** (wiki/nvidia/foundations/memory/bank-conflict/) -- if the kernel uses shared memory for inter-warp communication of warp totals, ensure the access pattern does not cause bank conflicts. The standard pattern of writing to `smem[warp_id]` is conflict-free (one thread per bank) but the subsequent read by warp 0 of all entries must also be checked.
-
-After each skill application, re-benchmark against the task-provided baseline and follow the bottleneck-triage procedure in reasoning/bottleneck-triage.md.
-
-## Step 3 -- Advanced techniques
-
-### Vectorized loads for scan input
-
-For large scans where each thread processes multiple elements sequentially before participating in the parallel scan tree, use `float4` / `int4` vectorized loads to increase memory throughput. Each thread serially scans its 4 (or more) elements, then the intra-warp and inter-warp parallel scan operates on those partial sums. After the parallel phase, each thread applies the prefix offset back to its individual elements.
-
-### Decoupled lookback for single-pass
-
-If single-pass custom-kernel performance is needed, implement the decoupled lookback protocol:
-1. Each block atomically publishes its status: `X` (not started), `P` (partial aggregate available), `A` (full inclusive prefix available).
-2. After computing its local scan, a block lookbacks through preceding blocks, accumulating their aggregates until it finds a block with status `A`.
-3. The block then updates its own status to `A` and writes its inclusive prefix for subsequent blocks.
-
-This is non-trivial to implement correctly. Memory ordering (`__threadfence()`, `cuda::atomic`) is critical. See wiki/nvidia/foundations/sync/memory-ordering/ for the correctness constraints.
-
-## Cross-references
-
-- **Skill whitelist for this pattern**: `ROUTING.md`
-- **Task packet template**: `TASK-PACKET.md`
-- **Bottleneck triage after benchmarking**: `reasoning/bottleneck-triage.md`

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/operator-routing/scan-cumulative/ROUTING.md

@@ -1,130 +0,0 @@
----
-title: Scan (Cumulative) Pattern -- Skill Routing
-pattern_class: cuda-core
-op: scan-cumulative
-status: draft
-id: routing-scan-cumulative-ROUTING
-type: operator-routing
-vendor: nvidia
-operator: scan-cumulative
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- pipeline-stages
-- double-buffering
-- vectorized-loads
-- cache-policy
-- kernel-fusion
-- loop-unrolling
-- shared-memory-optimization
-- communication-overlap
-kernel_types:
-- fused-kernel
-- attention
-- quantization
-confidence: inferred
-evidence_level: inferred
-tags:
-- pipeline-stages
-- double-buffering
-- vectorized-loads
-- cache-policy
-- kernel-fusion
-- loop-unrolling
-- shared-memory-optimization
-- communication-overlap
-- fused-kernel
-- attention
-- quantization
-- cuda-cpp
----
-# Scan (Cumulative) Pattern -- Skill Whitelist
-
-This file lists the optimization skills applicable to a custom scan (prefix sum / cumulative) kernel, in recommended application order. Only skills that currently exist under `wiki/nvidia/foundations/` (with a completed `skill.md`) are listed.
-
-## Applicable skills
-
-### 1. Warp Primitives (Shuffle Scan)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/warp-primitives/`
-- **Why it matters for scan**: the intra-warp inclusive scan via `__shfl_up_sync` is the fundamental building block of every custom scan kernel. Unlike reduction (which uses `__shfl_xor_sync` or `__shfl_down_sync` for butterfly patterns), scan uses `__shfl_up_sync` in a Hillis-Steele pattern: for offsets 1, 2, 4, 8, 16, each thread fetches the value from `offset` lanes earlier and adds it to its own accumulator. This completes a warp-wide inclusive scan in 5 steps without touching shared memory.
-- **When to apply**: always. Every custom scan kernel's intra-warp phase should use `__shfl_up_sync`. Falling back to shared-memory scan within a warp wastes bandwidth and adds `__syncwarp` overhead.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Thread 0 (or lane 0) does a serial reduction."
-
-### 2. Shared Memory Cache (prerequisite for block-level scan)
-
-- **Skill path**: `wiki/nvidia/foundations/memory/shared-memory-cache/`
-- **Why it matters for scan**: scan is fundamentally stateful — each output position depends on all preceding inputs — so a block-level scan must stage its working set in shared memory, then iteratively propagate partial results. Both the Hillis-Steele in-place variant (cuda-samples `scan1Inclusive`) and the Blelloch upsweep/downsweep variant rely on a single `__shared__` buffer as the scan workspace. The sibling skill's sub-skill S1 (smem as user-managed cache) is the exact mechanism; S3 (dynamic smem) is how the buffer is sized when block dim is chosen at launch.
-- **When to apply**: always for block-level scans over > 32 elements. For warp-level scans (`__shfl_up_sync` ladder), smem is not needed. For grid-level scans, each block's smem holds its tile; the cross-block combine uses memory-ordering primitives (skill 6 below), not smem.
-- **Specific guidance for scan**:
-  - Hillis-Steele requires double-buffered smem when done in-place (`smem[2][N]`, swap the read/write index each pass) to avoid the read-after-write race.
-  - Blelloch's upsweep/downsweep typically uses a single `smem[N]` buffer with power-of-two strides; those strides are the exact patterns that trigger bank conflicts (see skill 3 below).
-  - For block-sizes that force > 48 KB of smem (large tile per block), use the sibling skill's S3 opt-in pattern (`cudaFuncAttributeMaxDynamicSharedMemorySize`).
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Multi-pass block-level algorithm without smem working set."
-
-### 3. Barrier Optimization (scan is barrier-heavy)
-
-- **Skill path**: `wiki/nvidia/foundations/sync/barrier-optimization/`
-- **Why it matters for scan**: block-level scan algorithms (Hillis- Steele, Blelloch) are inherently barrier-heavy — each iteration of the scan tree must `__syncthreads()` between read and write of the smem buffer. A 1024-element block scan with Blelloch requires `2 × log2(1024) = 20` `__syncthreads()` calls per phase (upsweep
-  + downsweep), plus one at the end for final write-back. On H200
-  sm_90a at block size 1024, each `__syncthreads()` costs ~62 ns (measured: `sources/experience/hw-probes/barrier-cost/`), so barrier cost alone is ~1.2 μs per block — often a noticeable fraction of total kernel time for scans.
-- **When to apply**:
-  - **S1 narrow scope**: when the scan fits in a single warp (N ≤ 32 lanes), use `__shfl_up_sync` / `__shfl_down_sync` ladder with zero `__syncthreads()` calls. Saves all 20+ block barriers at the cost of only handling one warp's worth of scan; block- level scans then become "warp scan per warp, inter-warp aggregation via one `__syncthreads()`, then broadcast".
-  - **Decoupled-lookback single-pass scan** (50-classical-algo, pending bucket G): replaces multiple kernel launches + barriers with one kernel that streams tiles, cutting most barrier traffic. Defer until the classical-algo layer is bootstrapped.
-- **When NOT to apply**: if the scan already uses warp-shuffle primitives throughout and the barriers are only the unavoidable minimum, the barrier path is not the bottleneck — look at memory coalescing (skill #5) or ILP (skill #6) instead.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — "Scan kernel with `stall_barrier` > 20 %."
-
-### 4. Bank-Conflict Avoidance
-
-- **Skill path**: `wiki/nvidia/foundations/memory/bank-conflict/`
-- **Why it matters for scan**: block-level scan kernels use shared memory to communicate warp totals between warps. The classic Hillis-Steele shared-memory scan (cuda-samples scan1Inclusive) performs alternating read-write passes on shared memory with power-of-two strides, which causes bank conflicts at certain offsets. The +1 padding technique or switching to a warp-shuffle based approach avoids this.
-- **When to apply**: whenever the kernel uses `__shared__` memory for the inter-warp scan tree or for the primary scan data. Particular attention is needed when the shared memory array uses a stride that is a multiple of 32.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Narrow reduction into smem with heavy bank contention."
-
-### 5. Global Memory Coalescing
-
-- **Skill path**: `wiki/nvidia/foundations/memory/coalescing/`
-- **Why it matters for scan**: unlike reduction (read-only), scan kernels both read the full input and write the full output to global memory. Both load and store phases must be coalesced. The standard pattern `tid = blockIdx.x * blockDim.x + threadIdx.x; val = input[tid]; ... output[tid] = result;` is coalesced. Multi-dimensional scans along a non-contiguous axis require special attention.
-- **When to apply**: always. Verify both load and store are stride-1 access. For 2-D scans along the non-contiguous axis, consider transposing the data first.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Reads/writes memory with stride > 1 per thread."
-
-### 6. Instruction-Level Parallelism (ILP)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/ilp/`
-- **Why it matters for scan**: when each thread processes multiple input elements (the vectorized / multi-element-per-thread approach), the serial accumulation phase can benefit from multiple independent accumulator chains if the thread processes elements from non-adjacent segments. However, for a standard serial prefix sum within a thread's tile, ILP is limited because each step depends on the previous. ILP primarily helps the global-memory load phase when combined with vectorized access.
-- **When to apply**: after vectorized access is in place and the load/compute overlap can be improved. Less impactful for scan than for reduction because the serial dependency chain is inherent.
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Small tight loop inside kernel without #pragma unroll / ILP."
-
-### 7. Async Copy (LDGSTS Prefetching)
-
-- **Skill path**: `wiki/nvidia/foundations/memory/async-copy/`
-- **Why it matters for scan**: for iterative multi-tile scans where each block processes multiple tiles in sequence (e.g., a segmented scan with large segments), prefetching the next tile via `cuda::memcpy_async` while computing the current tile increases bytes-in-flight. Only beneficial when the scan involves non-trivial per-element compute (e.g., a scan of `sqrt(x) + exp(x)`). For simple sum-scan over a flat array, async copy typically adds overhead without benefit.
-- **When to apply**: when the kernel processes multiple tiles per block AND per-element computation is non-trivial. Evaluate carefully.
-- **Relevance to bottleneck triage**: Q4b in `reasoning/bottleneck-triage.md` -- bytes-in-flight analysis.
-
-### 8. Memory Ordering (correctness)
-
-- **Skill path**: `wiki/nvidia/foundations/sync/memory-ordering/`
-- **Why it matters for scan**: the decoupled-lookback single-pass scan algorithm requires correct global memory ordering between blocks. Each block publishes its partial/full aggregate via global memory and subsequent blocks read it. Without proper `__threadfence()` or `cuda::atomic` operations with appropriate memory order, data races cause silent corruption. This is a **correctness** skill, not a performance skill.
-- **When to apply**: whenever implementing a custom multi-block scan that communicates between blocks via global memory (e.g., decoupled lookback, or the three-kernel approach where block totals are exchanged).
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` -- "Data race or inconsistent cross-thread reads."
-
-### 9. Branch Elimination (Kogge-Stone lane guards + max-scan / min-scan comparators)
-
-- **Skill path**: `wiki/nvidia/foundations/compute/branch-elimination/`
-- **Why it matters for scan**:
-  - **Kogge-Stone / Hillis-Steele lane guards** (`if (lane_id >= offset) val = __shfl_up_sync(mask, val, offset) + val;`) are the canonical shuffle-scan pattern. The `if (lane_id >= offset)` guard is a simple-body predicate that compiler lowers to `FSEL` automatically — no inner-loop `BRA` is emitted. No rewrite needed.
-  - **Max-scan / min-scan variants** (common in quantile estimation, backward compatibility prefix-max): the inner comparator should be `fmaxf(prev, cur)` — **1 `FMNMX` op** vs 2 for the equivalent ternary. Same 1.56× saving as reduction max.
-  - **Decoupled-lookback status polling** (`if (status == FULL) break;`): the compiler **cannot** predicate this loop exit — it must emit a real `BRA`. Here the concern is not branch elimination but the lookback loop's warp vote and memory ordering (skill 8, memory-ordering).
-  - **Lane-ID masks for sub-warp scans** (cooperative_groups tiled_partition): the partition's `__shfl_up_sync` already embeds the lane-selection mask; no C-level `if` needed.
-- **When to apply**:
-  - Max-prefix / min-prefix scan: switch inner comparator from `(a > b) ? a : b` to `fmaxf(a, b)`.
-  - Any fused compute in the scan value (e.g. `val = fmaxf(val, 0.0f)` in a clamped prefix sum): one `FMNMX` vs two-op branch form.
-- **When NOT to apply**:
-  - Do NOT rewrite Hillis-Steele's `if (lane_id >= offset) val = ...` as arithmetic masking (`val = ((lane_id >= offset) ? shf(val, offset) : 0) + val`). The compiler predicates the original form to a single `FSEL`; the "arithmetic" version adds a compare + FMUL + FADD (measured 1.27× slower per step in branchless-patterns probe's cond-arith variant).
-  - Do NOT bit-trick the lookback status check — status polling needs correct memory ordering (`memory-ordering` skill), not branch elimination.
-- **Measured on H200**: `fmaxf` 1.56× per comparator step (branchless-patterns probe Group A).
-- **Relevance to bottleneck triage**: Q4 in `reasoning/bottleneck-triage.md` — max-scan kernel showing `FSETP + FSEL` pairs instead of `FMNMX` in the shuffle-scan inner loop. Correct response: switch comparator to `fmaxf`.