@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/compute/half-precision-math.md

@@ -1,328 +0,0 @@
----
-title: Half-Precision Math (fp16 / bf16 scalar and packed)
-status: verified
-evidence_level: measured
-cuda_version_tested: '12.9'
-driver_version_tested: 570.124.06
-toolchain: nvcc 12.9.86 + ptxas 12.9
-measured_on: H200-SXM
-applies_to_pattern_class:
-- cuda-core
-applies_to_ops:
-- elementwise
-- normalization
-applies_to_dtypes:
-- fp16
-- bf16
-- fp32
-requires_sm: '>=7.0'
-requires_features:
-- fp16-simd
-- bf16-simd
-single_kernel_useful: true
-precision:
-  fp16:
-    exponent_bits: 5
-    mantissa_bits: 10
-    max_finite: 65504
-    min_normal: 6.1e-05
-    eps_relative: 0.000488
-    range_decimal: 6.1e-5 to 6.55e4
-    accumulation_safe: false
-    notes: 'Narrow dynamic range. Safe for activations in inference; dangerous for reductions unless promoted to fp32. Denormals: flushed unless the instruction is a `.noftz` variant (e.g., atom.add.noftz.f16 preserves them).'
-  bf16:
-    exponent_bits: 8
-    mantissa_bits: 7
-    max_finite: 3.39e+38
-    min_normal: 1.18e-38
-    eps_relative: 0.00781
-    range_decimal: 1.2e-38 to 3.4e+38 (same as fp32)
-    accumulation_safe: false
-    notes: 'Training-friendly: covers fp32 dynamic range, so no overflow in typical weight / gradient paths. Precision is coarser than fp16 (~7 bits mantissa vs 10). Still not safe for long reduction chains — promote accumulator to fp32.'
-  fp32_accumulator_rule: When input is fp16/bf16 but the operation is a reduction, dot product, softmax normaliser, variance, or any sum of many terms, accumulate in fp32 and only cast back at store time. This is the mixed-precision pattern; skill §S3 documents the canonical shape.
-source:
-- path: spec
-  anchor: Reference
-artifacts:
-  code: artifacts/experience/hw-probes/half2-throughput/half2_throughput_probe.cu
-  build: artifacts/experience/hw-probes/half2-throughput/build.sh
-  introspection: artifacts/experience/hw-probes/half2-throughput/device.json
-  profile: ''
-related_apis:
-- __hadd
-- __hadd2
-- __hmul
-- __hmul2
-- __hfma
-- __hfma2
-- __hfma_relu
-- __hfma2_relu
-- __float2half_rn
-- __half2float
-- __float2bfloat16_rn
-- __bfloat162float
-- __halves2half2
-- __floats2half2_rn
-- __floats2bfloat162_rn
-- hexp
-- hlog
-- hrsqrt
-- h2exp
-- h2log
-- h2rsqrt
-- add.f16x2
-- fma.rn.f16x2
-- fma.rn.bf16x2
-- fma.rn.relu.f16
-- tanh.approx.f16
-- atom.add.noftz.f16
-related_skills:
-- fast-math
-- vectorized-access
-- ilp
-- register-pressure
-- atomic-reduction
-experience_refs:
-- sources/experience/hw-probes/half2-throughput.md
-id: skill-half-precision-math
-type: skill
-vendor: nvidia
-tags:
-- cuda-cpp
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- data-reuse
-- kernel-fusion
-- software-exp
-- fused-kernel
-- gemm
-- attention
-- ptx
-applies_to:
-- general
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1370-L1440
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L25350-L25420
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/parallel-thread-execution/cuda_parallel-thread-execution_index.html.md
-  anchor: L11000-L11150
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-techniques:
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- data-reuse
-- kernel-fusion
-- software-exp
-kernel_types:
-- fused-kernel
-- gemm
-- attention
-confidence: experimental
-artifact_dir: artifacts/experience/hw-probes/half2-throughput
----
-## What
-
-Half-precision math on H200 sm_9.0a means **fp16** (`__half`, `__half2`) or **bf16** (`__nv_bfloat16`, `__nv_bfloat162`) arithmetic, issued as scalar instructions or as 2-way packed SIMD via the `*.f16x2` / `*.bf16x2` PTX forms (`__hfma2`, `__hadd2`, `__hmul2`). Both formats consume 16 bits of storage per value — half the footprint of fp32 — which **doubles effective memory bandwidth** for pure-load kernels (the dominant benefit for elementwise / normalization kernels that are bandwidth-bound).
-
-The arithmetic-throughput advantage is smaller than folklore suggests. Measured on H200 sm_9.0a:
-
-- Scalar fp16 / bf16 FMA run at **~1.58×** scalar fp32 FMA — nvcc auto-packs the scalar chains into `HFMA2.MMA` / `HFMA2.MMA.BF16_V2` SASS when ILP permits, so scalar fp16/bf16 already gets most of the packing benefit.
-- Packed `__hfma2` (fp16) runs at **1.84×** scalar fp32 — only **1.16× more** than scalar fp16. Explicit packing buys the residual register-layout / alignment headroom, not a new hardware path.
-- Packed bf16 runs at **1.62×** scalar fp32 — **12% slower than fp16 packed** because `HFMA2.MMA.BF16_V2` has a lower-throughput variant of the half-precision HFMA2 pipe on sm_9.0a.
-- bf16 scalar vs bf16 packed is only **2.4%** — within noise; the compiler already auto-packs bf16, and the BF16_V2 pipe's lower ceiling absorbs any alignment residual.
-
-Consequently, the skill's decision rules are not "always pack" — they are "choose the format for **precision and range**, then use packed where packing amortises over enough FMAs to matter".
-
-## Why
-
-Three reasons to move a computation from fp32 to fp16 / bf16 on H200:
-
-1. **Memory bandwidth is halved**. Every load and store moves 2 bytes instead of 4. For memory-bound kernels (the common case in elementwise and in the scaling phase of normalization), this is a **2× effective bandwidth** gain — a much bigger lever than the compute throughput gain.
-2. **Compute throughput is 1.58–1.84× vs fp32**. Measurable but smaller than the memory win, and the packing pays only 1.16× on top. Useful for compute-heavy fused chains (activations with transcendentals, fused norm+matmul epilogues).
-3. **Tensor Cores consume fp16 / bf16 natively**. A skill outside this MVP's scope, but the format choice here is downstream of a later TC-based workflow.
-
-The cost is **precision loss** — see §Precision below. The decision rule is: **use the format that preserves correctness, then pack where the compute intensity justifies it**.
-
-## Precision (annotation required in every caller)
-
-*This section exists because half precision introduces **correctness risk**, not just performance tuning. Callers (pattern INDEX.md, skill.md chains) must read this before selecting a half format.*
-
-### Range and resolution
-
-| Format | Exp bits | Mantissa bits | Max finite | Min normal | Unit of last place (at 1.0) |
-|--------|---------:|--------------:|-----------:|-----------:|----------------------------:|
-| fp32   |        8 |            23 |   3.4e+38 |  1.18e-38 |                    1.19e-7 |
-| fp16   |    **5** |        **10** | **6.55e+4** | **6.10e-5** |               **4.88e-4** |
-| bf16   |    **8** |         **7** | **3.4e+38** | **1.18e-38** |              **7.81e-3** |
-
-- **fp16 has the same mantissa resolution as a short float** — about 3 decimal digits of precision. Any time two values differ by <0.05%, fp16 can't distinguish them.
-- **fp16 has a narrow exponent range** — the infamous "65k ceiling". Overflow is trivially easy in any sum of ~1000 elements of magnitude ~100.
-- **bf16 has fp32's range** — overflow is a non-issue — but the mantissa is only 7 bits, about **16× coarser** than fp16. Any computation where small increments matter (decay terms, epsilons, running averages near steady state) will lose them.
-
-### Accumulation rule (load-bearing for correctness)
-
-**Never accumulate a sum of more than a few terms in fp16 or bf16.** Promote to fp32 at the point of summation, cast back only at the store. This is enforced by the canonical "mixed-precision reduction" pattern (skill §S3):
-
-```cuda
-__half* in_h;
-float sum = 0.0f;
-for (int i = tid; i < N; i += stride) {
-    float v = __half2float(in_h[i]);
-    sum = fmaf(v, v, sum);    // fp32 accumulation
-}
-// ... warp/block reduce in fp32 ...
-out_h[tid] = __float2half_rn(sum);   // cast at write
-```
-
-Why: in fp16, summing a thousand terms of ~1.0 accumulates rounding error of order `1000 × 4.88e-4 ≈ 0.5` per unit of sum — any sum near or above 2000 already carries a full ULP of error. In bf16 the situation is **worse** (mantissa 7 bits), just without the overflow.
-
-### Denormals
-
-fp16 and bf16 both represent subnormal values. They are **flushed to zero** unless:
-
-- The instruction is a `.noftz` variant (e.g. `atom.add.noftz.f16` — "no flush-to-zero").
-- The compiler option / kernel attribute opts out of FTZ.
-
-In practice, flush-to-zero is the default, and subnormals round to 0. For most workloads this is fine; for workloads using very small epsilons (normalisation denominators, attention masks with -inf floors) verify that the epsilon is above the min-normal threshold (fp16: 6.1e-5). See pitfall P4.
-
-### When fp16 / bf16 is WRONG
-
-- **Any sum of > ~100 fp16 terms** without fp32 promotion. Overflow hits at sum ≈ 6.55e4; precision drifts long before that (pitfall P1).
-- **Any gradient / backward pass without bf16 or mixed-precision discipline**. Training typically needs bf16's range.
-- **Any computation where epsilon < min-normal** (fp16: 6e-5, bf16: 1e-38). Check the epsilon before selecting the format.
-- **Any atomic reduction that expects fp32 convergence.** Prefer hierarchical fp32 fan-in over native fp16/bf16 atomics (see §S6 and the `atomic-reduction` skill).
-- **Any numerically sensitive operation** (matrix inverse, log-sum-exp with large dynamic range, solving linear systems) — fp16/bf16 mantissas are too coarse.
-
-## When to use
-
-### S1. Use fp16 / bf16 for storage and memory-bound elementwise
-
-For a straight elementwise kernel (`out[i] = activation(in[i])`), the bottleneck is memory; fp16/bf16 halves the bytes moved. This is **2× effective bandwidth** immediately. The compute choice inside the kernel follows S3 (promote to fp32, compute, cast back) — the memory saving is preserved either way.
-
-### S2. Use packed `__hfma2` only for compute-bound FMA chains (fp16 only)
-
-Measured on H200: packed `__hfma2` over `__half2` is **1.16×** scalar `__hfma` on fp16; on bf16 the same explicit-packing gain collapses to **1.02×** (within noise). Packing is justified when:
-
-- The data type is **fp16** (for bf16, explicit packing adds nothing measurable — the compiler already auto-packs scalar into `HFMA2.MMA.BF16_V2`).
-- The kernel is **compute-bound** — in memory-bound kernels the 16% is invisible behind memory stalls (pitfall P11).
-- The FMA chain is long enough to amortise `half2` pack/unpack overhead. Rule of thumb: ≥ 50 FMAs per pack.
-- The data layout allows natural 32-bit alignment (e.g. row-stride even in fp16 elements). Misalignment costs more than the packing win (pitfall P3).
-
-```cuda
-// Packed FMA chain (compute-bound epilogue)
-half2* in_h2; half2* out_h2;
-half2 a = in_h2[i];
-half2 b = __floats2half2_rn(scale, scale);
-half2 bias2 = __floats2half2_rn(bias, bias);
-// One instruction does 2 fp16 FMAs:
-half2 y = __hfma2(a, b, bias2);
-out_h2[i] = y;
-```
-
-If the kernel is memory-bound, skip packing and use scalar `__hfma` — the code is simpler, the alignment risk is gone, and the 1.16× compute gain is invisible behind the memory wall.
-
-### S3. Mixed-precision accumulation pattern (mandatory for reductions)
-
-Any reduction, dot product, norm, softmax numerator, variance accumulator, or running average **must** use fp32 accumulation when the input is fp16 or bf16. See the probe record for the measured rationale; see §Precision above for the numerical reason.
-
-```cuda
-__global__ void fp16_dot_with_fp32_acc(
-    const __half* __restrict__ a,
-    const __half* __restrict__ b,
-    float* __restrict__ out, int N)
-{
-    float acc = 0.0f;
-    int tid = blockIdx.x * blockDim.x + threadIdx.x;
-    int stride = gridDim.x * blockDim.x;
-    for (int i = tid; i < N; i += stride) {
-        float fa = __half2float(a[i]);
-        float fb = __half2float(b[i]);
-        acc = fmaf(fa, fb, acc);
-    }
-    // Warp / block reduce `acc` in fp32 (see reduction skill).
-    // Final atomicAdd(out, acc) in fp32 (see atomic-reduction skill).
-}
-```
-
-### S4. Half-precision transcendentals (`h2exp`, `hrsqrt`, etc.)
-
-`hexp`, `hlog`, `hrsqrt`, `hsqrt` and their packed `h2*` forms are available in `<cuda_fp16.h>`. On H200 the packed forms **should** map to dedicated instructions per PG §5.4.11.2, but this has not been re-measured by this probe — decomposition into fp32 ops on older arches is a documented risk (pitfall P10). Use them for softmax / normalisation denominators where the input is already fp16 and the precision loss is acceptable (normaliser eps must be ≥ 6e-5 for fp16; see §Precision).
-
-### S5. Fused FMA+ReLU (`__hfma2_relu`)
-
-Maps to PTX `fma.rn.relu.f16x2`. Computes `max(a*b + c, 0)` in a single instruction. Useful for the activation epilogue of fused elementwise chains. Instruction-count reduction is small (~4%) and wall-clock benefit is only visible in compute-bound kernels (pitfall P11). Do not adopt this hint in memory-bound elementwise.
-
-### S6. Native fp16 / bf16 atomicAdd
-
-Available on sm_70+ (fp16) and sm_80+ (bf16). PTX maps to `atom.add.noftz.f16` / `atom.add.noftz.bf16`. `.noftz` preserves denormals through the atomic. Contention throughput is worse than fp32 atomicAdd (pitfall P12); prefer the hierarchical reduction pattern from the `atomic-reduction` skill and atomic only the final per-block partials — that way the atomic is in fp32 and cheap.
-
-## When NOT to use
-
-- **Long reductions / running sums / softmax denominators without fp32 promotion.** fp16 overflow at ~6e4, bf16 mantissa error at ~0.8% per addition — numerically unsound (pitfalls P1, P2).
-- **Packed `__hfma2` in memory-bound kernels.** Measured 1.16× over scalar fp16 is invisible behind memory stalls (pitfall P11).
-- **Explicit `__hfma2` on `__nv_bfloat162` for compute speedup.** Measured 1.02× over scalar bf16 on H200 — noise-level. The compiler already auto-packs scalar bf16 into `HFMA2.MMA.BF16_V2`; explicit packing is redundant for bf16.
-- **bf16 in precision-sensitive downstream consumers.** bf16's ~0.8% eps (vs fp16's ~0.05%) is coarser; any operation whose result is within 1% of a decision threshold will have false crossings.
-- **Atomics on a single fp16/bf16 destination under contention.** Use hierarchical fp32 fan-in instead (`atomic-reduction` S1).
-- **Kernels built via `framework extension build` that use `__half` via operators.** Framework extension build defines `-D__CUDA_NO_HALF_OPERATORS__` etc. — use explicit `__hadd`, `__hgt`, `__float2half_rn` calls (pitfall P13).
-
-## Measured Characteristics
-
-Measured on H200-SXM (sm_9.0a, CUDA 12.9, driver 570.124.06) using sources/experience/hw-probes/half2-throughput/ — five FMA variants in a 4-chain ILP compute-bound harness. Full record: sources/experience/hw-probes/half2-throughput.md.
-
-### Scalar-equivalent GFLOPS (counts 2 FP ops per packed instruction)
-
-| variant       | scalar GFLOPS | vs fp32 | vs own-scalar |
-|---------------|--------------:|--------:|--------------:|
-| fp32-scalar   |         28633 |   1.00× |          —    |
-| fp16-scalar   |         45351 | **1.58×** |          —    |
-| fp16-packed   |         52608 | **1.84×** |    **1.16×** |
-| bf16-scalar   |         45381 |   1.59× |          —    |
-| bf16-packed   |         46478 |   1.62× |    **1.02×** |
-
-Key measured findings on H200 sm_9.0a:
-
-- **`__hfma2` is 1.16× scalar `__hfma`**, not 2×. The dominant win is scalar fp16 vs scalar fp32 (1.58×, from compiler auto-packing into `HFMA2.MMA`); explicit packing adds only 16% more.
-- **bf16 packed is 12% slower than fp16 packed**. Scalar bf16 equals scalar fp16 at 45 TFLOPS; the divergence appears only on the packed path because `HFMA2.MMA.BF16_V2` has lower pipe throughput than `HFMA2.MMA`.
-- **bf16 scalar ≈ bf16 packed** (1.02×, within noise). Explicit `__hfma2` on `__nv_bfloat162` adds essentially nothing — auto-packing already lifts scalar bf16 to the BF16_V2 pipe's saturation ceiling.
-- **fp32 scalar runs at 28.6 TFLOPS = 43% of H200's 67 TFLOPS peak**, with NCU Compute SOL = 78%. Probe is compute-bound on issue slot but not register-file-saturated; absolute peak would need wider ILP (a separate skill domain).
-
-## Principles
-
-1. **Format choice is about precision + range, not throughput.** The throughput differences (1.16× packing, 1.58× scalar fp16 vs fp32) are small compared to the correctness risk from coarse mantissa or narrow range.
-2. **Always accumulate in fp32.** Reductions, softmax, norms, variances — fp32 the accumulator, cast at the store.
-3. **Memory bandwidth halving is the real win.** For bandwidth-bound kernels (the common case), fp16/bf16 gives 2× instantly; don't obsess over packing.
-4. **Prefer fp16 to bf16 unless range forces bf16.** Measured 12% throughput edge + 16× finer eps.
-5. **`.noftz` atomics preserve denormals;** scalar non-atomic fp16 ops flush. Verify your epsilons.
-
-## Open questions
-
-- Q1. **RESOLVED** (SASS audit 2026-04-23): `__hfma2` is only 1.16× scalar `__hfma` because **nvcc auto-packs scalar `__hfma` chains into `HFMA2.MMA` SASS opcodes**. Both the scalar and packed templates produce the same `HFMA2.MMA` instruction mix; "scalar vs packed" at C-source is largely erased at SASS when ILP provides independent pairs. The 1.16× residual is register-layout / alignment cost, not the packing gain itself. See probe record §"PTX / SASS audit".
-- Q2. **PARTIALLY RESOLVED** (SASS audit 2026-04-23): bf16 packed is 12% slower than fp16 packed because it emits `HFMA2.MMA.BF16_V2` opcodes — a distinct HFMA2 variant, **not** a fall-through to `FFMA`. Both remain on the half-precision FMA pipe family; the BF16_V2 variant simply has lower measured throughput than the base `HFMA2.MMA` on sm_9.0a hardware. Root cause (pipeline width, multiplier latency, or register port allocation) would need an isolated-instruction microbench to resolve — out of scope here. The compiler hypothesis "bf16 uses FP32 slot" is refuted.
-- Q3. On sm_9.0a do `h2exp`, `h2log`, `h2rsqrt` emit single instructions (as PG §5.4.11.2 implies) or decompose into fp32 ops (pitfall P10)? Follow-up probe `sources/experience/hw-probes/half-transcendental/` (open).
-- Q4. Is `__hfma2_relu` measurably faster than `__hfma2` + explicit `max(x, 0)` in a compute-bound kernel? ~4% instruction reduction is documented; wall-clock gain in compute-bound kernels is not (pitfall P11). Follow-up probe `sources/experience/hw-probes/half-fma-relu/` (open).
-- Q5. Does native `atomicAdd(__half*, __half)` on H200 contention match the "worse than fp32" behavior described in pitfall P12? Not re-measured. Follow-up.
-
-## Legacy references
-
-- `legacy_sandbox_path`: `corpus/nvidia/legacy-optimization/compute/half-precision-math/skill.md`. The legacy skill kept six sub-skills (S1 packed h2, S2 bf16, S3 mixed-precision accumulation, S4 transcendentals, S5 hfma_relu, S6 native atomics). This port:
-  - Re-orders by the measured H200 hierarchy: memory-bw-first (§S1), packed-is-marginal (§S2, fp16 only), accumulation-in-fp32 (§S3), transcendentals (§S4), fused FMA+ReLU (§S5), native atomics (§S6).
-  - Adds the **Precision** section before any sub-skill — making the correctness-risk annotation the first thing a pattern-level INDEX or ROUTING caller reads.
-  - The legacy "2× from packing" claim is corrected by the half2-throughput probe to 1.16× on fp16 / 1.02× on bf16.
-- Legacy pitfalls retained in body (rewritten to stand on PG / PTX ISA sources): P1 (fp16 overflow), P2 (bf16 mantissa loss), P3 (packed alignment), P4 (denormal FTZ), P5 (conversion overhead), P6 (bf16 sm_80+ only), P13 (the external framework extension build flags). P10 / P11 / P12 kept as open risks (not re-measured on H200; follow-up probes listed in §Open questions).
-- Legacy sandbox-only pitfalls dropped from body (no independent doc source, not re-measured by the half2-throughput probe): "L1 cache hit rate drop with half2" and "BF16 packed changes cache-line reuse" — these were empirical observations without authoritative grounding; they are available in the legacy-knowledge tree if needed but are not carried forward as measured facts.
-- Measured pitfalls added by the half2-throughput probe (2026-04-23): P14 (`__hfma2` is 1.16× not 2×), P15 (bf16 packed is 12% slower than fp16 packed), P16 (`<cuda_bf16.h>` overload-resolution gotcha).
-- **Related but distinct**:
-  - `wiki/nvidia/foundations/compute/fast-math/` covers fp32-specific fast-math approximations (`-use_fast_math` only affects fp32). The two don't compose automatically.
-  - `wiki/nvidia/foundations/memory/vectorized-access/`: packed `half2` aligns naturally to 4-byte loads. Use vectorized-access for load/store widening, this skill for the arithmetic inside.
-  - `wiki/nvidia/foundations/memory/register-pressure/`: packed formats double accumulator width; check regs-per-thread after adopting §S2.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/compute/ilp/pitfalls.md

@@ -1,148 +0,0 @@
----
-title: Instruction-Level Parallelism (ILP) - Pitfalls
-status: draft
-evidence_level: measured
-applies_to_pattern_class:
-- cuda-core
-applies_to_ops:
-- reduction
-- scan
-- elementwise
-- normalization
-requires_sm: '>=3.0'
-single_kernel_useful: true
-source:
-- path: spec
-  anchor: Reference
-id: pitfall-ilp
-type: pitfall
-vendor: nvidia
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1064-L1065
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1124
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L24765-L24805
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1229-L1232
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-techniques:
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- register-budgeting
-- loop-unrolling
-- communication-overlap
-kernel_types:
-- fused-kernel
-confidence: experimental
-tags:
-- pipeline-stages
-- vectorized-loads
-- cache-policy
-- register-budgeting
-- loop-unrolling
-- communication-overlap
-- fused-kernel
-- ptx
-- cuda-cpp
----
-## P1: Register pressure from excessive accumulators
-
-**Symptom**: Kernel occupancy drops significantly after adding ILP accumulators, and overall throughput decreases despite higher per-thread throughput.
-
-**Cause**: Each additional accumulator chain adds registers. On H200, the register file is 65536 32-bit registers per SM. If a kernel already uses, say, 128 registers/thread at 256 threads/SM (2 blocks of 128), adding 4 extra FP32 accumulators increases register usage to 132 registers/thread, potentially preventing the launch of a second block. The loss of TLP (thread-level parallelism) from reduced occupancy can outweigh the gain from ILP.
-
-**Detection**: Compile with `nvcc -Xptxas=-v` to see register usage per kernel. Compare occupancy before and after adding accumulators using the CUDA Occupancy Calculator.
-
-**Fix**: Limit the number of accumulators to the pipeline depth (4 for FP32 on H200). Going beyond 4 gives no ILP benefit (measured: 8-acc = 4-acc = 1.01 cycles/FMA) but increases register pressure. If occupancy is already critical, consider using `__launch_bounds__` or `-maxrregcount` to cap registers, and accept a smaller ILP factor (2 accumulators give 2x, which may be the best tradeoff).
-
-## P2: Compiler defeats manual ILP via optimization
-
-**Symptom**: Hand-written multi-accumulator code runs no faster than single accumulator.
-
-**Cause**: The compiler may recognize that the accumulators are independent and re-merge them, or it may already be applying ILP-like scheduling internally. More commonly, `-O3` with `#pragma unroll` may produce code where the unrolled iterations are *not* truly independent (e.g., because of address computation dependencies or aliasing concerns).
-
-**Detection**: Inspect the generated SASS with `cuobjdump -sass <binary>`. Look for back-to-back dependent instructions with no independent instructions interleaved between them.
-
-**Fix**: Use `__fmaf_rn` (intrinsic FMA) instead of plain `a * b + c` if the compiler is fusing/reordering in unexpected ways. Mark accumulators as `volatile` during debugging to confirm they are not being merged (but remove `volatile` for production, as it prevents register allocation). Alternatively, use inline PTX assembly (`asm volatile`) to guarantee instruction ordering.
-
-## P3: Applying ILP to memory-bound kernels
-
-**Symptom**: Adding multiple accumulators to a simple elementwise or copy-like kernel shows no speedup.
-
-**Cause**: The kernel is bottlenecked on memory bandwidth, not arithmetic throughput. The FP32 pipeline is already idle most of the time, waiting for data from L2 or HBM. Adding ILP to arithmetic instructions cannot fix a memory bottleneck.
-
-**Detection**: Use `ncu` (Nsight Compute) to check the `sm__pipe_fma_cycles_active` metric. If FMA pipe utilization is already low (e.g., < 20%), the kernel is memory-bound and arithmetic ILP will not help.
-
-**Fix**: For memory-bound kernels, ILP on the *memory access* side is what matters -- issuing multiple independent loads so the memory subsystem can overlap requests. This is a different technique: use vectorized loads (`float4`), or load multiple elements before processing any of them (software pipelining). See the `vectorized-access` and `coalescing` skills.
-
-## P4: Excessive unrolling increases instruction cache pressure
-
-**Symptom**: Large unroll factors (e.g., `#pragma unroll 32` or full unroll of a long loop) cause unexpected slowdown.
-
-**Cause**: Unrolling a loop N times multiplies the instruction footprint by ~N. If the unrolled code exceeds the L0/L1 instruction cache capacity, the SM stalls on instruction cache misses. On H200, the instruction cache is limited (exact size not publicly documented), and very large unrolled loops can trigger this.
-
-**Detection**: In Nsight Compute, check `smsp__inst_executed` vs `smsp__warps_launched` -- if the ratio is unexpectedly high, the loop may be over-unrolled. Also look for `stall_imc` (instruction miss count) stalls.
-
-**Fix**: Limit unrolling to the ILP-useful depth. For FP32, `#pragma unroll 4` is sufficient to saturate the pipeline. Unrolling beyond 4 increases code size without throughput benefit.
-
-## P5: Non-independent loop iterations mistaken for ILP
-
-**Symptom**: `#pragma unroll` applied to a loop where iterations have dependencies does not improve throughput.
-
-**Cause**: Unrolling only exposes ILP if iterations are truly independent. In a scan (prefix sum), for example, iteration `i` depends on iteration `i-1`, so unrolling merely inlines the loop without creating independent chains.
-
-**Example**:
-```cuda
-// WRONG: these iterations are dependent -- unrolling does not add ILP
-float prefix = 0.0f;
-#pragma unroll 4
-for (int i = 0; i < N; ++i) {
-    prefix += in[i];       // depends on previous prefix
-    out[i] = prefix;
-}
-```
-
-**Fix**: Restructure the algorithm to have independent sub-computations. For reductions, split input into K independent partial sums (one per accumulator), then combine at the end. For scans, use the Blelloch or Hillis-Steele parallel algorithms where multiple elements can be processed independently within each step.
-
-## P6: FP associativity changes with multiple accumulators
-
-**Symptom**: Multi-accumulator reduction produces slightly different numerical results compared to single-accumulator version.
-
-**Cause**: Floating-point addition is not associative. Changing the order of summation (by splitting into K independent partial sums and then combining) changes the rounding pattern. For fp32, the difference is typically within a few ULP for well-conditioned sums, but for ill-conditioned sums (e.g., alternating large positive and negative values), the difference can be significant.
-
-**Detection**: Compare max absolute error between single-accumulator and multi-accumulator results on representative inputs.
-
-**Fix**: This is inherent to the technique and usually acceptable. If exact bitwise reproducibility with the serial order is required, ILP via multiple accumulators cannot be used for reduction. Consider compensated summation (Kahan) if both ILP and high numerical accuracy are needed -- Kahan summation can be applied independently to each accumulator chain.
-
-## P7: Forgetting to combine accumulators after the loop
-
-**Symptom**: Incorrect final result (only 1/K of the true sum).
-
-**Cause**: After a K-accumulator reduction loop, the programmer forgets to add the K partial sums together before the warp-level or block-level reduction step.
-
-**Example**:
-```cuda
-float acc0 = 0, acc1 = 0, acc2 = 0, acc3 = 0;
-for (int i = tid; i < N; i += stride * 4) {
-    acc0 += in[i];
-    acc1 += in[i + stride];
-    acc2 += in[i + stride * 2];
-    acc3 += in[i + stride * 3];
-}
-// BUG: only acc0 is used in the warp reduction
-float val = acc0;  // Should be: acc0 + acc1 + acc2 + acc3
-```
-
-**Fix**: Always combine all accumulators before proceeding to the collective reduction step: `float val = acc0 + acc1 + acc2 + acc3;`.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/compute/ilp.md

@@ -1,188 +0,0 @@
----
-title: Instruction-Level Parallelism (ILP)
-status: draft
-evidence_level: measured
-applies_to_pattern_class:
-- cuda-core
-applies_to_ops:
-- reduction
-- scan
-- elementwise
-- normalization
-requires_sm: '>=3.0'
-requires_features: []
-single_kernel_useful: true
-cuda_version_tested: '12.9'
-driver_version_tested: 570.124.06
-toolchain: nvcc 12.9 + ptxas 12.9
-measured_on: H200-SXM
-source:
-- path: spec
-  anchor: Reference
-artifacts:
-  code: artifacts/experience/hw-probes/ilp/ilp_fma_probe.cu
-  build: nvcc -arch=sm_90a -O3 -std=c++17 -lineinfo -o ilp_fma_probe ilp_fma_probe.cu
-  introspection: ''
-  profile: ''
-related_apis: []
-related_skills:
-- warp-primitives
-- fast-math
-id: skill-ilp
-type: skill
-vendor: nvidia
-tags:
-- cuda-cpp
-- pipeline-stages
-- register-budgeting
-- loop-unrolling
-- tile-scheduling
-- communication-overlap
-- fused-kernel
-applies_to:
-- general
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1113
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1124
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1229-L1232
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L3452
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L24765-L24805
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1064-L1065
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- pipeline-stages
-- register-budgeting
-- loop-unrolling
-- tile-scheduling
-- communication-overlap
-kernel_types:
-- fused-kernel
-confidence: experimental
-artifact_dir: artifacts/experience/hw-probes/ilp
----
-## What
-
-Instruction-Level Parallelism (ILP) is a technique where a single thread maintains multiple independent instruction chains in flight simultaneously, allowing the hardware to overlap the execution of one chain's instructions with the pipeline latency of another. On NVIDIA GPUs, the FP32 arithmetic pipeline has a latency of 4 cycles (measured on H200, consistent with the best-practices guide statement for compute capability 7.0+). A single dependent chain of FMA instructions therefore achieves only 1/4 of the peak throughput. By restructuring code to have 4 independent chains, a single thread can fully saturate the pipeline and reach peak throughput.
-
-ILP is one of two complementary mechanisms for hiding arithmetic latency:
-
-1. **Occupancy (TLP)**: the warp scheduler switches between warps on every cycle, so if enough warps are resident, a stalled warp is replaced by a ready one. This requires many active threads.
-2. **ILP**: within a single warp, the scheduler can issue independent instructions from the same warp on consecutive cycles. This requires independent instruction chains within each thread, but does not require additional warps.
-
-The best-practices guide explicitly notes (L1124): "with a high degree of exposed instruction-level parallelism (ILP) it is, in some cases, possible to fully cover latency with a low occupancy."
-
-## Why
-
-On H200 (sm_90a), the FP32 pipeline has a throughput of 128 operations per clock per SM (Table 5 of the best-practices guide). With a warp size of 32 threads, this means 128 / 32 = 4 warp-instructions per clock per SM (one per sub-partition). For a single warp running on one sub-partition, the throughput ceiling is 1 FMA per clock per warp.
-
-However, with a single dependent FMA chain, the measured throughput is only 0.25 FMA per clock per warp (4.01 cycles per FMA), because each instruction must wait for the previous one to complete. With 4 independent accumulator chains, the measured throughput is 0.99 FMA per clock per warp (1.01 cycles per FMA) -- a **3.97x improvement** from ILP alone.
-
-This matters in practice because:
-
-- **Reduction kernels** with a single accumulator leave 75% of the FP32 pipeline idle during the accumulation phase.
-- **Compute-bound kernels** at low occupancy (e.g., due to high register or shared-memory usage) cannot rely on TLP to hide latency; ILP is the only alternative.
-- **Grid-stride loops** that process multiple elements per thread naturally expose ILP when each element is accumulated into a separate register.
-
-## When to use
-
-- **Reduction accumulation loops**: replace a single accumulator with N independent accumulators (N >= 4 for FP32 on H200). After the loop, combine them with a small tree reduction.
-
-- **Grid-stride loops processing multiple elements per thread**: load and process K elements per iteration, accumulating into K independent registers. This is ILP by design.
-
-- **Compute-bound tight loops**: any loop body where the critical path is a chain of dependent arithmetic operations. Unrolling + splitting into independent sub-chains exposes ILP.
-
-- **Low-occupancy kernels**: when register pressure or shared-memory usage limits occupancy, ILP compensates for the reduced TLP by keeping the pipeline fed from within each warp.
-
-- **`#pragma unroll`**: unrolling a loop exposes multiple iterations to the compiler, which can then schedule independent operations from different iterations in parallel. This is the simplest way to add ILP: if iterations are independent, `#pragma unroll` (or `#pragma unroll N`) lets the compiler see and schedule them concurrently.
-
-## When NOT to use
-
-- **Memory-bound kernels (elementwise, simple copy)**: if the kernel is bottlenecked on memory bandwidth, adding ILP to the arithmetic does not help -- the arithmetic pipeline is already idle waiting for data. In this case, ILP on memory access patterns (e.g., issuing multiple loads in flight) is what matters, not arithmetic ILP.
-
-- **Already at full occupancy with independent warps**: if the SM has enough resident warps to keep the pipeline busy via TLP, additional ILP within each thread provides no benefit and wastes registers.
-
-- **When register pressure is already critical**: each additional accumulator chain requires additional registers. On H200, the register file is 65536 32-bit registers per SM. If the kernel is already register-limited (the occupancy calculator shows register as the limiting factor), adding more accumulators can reduce occupancy, potentially hurting total throughput even as per-thread throughput improves. The sweet spot is typically 4 accumulators for FP32 (matching the pipeline depth).
-
-- **Short loops with few iterations**: if the total iteration count is small (e.g., < 8), unrolling and splitting into chains adds code size with negligible benefit. The pipeline stall from a 4-iteration loop is only ~16 cycles total.
-
-## Classical example: 4-accumulator reduction
-
-This pattern replaces a single-accumulator grid-stride reduction with 4 independent chains, achieving near-peak FMA throughput during the accumulation phase.
-
-```cuda
-__global__ void reduce_sum_ilp4(const float* __restrict__ in,
-                                float*       __restrict__ out,
-                                int n) {
-    // 4 independent accumulators per thread
-    float acc0 = 0.0f, acc1 = 0.0f, acc2 = 0.0f, acc3 = 0.0f;
-
-    int tid    = blockIdx.x * blockDim.x + threadIdx.x;
-    int stride = blockDim.x * gridDim.x;
-
-    // Grid-stride loop: each thread processes 4 elements per iteration
-    int i = tid * 4;
-    for (; i + 3 < n; i += stride * 4) {
-        acc0 += in[i];
-        acc1 += in[i + 1];
-        acc2 += in[i + 2];
-        acc3 += in[i + 3];
-    }
-    // Handle remainder
-    for (; i < n; i += stride) {
-        acc0 += in[i];
-    }
-
-    // Combine accumulators (serial, but only 3 adds)
-    float val = acc0 + acc1 + acc2 + acc3;
-
-    // Warp-level reduction (see warp-primitives skill)
-    for (int offset = 16; offset > 0; offset >>= 1)
-        val += __shfl_xor_sync(0xFFFFFFFF, val, offset);
-
-    if (threadIdx.x % 32 == 0)
-        atomicAdd(out, val);
-}
-```
-
-In this example, the 4 `+=` operations in the inner loop body are independent of each other, so the hardware can issue them on 4 consecutive cycles without pipeline stalls. Compared to a single-accumulator version, the accumulation phase runs at approximately 4x the throughput.
-
-## ILP via `#pragma unroll`
-
-When loop iterations are independent, `#pragma unroll` is the simplest way to expose ILP without manually managing multiple accumulators. The compiler unrolls the specified number of iterations and schedules independent operations across them.
-
-```cuda
-// Without unroll: compiler may not unroll, limiting ILP
-for (int i = 0; i < N; ++i)
-    out[tid + i * stride] = __fmaf_rn(in[tid + i * stride], scale, bias);
-
-// With unroll: compiler sees 4 independent iterations, schedules ILP
-#pragma unroll 4
-for (int i = 0; i < N; ++i)
-    out[tid + i * stride] = __fmaf_rn(in[tid + i * stride], scale, bias);
-```
-
-Key rules for `#pragma unroll` (from the programming guide L24765-L24805):
-- No argument: fully unroll if trip count is compile-time constant.
-- Argument N: unroll N iterations (N must be a positive integral constant expression).
-- Argument 0 or 1: disable unrolling.
-- Negative values: the pragma is ignored.
-
-## Measured Characteristics
-
-- ILP FMA accumulator-chain probe: On H200 (sm_90a, CUDA 12.9), a dependent FMA chain (1 accumulator) costs **4.01 cycles/FMA**. With 2 independent chains: **2.00 cycles/FMA** (2.00x speedup). With 4 independent chains: **1.01 cycles/FMA** (3.97x speedup). With 8 independent chains: **1.01 cycles/FMA** (no further gain). The FP32 pipeline depth is confirmed to be 4 stages; 4 independent chains are necessary and sufficient to reach peak single-warp throughput.