@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/compute/occupancy-tuning/apis.md

@@ -1,130 +0,0 @@
----
-func_name: cudaOccupancyMaxActiveBlocksPerMultiprocessor
-namespace: runtime
-header: cuda_runtime_api.h
-signature: cudaError_t cudaOccupancyMaxActiveBlocksPerMultiprocessor(int* numBlocks, const void* func, int blockSize, size_t dynamicSMemSize)
-status: documented
-has_end_to_end_example: true
-source:
-- path: spec
-  anchor: Reference
-id: api-occupancy-tuning-ref
-type: api-definition
-vendor: nvidia
-title: Apis
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA API References/cuda-runtime-api/cuda_cuda-runtime-api_index.html.md
-  anchor: L3862-L3890
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-hardware_features:
-- cluster
-techniques:
-- shared-memory-optimization
-confidence: inferred
-evidence_level: inferred
-tags:
-- cluster
-- shared-memory-optimization
-- cuda-cpp
----
-# Occupancy Tuning API Reference
-
-This file lists the APIs touched by the occupancy-tuning skill. Each entry records the function name, namespace, and signature as found in upstream documentation.
-
-## Occupancy Query Functions
-
-### `cudaOccupancyMaxActiveBlocksPerMultiprocessor`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C signature**: `__host__ __device__ cudaError_t cudaOccupancyMaxActiveBlocksPerMultiprocessor(int* numBlocks, const void* func, int blockSize, size_t dynamicSMemSize)`
-- **C++ template**: `template <class T> __host__ cudaError_t cudaOccupancyMaxActiveBlocksPerMultiprocessor(int* numBlocks, T func, int blockSize, size_t dynamicSMemSize)`
-- **Semantics**: Returns in `*numBlocks` the maximum number of active blocks per SM for the given kernel, block size, and dynamic shared memory size. This is the primary API for computing theoretical occupancy.
-- **Occupancy derivation**: `occupancy = (numBlocks * blockSize / warpSize) / (maxThreadsPerSM / warpSize)`
-- **Host and device callable**: Can be called from both host and device code (C version only; C++ template is host-only).
-- **Source**: CUDA Runtime API Reference L3862-L3890
-
-### `cudaOccupancyMaxActiveBlocksPerMultiprocessorWithFlags`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C signature**: `__host__ cudaError_t cudaOccupancyMaxActiveBlocksPerMultiprocessorWithFlags(int* numBlocks, const void* func, int blockSize, size_t dynamicSMemSize, unsigned int flags)`
-- **C++ template**: `template <class T> __host__ cudaError_t cudaOccupancyMaxActiveBlocksPerMultiprocessorWithFlags(int* numBlocks, T func, int blockSize, size_t dynamicSMemSize, unsigned int flags)`
-- **Semantics**: Same as `cudaOccupancyMaxActiveBlocksPerMultiprocessor` but accepts a flags parameter. Supported flags:
-  - `cudaOccupancyDefault`: default behavior (same as the non-flags version)
-  - `cudaOccupancyDisableCachingOverride`: suppresses the default behavior where the occupancy calculator pretends caching is disabled when per-block SM resource usage would result in zero occupancy with caching enabled
-- **Source**: CUDA Runtime API Reference L3894-L3929
-
-## Launch Configuration Functions
-
-### `cudaOccupancyMaxPotentialBlockSize`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C++ template**: `template <class T> __host__ cudaError_t cudaOccupancyMaxPotentialBlockSize(int* minGridSize, int* blockSize, T func, size_t dynamicSMemSize = 0, int blockSizeLimit = 0)`
-- **Semantics**: Returns in `*minGridSize` and `*blockSize` a grid/block size pair that achieves the maximum potential occupancy (i.e., the maximum number of active warps with the smallest number of blocks). The `blockSizeLimit` parameter caps the block size (0 means no limit).
-- **Important**: This optimizes for occupancy, not latency. The suggested block size may not be the fastest in practice — always benchmark alternatives.
-- **Source**: CUDA Runtime API Reference L17092-L17121
-
-### `cudaOccupancyMaxPotentialBlockSizeVariableSMem`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C++ template**: `template <typename UnaryFunction, class T> __host__ cudaError_t cudaOccupancyMaxPotentialBlockSizeVariableSMem(int* minGridSize, int* blockSize, T func, UnaryFunction blockSizeToDynamicSMemSize, int blockSizeLimit = 0)`
-- **Semantics**: Same as `cudaOccupancyMaxPotentialBlockSize` but accepts a unary function that maps block size to dynamic shared memory size. Use this when the amount of dynamic shared memory depends on the block size (e.g., `smem = blockSize * sizeof(float)`).
-- **Source**: CUDA Runtime API Reference L17141-L17181
-
-### `cudaOccupancyMaxPotentialBlockSizeWithFlags`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C++ template**: `template <class T> __host__ cudaError_t cudaOccupancyMaxPotentialBlockSizeWithFlags(int* minGridSize, int* blockSize, T func, size_t dynamicSMemSize = 0, int blockSizeLimit = 0, unsigned int flags = 0)`
-- **Semantics**: Same as `cudaOccupancyMaxPotentialBlockSize` with an additional flags parameter (same flags as `cudaOccupancyMaxActiveBlocksPerMultiprocessorWithFlags`).
-- **Source**: CUDA Runtime API Reference (see also section)
-
-### `cudaOccupancyMaxPotentialBlockSizeVariableSMemWithFlags`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C++ template**: `template <typename UnaryFunction, class T> __host__ cudaError_t cudaOccupancyMaxPotentialBlockSizeVariableSMemWithFlags(int* minGridSize, int* blockSize, T func, UnaryFunction blockSizeToDynamicSMemSize, int blockSizeLimit = 0, unsigned int flags = 0)`
-- **Semantics**: Combines variable shared memory and flags support.
-- **Source**: CUDA Runtime API Reference L17185-L17228
-
-## Shared Memory Query
-
-### `cudaOccupancyAvailableDynamicSMemPerBlock`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C signature**: `__host__ cudaError_t cudaOccupancyAvailableDynamicSMemPerBlock(size_t* dynamicSmemSize, const void* func, int numBlocks, int blockSize)`
-- **C++ template**: `template <class T> __host__ cudaError_t cudaOccupancyAvailableDynamicSMemPerBlock(size_t* dynamicSmemSize, T* func, int numBlocks, int blockSize)`
-- **Semantics**: Returns in `*dynamicSmemSize` the maximum size of dynamic shared memory that allows `numBlocks` blocks per SM. This is the inverse of the occupancy query: instead of "given SMem, how many blocks?", it answers "given N blocks, how much SMem?"
-- **Use case**: Determine how much shared memory a kernel can use while maintaining a target occupancy level.
-- **Source**: CUDA Runtime API Reference L3828-L3858
-
-## Cluster Occupancy (Hopper+)
-
-### `cudaOccupancyMaxActiveClusters`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C signature**: `__host__ cudaError_t cudaOccupancyMaxActiveClusters(int* numClusters, const void* func, const cudaLaunchConfig_t* launchConfig)`
-- **Semantics**: Returns the maximum number of active clusters on the device. For applications using Thread Block Clusters, the Hopper tuning guide recommends using this API instead of the per-SM occupancy APIs.
-- **Source**: CUDA Runtime API Reference L3933-L3968
-
-### `cudaOccupancyMaxPotentialClusterSize`
-
-- **Namespace**: runtime
-- **Header**: `cuda_runtime_api.h`
-- **C signature**: `__host__ cudaError_t cudaOccupancyMaxPotentialClusterSize(int* clusterSize, const void* func, const cudaLaunchConfig_t* launchConfig)`
-- **Semantics**: Returns the maximum potential cluster size for a kernel. Used to determine the optimal cluster configuration for Thread Block Clusters.
-- **Source**: CUDA Runtime API Reference L3972-L4009
-
-## Related Probes
-
-- `cudaOccupancyMaxActiveBlocksPerMultiprocessor` — end-to-end example, build command, and H200 measurement. See probe record.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/compute/occupancy-tuning/pitfalls.md

@@ -1,177 +0,0 @@
----
-title: Occupancy Tuning - Pitfalls
-status: draft
-evidence_level: measured
-applies_to_pattern_class:
-- cuda-core
-applies_to_ops:
-- elementwise
-- reduction
-- normalization
-- scan
-requires_sm: '>=3.0'
-single_kernel_useful: true
-source:
-- path: spec
-  anchor: Reference
-id: pitfall-occupancy-tuning
-type: pitfall
-vendor: nvidia
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1089
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1124
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1093-L1097
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1133-L1134
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1123
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- cache-policy
-- register-budgeting
-- shared-memory-optimization
-- tile-scheduling
-kernel_types:
-- fused-kernel
-confidence: experimental
-tags:
-- cache-policy
-- register-budgeting
-- shared-memory-optimization
-- tile-scheduling
-- fused-kernel
-- cuda-cpp
----
-## P1: Assuming higher occupancy always means better performance
-
-**Symptom**: Kernel performance does not improve (or gets worse) after increasing occupancy via register reduction or block size changes.
-
-**Cause**: The best-practices guide explicitly states: "Higher occupancy does not always equate to higher performance — there is a point above which additional occupancy does not improve performance." The measured data confirms this: for a register-heavy kernel (56 regs/thread), block size 512 at 50% occupancy is 10% faster than block size 64 at 56.2% occupancy.
-
-Occupancy helps hide latency by providing more warps for the scheduler to switch to. But once there are enough warps to cover the critical-path latency, additional warps provide diminishing returns. For compute-bound kernels with high ILP, even 25% occupancy may be sufficient.
-
-**Detection**: Compare latency at different occupancy levels using the occupancy sweep probe pattern. If increasing occupancy does not reduce latency, you have reached the saturation point.
-
-**Fix**: Do not chase 100% occupancy. Profile at multiple block sizes and register counts. Choose the configuration with the best latency, not the highest occupancy.
-
-## P2: Using cudaOccupancyMaxPotentialBlockSize as the final answer
-
-**Symptom**: The block size suggested by `cudaOccupancyMaxPotentialBlockSize` is not the fastest in practice.
-
-**Cause**: This API returns the block size that maximizes **theoretical occupancy**, not the block size that minimizes latency. The measured data shows that for vec_add, the API suggests 1024 but 512 is 1.7% faster. For the register-heavy kernel, the API suggests 576 but 512 is fastest among the tested sizes.
-
-The API does not account for:
-- Block scheduling overhead (more blocks = more overhead)
-- Memory coalescing efficiency (varies with block size)
-- Cache behavior (L1/L2 hit rates depend on access patterns per block)
-- Instruction-level parallelism within each thread
-
-**Detection**: Always benchmark the suggested block size against nearby alternatives (e.g., 128, 256, 512).
-
-**Fix**: Use `cudaOccupancyMaxPotentialBlockSize` as a starting point, then sweep block sizes around the suggestion. The occupancy-optimal block size is a good first guess, but the latency-optimal block size must be measured.
-
-## P3: Forgetting register allocation granularity
-
-**Symptom**: Occupancy calculated by hand does not match the occupancy API result.
-
-**Cause**: On sm_90a, registers are allocated per warp, rounded up to the nearest 256 registers per warp. A kernel using 37 registers per thread requires 37 × 32 = 1184 registers per warp, which rounds up to 1280 (next multiple of 256). This rounding affects the total register consumption per block and thus the number of blocks that fit on an SM.
-
-The best-practices guide gives an example: "on a device of compute capability 7.0, a kernel with 128-thread blocks using 37 registers per thread results in an occupancy of 75% with 12 active 128-thread blocks per multi-processor, whereas a kernel with 320-thread blocks using the same 37 registers per thread results in an occupancy of 63%."
-
-**Detection**: Use `cudaOccupancyMaxActiveBlocksPerMultiprocessor` instead of hand calculation. The API accounts for all granularity and rounding effects.
-
-**Fix**: Never hand-calculate occupancy for performance decisions. Always use the occupancy API or the Nsight Compute occupancy calculator.
-
-## P4: Ignoring shared memory as an occupancy limiter
-
-**Symptom**: Kernel has low occupancy despite low register usage.
-
-**Cause**: Shared memory can limit occupancy independently of registers. On H200, each SM has 233,472 bytes of shared memory. If a kernel uses 128 KB of shared memory per block, only 1 block fits per SM (233472 / 131072 = 1.78 → 1 block), giving occupancy of blockSize / 2048 regardless of register usage.
-
-The best-practices guide notes: "Shared memory can also act as a constraint on occupancy." This is especially common in tiled kernels (matrix multiply, convolution) where the tile size determines shared memory usage.
-
-**Detection**: Check `cudaOccupancyMaxActiveBlocksPerMultiprocessor` and compare against the register-derived limit. If the API returns fewer blocks than the register limit allows, shared memory is the bottleneck.
-
-**Fix**: Reduce shared memory per block (smaller tiles, different data types), or use `cudaFuncSetAttribute` with `cudaFuncAttributePreferredSharedMemoryCarveout` to increase the shared memory partition. Consider processing multiple elements per thread to amortize shared memory usage.
-
-## P5: Block size not a multiple of warp size
-
-**Symptom**: Kernel performance is lower than expected for the given occupancy.
-
-**Cause**: The best-practices guide recommends: "The number of threads per block should be a multiple of 32 threads, because this provides optimal computing efficiency and facilitates coalescing." A block size that is not a multiple of 32 wastes warp slots: the last partial warp still consumes a full warp's worth of scheduling resources but does less work.
-
-**Detection**: Check if `blockSize % 32 != 0`. The occupancy API accounts for this correctly, but the effective throughput per warp is reduced.
-
-**Fix**: Always use block sizes that are multiples of 32 (warp size). Common choices: 64, 128, 256, 512, 1024.
-
-## P6: Forcing high occupancy causes register spilling
-
-**Symptom**: Kernel becomes slower after adding `__launch_bounds__` with a high `minBlocksPerMultiprocessor` or after setting `-maxrregcount` to a low value. `-Xptxas=-v` shows non-zero spill stores/loads.
-
-**Cause**: The best-practices guide warns: "A lower occupancy kernel will have more registers available per thread than a higher occupancy kernel, which may result in less register spilling to local memory." Forcing high occupancy by capping registers causes the compiler to spill excess values to local memory (which resides in global memory). The latency of spilled loads/stores far exceeds the benefit of having more active warps.
-
-**Detection**: Compile with `-Xptxas=-v` and check for non-zero "spill stores" and "spill loads." Any non-zero spill count is a red flag.
-
-**Fix**: Reduce `minBlocksPerMultiprocessor` or increase `-maxrregcount` until spill counts reach zero. A kernel at 50% occupancy with zero spills is almost always faster than the same kernel at 100% occupancy with spills. See the register-pressure skill for detailed guidance.
-
-## P7: Assuming block size alone determines occupancy
-
-**Symptom**: Changing block size does not change occupancy as expected.
-
-**Cause**: The best-practices guide states: "When choosing the block size, it is important to remember that multiple concurrent blocks can reside on a multiprocessor, so occupancy is not determined by block size alone." A smaller block size allows more blocks per SM, potentially maintaining the same total thread count. For example, on H200 with a kernel using 12 registers per thread:
-- Block size 256: 8 blocks × 256 = 2048 threads = 100% occupancy
-- Block size 128: 16 blocks × 128 = 2048 threads = 100% occupancy
-- Block size 64: 32 blocks × 64 = 2048 threads = 100% occupancy
-
-All achieve the same occupancy despite different block sizes.
-
-**Detection**: Use `cudaOccupancyMaxActiveBlocksPerMultiprocessor` to check how many blocks fit per SM at each block size.
-
-**Fix**: Understand that occupancy = (numBlocks × blockSize) / maxThreads. Both numBlocks and blockSize matter. The occupancy API handles this correctly.
-
-## P8: Not opting in for large dynamic shared memory
-
-**Symptom**: Kernel launch fails with "invalid configuration argument" when requesting more than 48 KB of dynamic shared memory.
-
-**Cause**: By default, kernels are limited to 48 KB (49,152 bytes) of dynamic shared memory. Using more requires an explicit opt-in via `cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, bytes)` before the launch. The occupancy API (`cudaOccupancyAvailableDynamicSMemPerBlock`) can tell you how much dynamic shared memory is available at a given occupancy target.
-
-**Detection**: If the kernel uses `extern __shared__` and the third launch parameter exceeds 49152, the launch will fail without the opt-in call.
-
-**Fix**: Call `cudaFuncSetAttribute` before the first launch:
-```cuda
-cudaFuncSetAttribute(my_kernel,
-    cudaFuncAttributeMaxDynamicSharedMemorySize, required_bytes);
-my_kernel<<<grid, block, required_bytes>>>(...);
-```
-
-## P9: Overlooking the max-blocks-per-SM limit
-
-**Symptom**: Occupancy is lower than expected even with very small block sizes and low register/shared memory usage.
-
-**Cause**: H200 limits each SM to 32 concurrent blocks. With block size 32 (one warp per block), the maximum occupancy from blocks alone is 32 × 32 = 1024 threads = 50%, even though registers and shared memory would allow 2048 threads. This is the "max blocks per SM" limit.
-
-**Detection**: If `cudaOccupancyMaxActiveBlocksPerMultiprocessor` returns 32 (or the device max) and `32 × blockSize < 2048`, the block count limit is the bottleneck.
-
-**Fix**: Increase block size so that fewer blocks are needed to reach full occupancy. Block size 64 (2 warps) allows 32 × 64 = 2048 = 100%. Block size 32 (1 warp) is capped at 50%.
-
-## P10: Using occupancy APIs in performance-critical paths
-
-**Symptom**: Unexpected latency spikes in the host-side launch path.
-
-**Cause**: The occupancy APIs (`cudaOccupancyMaxActiveBlocksPerMultiprocessor`, `cudaOccupancyMaxPotentialBlockSize`, etc.) involve driver calls that can take microseconds to milliseconds. Calling them before every kernel launch adds overhead to the launch path.
-
-**Detection**: Profile the host-side code. If occupancy API calls appear in hot paths, they are adding unnecessary latency.
-
-**Fix**: Call occupancy APIs once during initialization (or when the kernel configuration changes) and cache the results. The occupancy for a given kernel + block size + dynamic SMem is deterministic and does not change at runtime.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/compute/occupancy-tuning.md

@@ -1,342 +0,0 @@
----
-title: Occupancy Tuning
-status: draft
-evidence_level: measured
-applies_to_pattern_class:
-- cuda-core
-applies_to_ops:
-- elementwise
-- reduction
-- normalization
-- scan
-requires_sm: '>=3.0'
-requires_features: []
-single_kernel_useful: true
-cuda_version_tested: '12.9'
-driver_version_tested: 570.124.06
-toolchain: nvcc 12.9 + ptxas 12.9
-measured_on: H200-SXM
-source:
-- path: spec
-  anchor: Reference
-artifacts:
-  code: artifacts/experience/hw-probes/occupancy-sweep/occupancy_sweep_probe.cu
-  build: nvcc -arch=sm_90a -O3 -std=c++17 -lineinfo -Xptxas=-v -o occupancy_sweep_probe occupancy_sweep_probe.cu
-  introspection: artifacts/experience/hw-probes/occupancy-sweep/h200_device_static.json
-  profile: ''
-related_apis:
-- cudaOccupancyMaxActiveBlocksPerMultiprocessor
-- cudaOccupancyMaxPotentialBlockSize
-- cudaOccupancyAvailableDynamicSMemPerBlock
-related_skills:
-- register-pressure
-- compiler-hints
-- ilp
-id: skill-occupancy-tuning
-type: skill
-vendor: nvidia
-tags:
-- cuda-cpp
-- register-budgeting
-- shared-memory-optimization
-- tile-scheduling
-- fused-kernel
-applies_to:
-- general
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1087-L1089
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1089
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1093-L1097
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1123-L1124
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1133-L1134
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Architecture Guides/hopper-tuning-guide/cuda_hopper-tuning-guide_index.html.md
-  anchor: L35-L42
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA API References/cuda-runtime-api/cuda_cuda-runtime-api_index.html.md
-  anchor: L3862-L3890
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA API References/cuda-runtime-api/cuda_cuda-runtime-api_index.html.md
-  anchor: L17092-L17121
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA API References/cuda-runtime-api/cuda_cuda-runtime-api_index.html.md
-  anchor: L3828-L3858
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- register-budgeting
-- shared-memory-optimization
-- tile-scheduling
-kernel_types:
-- fused-kernel
-confidence: experimental
-artifact_dir: artifacts/experience/hw-probes/occupancy-sweep
----
-## What
-
-Occupancy tuning is the process of choosing a launch configuration (block size, shared memory allocation, register cap) that maximizes the ratio of active warps to maximum warps per SM. On H200 (sm_90a), each SM can host up to 64 warps (2048 threads). Occupancy is defined as:
-
-```
-occupancy = active_warps_per_SM / max_warps_per_SM
-```
-
-where `max_warps_per_SM = 64` on H200 (same as Ampere, per the Hopper tuning guide). Active warps are determined by three limiting factors:
-
-1. **Register file**: 65,536 32-bit registers per SM. If each thread uses R registers, the maximum number of threads is `65536 / R` (subject to allocation granularity: registers are allocated per warp, rounded up to the nearest 256 registers per warp).
-
-2. **Shared memory**: 233,472 bytes per SM on H200. If each block uses S bytes of shared memory, the maximum number of blocks is `233472 / S` (subject to the shared memory carveout configuration).
-
-3. **Max threads/blocks per SM**: 2048 threads and 32 blocks per SM. Even with zero register and shared memory pressure, occupancy is capped by these hardware limits.
-
-The occupancy for a given kernel and block size is:
-
-```
-numBlocks = min(reg_limit, smem_limit, max_blocks_per_SM)
-active_threads = numBlocks × blockSize
-active_warps = active_threads / 32
-occupancy = active_warps / 64
-```
-
-## Why
-
-Occupancy determines how many warps the warp scheduler can choose from when one warp stalls (on memory, arithmetic dependency, or synchronization). More active warps means better latency hiding. However, the best-practices guide explicitly warns: "Higher occupancy does not always equate to higher performance — there is a point above which additional occupancy does not improve performance."
-
-The measured data confirms this. On H200:
-
-- For a simple vec_add (12 regs/thread), all block sizes from 64 to 1024 achieve 100% occupancy, yet block size 512 is **29% faster** than block size 64. The difference comes from scheduling overhead: more blocks means more block-setup cost and less efficient memory coalescing per block.
-
-- For a register-heavy kernel (56 regs/thread), block size 64 achieves 56.2% occupancy while block size 512 achieves only 50.0% occupancy — yet block size 512 is **10% faster**. The lower-occupancy configuration has fewer blocks to schedule and better per-block memory throughput.
-
-The key insight: **50% occupancy with zero spills often beats 100% occupancy with spills** (see register-pressure skill). Even without spills, the relationship between occupancy and performance is non-monotonic.
-
-## When to use
-
-- **Initial kernel development**: Use `cudaOccupancyMaxPotentialBlockSize` to get a reasonable starting block size. This API returns the block size that maximizes theoretical occupancy, which is a good first approximation.
-
-- **Register-bound kernels**: When `-Xptxas=-v` shows high register usage (e.g., >32 regs/thread on H200), use the occupancy API to understand how register pressure limits active warps. Consider `__launch_bounds__` or `-maxrregcount` to trade registers for occupancy (see compiler-hints skill).
-
-- **Shared-memory-bound kernels**: When a kernel uses significant shared memory per block, use `cudaOccupancyAvailableDynamicSMemPerBlock` to determine how much dynamic shared memory is available while maintaining a target number of blocks per SM.
-
-- **Block size selection**: When choosing between block sizes, always verify occupancy with `cudaOccupancyMaxActiveBlocksPerMultiprocessor` and then benchmark. The occupancy-optimal block size is a starting point, not the final answer.
-
-- **Diagnosing low occupancy**: When a kernel shows unexpectedly low performance, check occupancy first. If occupancy is below 50%, the limiting factor (registers, shared memory, or max threads) tells you which resource to optimize.
-
-## When NOT to use
-
-- **Do not blindly maximize occupancy**: The best-practices guide warns that "improving occupancy from 66 percent to 100 percent generally does not translate to a similar increase in performance." If the kernel is compute-bound with high ILP, lower occupancy with more registers per thread may be faster.
-
-- **Do not sacrifice correctness for occupancy**: Forcing register counts down with `-maxrregcount` or aggressive `__launch_bounds__` can cause register spilling. Spills to local memory have global-memory latency and typically hurt performance far more than reduced occupancy.
-
-- **Do not assume occupancy predicts latency**: The measured data shows that at equal occupancy (100%), block size 512 is 29% faster than block size 64 for a memory-bound kernel. Occupancy is a necessary condition for good performance, not a sufficient one.
-
-- **Do not use occupancy APIs inside performance-critical paths**: The occupancy APIs are host-side queries that involve driver calls. Call them once during initialization, not inside a loop.
-
-## H200 occupancy limits (sm_90a)
-
-From the Hopper tuning guide and `kp_introspect device-static`:
-
-| Resource | Limit |
-|---|---|
-| Max warps per SM | 64 |
-| Max threads per SM | 2048 |
-| Max blocks per SM | 32 |
-| Register file per SM | 65,536 × 32-bit |
-| Max registers per thread | 255 |
-| Shared memory per SM | 233,472 bytes |
-| Max shared memory per block (opt-in) | 232,448 bytes |
-| Default shared memory per block | 49,152 bytes |
-| Register allocation granularity | 256 registers per warp |
-
-### Register-derived occupancy table (H200)
-
-For a kernel using R registers per thread, the maximum occupancy is:
-
-| R (regs/thread) | Max threads/SM | Occupancy | Limiting factor |
-|---|---|---|---|
-| ≤ 32 | 2048 | 100% | Max threads |
-| 33–40 | 1536–1632 | 75–80% | Registers |
-| 41–64 | 1024–1536 | 50–75% | Registers |
-| 65–128 | 512–1008 | 25–50% | Registers |
-| 129–255 | 256–504 | 12.5–25% | Registers |
-
-Note: actual occupancy depends on register allocation granularity (rounded to 256 per warp) and the interaction with block size.
-
-## Block size heuristics
-
-The best-practices guide recommends: "The number of threads per block should be a multiple of 32 threads, because this provides optimal computing efficiency and facilitates coalescing."
-
-Common starting points and their tradeoffs:
-
-| Block size | Typical use | Pros | Cons |
-|---|---|---|---|
-| 64 | Debugging, fine-grained work | Low latency per block | High scheduling overhead, may hit max-blocks limit |
-| 128 | Moderate register pressure | Good balance, fits many blocks/SM | Less ILP opportunity per block |
-| **256** | **General-purpose default** | **Sweet spot for most kernels** | May not maximize occupancy for high-register kernels |
-| 512 | Memory-bound kernels | Low scheduling overhead, good coalescing | Fewer blocks/SM, less flexibility |
-| 1024 | Maximize occupancy for low-reg kernels | Minimal block overhead | Only 2 blocks/SM, poor occupancy if reg-heavy |
-
-**256 is often the sweet spot** because:
-- It is a multiple of the warp size (32), ensuring full warps.
-- It allows 4–8 blocks per SM on H200 (depending on register usage), giving the scheduler flexibility.
-- It balances scheduling overhead against per-block resource consumption.
-- It works well with warp-level primitives (see warp-primitives skill).
-
-## Using the Occupancy API
-
-### cudaOccupancyMaxActiveBlocksPerMultiprocessor
-
-Returns the maximum number of blocks that can reside on an SM for a given kernel and block size:
-
-```cuda
-int numBlocks;
-cudaOccupancyMaxActiveBlocksPerMultiprocessor(
-    &numBlocks, my_kernel, blockSize, dynamicSMemSize);
-
-int activeWarps = numBlocks * blockSize / 32;
-double occupancy = (double)activeWarps / 64.0;  // 64 max warps on H200
-```
-
-### cudaOccupancyMaxPotentialBlockSize
-
-Returns the block size that maximizes occupancy and the minimum grid size needed to achieve that occupancy across the whole device:
-
-```cuda
-int minGridSize, blockSize;
-cudaOccupancyMaxPotentialBlockSize(
-    &minGridSize, &blockSize, my_kernel, dynamicSMemSize, blockSizeLimit);
-```
-
-**Important**: This API optimizes for occupancy, not latency. The measured data shows that the suggested block size (1024 for vec_add) is not always the fastest (512 was 1.7% faster). Treat the suggestion as a starting point.
-
-### cudaOccupancyAvailableDynamicSMemPerBlock
-
-Returns the maximum dynamic shared memory per block that allows a target number of blocks per SM:
-
-```cuda
-size_t dynamicSmemSize;
-cudaOccupancyAvailableDynamicSMemPerBlock(
-    &dynamicSmemSize, my_kernel, numBlocks, blockSize);
-```
-
-This is useful when you want to maximize shared memory usage while maintaining a minimum occupancy level.
-
-## Classical example: occupancy-aware launch configuration
-
-```cuda
-__global__ void my_kernel(const float* __restrict__ in,
-                          float*       __restrict__ out,
-                          int n) {
-    extern __shared__ float smem[];
-    int idx = blockIdx.x * blockDim.x + threadIdx.x;
-    // ... kernel body using smem ...
-}
-
-void launch_my_kernel(const float* in, float* out, int n) {
-    int blockSize, minGridSize;
-
-    // Step 1: Get occupancy-optimal block size
-    cudaOccupancyMaxPotentialBlockSize(
-        &minGridSize, &blockSize, my_kernel, 0, 0);
-
-    // Step 2: Check if we can add dynamic shared memory without
-    //         reducing occupancy
-    int numBlocks;
-    cudaOccupancyMaxActiveBlocksPerMultiprocessor(
-        &numBlocks, my_kernel, blockSize, 0);
-
-    size_t maxDynSmem;
-    cudaOccupancyAvailableDynamicSMemPerBlock(
-        &maxDynSmem, my_kernel, numBlocks, blockSize);
-
-    // Step 3: Use the maximum available dynamic shared memory
-    size_t dynSmem = min(maxDynSmem, (size_t)65536);  // cap at 64KB
-
-    // Step 4: Compute grid size
-    int gridSize = (n + blockSize - 1) / blockSize;
-
-    // Step 5: Opt-in for large shared memory if needed
-    if (dynSmem > 49152) {
-        cudaFuncSetAttribute(my_kernel,
-            cudaFuncAttributeMaxDynamicSharedMemorySize, dynSmem);
-    }
-
-    my_kernel<<<gridSize, blockSize, dynSmem>>>(in, out, n);
-}
-```
-
-## Measured Characteristics
-
-- Occupancy sweep probe: On H200 (sm_90a, CUDA 12.9), sweeping block sizes 64–1024 on two kernels:
-
-  **Simple vec_add (12 regs/thread, memory-bound)**: All block sizes achieve 100% occupancy. Block size 512 is the fastest (0.0057 ms median), 29% faster than block size 64 (0.0080 ms). `cudaOccupancyMaxPotentialBlockSize` suggests 1024, but 512 is 1.7% faster.
-
-  **Register-heavy vec_add_regpress (56 regs/thread)**: Block sizes 64/128 achieve 56.2% occupancy; block sizes 256/512/1024 achieve 50.0% occupancy. Despite lower occupancy, block size 512 (0.0297 ms) is 10% faster than block size 64 (0.0330 ms). This confirms that higher occupancy does not guarantee better performance.
-
-## Health thresholds
-
-These are the numbers commonly used to decide whether a launch configuration is
-"good enough". They are **rules of thumb, not architectural constants** — the
-distinction matters, because two of them are derived and three are not.
-
-### Derived — these follow from the launch geometry
-
-```
-waves = grid_size / (SM_count * blocks_resident_per_SM)
-```
-
-**A wave count below 1 means some SMs never receive a block at all.** Below 2,
-the last (partial) wave is a fixed tail whose cost is not amortised by anything
-— see [tail-effect](../../patterns/tail-effect.md). Making `grid_size` a
-multiple of `SM_count * blocks_per_SM` removes the partial wave entirely; this
-is arithmetic, not a measurement.
-
-`launch__waves_per_multiprocessor` reports this directly.
-
-### Rules of thumb — transcribed, not verified here
-
-| Number | Used as | Status |
-|---|---|---|
-| achieved occupancy >= 50% | below this, latency hiding is usually the first thing to check | ⚠️ **unverified** |
-| waves >= 2 | one wave leaves the tail unamortised | ⚠️ **unverified** as a threshold; the wave arithmetic above is not |
-| active warps per SM >= 16 | enough resident warps for the scheduler to switch on a stall | ⚠️ **unverified**, and explicitly architecture-dependent in the source |
-
-⚠️ *The 50% / 2-wave / 16-warp figures are transcribed from an internal
-guideline document and have not been reproduced on our own hardware. They are
-recorded here as the convention in use, not as measured behaviour. Occupancy in
-particular is a means, not an end: a kernel with enough ILP can saturate the
-machine at well under 50%, and several of the fastest kernels in this wiki do.*
-
-## Shared memory is allocated per block, and that quantises occupancy
-
-The occupancy formula above divides the SM's shared-memory budget by the
-**per-block** request. That division is integer: a block asking for slightly
-more than a third of the budget gets you two resident blocks, not 2.9.
-
-```
-blocks_by_smem = floor(smem_per_SM / smem_per_block)
-```
-
-Two consequences that surprise people:
-
-- **Block size and shared memory interact.** When the request is a function of
-  the block size (`smem = blockSize * sizeof(float)` is the common shape), doubling
-  the block size halves how many blocks fit -- the warp count per SM may not move
-  at all. `cudaOccupancyMaxPotentialBlockSizeVariableSMem` exists for exactly this
-  case; see [apis](occupancy-tuning/apis.md).
-- **Rounding up is free until it is not.** The hardware allocates in fixed
-  granules, so a request just over a granule boundary costs a whole granule. If
-  occupancy drops one block when you added a few bytes of shared memory, this is
-  why.