@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/hardware/foundation/memory-hierarchy.md

@@ -1,214 +0,0 @@
----
-id: hw-foundation-memory-hierarchy
-title: Memory Hierarchy
-type: hardware
-vendor: nvidia
-architectures:
-- sm90
-- sm90a
-languages:
-- '*'
-tags:
-- cuda-cpp
-confidence: source-reported
-evidence_level: spec
-related: []
-sources: []
-aliases: []
----
-# CUDA Memory Hierarchy Reference Card
-
-Quick-reference for kernel optimization decisions. Focus: NVIDIA H200 SXM (Hopper, compute capability sm_90a). The H200 shares the same GH100 compute die as the H100, with upgraded HBM3e memory.
-
----
-
-## 1. Memory Levels Overview
-
-| Level | Location | Capacity (H200 SXM) | Bandwidth | Latency | Scope |
-|-------|----------|---------------------|-----------|---------|-------|
-| Registers | On-chip | 64K x 32-bit per SM (256 KB) | Highest (zero extra cycles typical) | ~1 cycle | Per thread |
-| Shared Memory | On-chip | Up to 228 KB per SM (configurable) | ~19 TB/s aggregate across GPU | ~20-30 cycles | Per thread block |
-| L1 Cache | On-chip (unified with shared mem) | 256 KB total per SM (shared mem + L1) | Same as shared memory | ~30-40 cycles | Per SM |
-| L2 Cache | On-chip | 50 MB | ~12 TB/s | ~200 cycles | Global (all SMs) |
-| HBM (Global Memory) | Off-chip | 141 GB HBM3e | 4.8 TB/s | ~400-600 cycles | Global (all SMs + host) |
-| Constant Memory | Off-chip (cached) | 64 KB | Cached: ~constant cache BW; Uncached: HBM BW | Cache hit: ~few cycles; Miss: HBM latency | Global (read-only) |
-
-**Key insight:** Moving data from HBM to registers crosses ~3 orders of magnitude in latency. Maximizing data reuse in registers and shared memory is the most impactful optimization.
-
----
-
-## 2. Cache Line Sizes
-
-| Cache | Line / Sector Size | Notes |
-|-------|-------------------|-------|
-| L1 | 128-byte cache line | Coalescing unit for global memory through L1 |
-| L2 | 32-byte sector | Global memory transactions are 32-byte aligned sectors |
-
-- Global memory is accessed via 32-byte, 64-byte, or 128-byte transactions, naturally aligned.
-- On compute capability 6.0+, the data access unit is 32 bytes regardless of whether loads are cached in L1.
-- A warp's global memory access coalesces into the minimum number of 32-byte transactions needed to service all threads.
-
----
-
-## 3. Shared Memory Bank Structure
-
-- **32 banks**, each 32 bits (4 bytes) wide.
-- **Bank mapping:** Successive 32-bit words map to successive banks (bank index = (address / 4) % 32).
-- **Bandwidth per bank:** 32 bits per clock cycle.
-- **Max aggregate bandwidth:** 32 banks serviced simultaneously per warp request.
-
-### Bank Conflict Rules
-
-| Access Pattern | Bank Conflicts | Throughput Impact |
-|---------------|---------------|-------------------|
-| 32 threads access 32 distinct banks | None (ideal) | Full bandwidth |
-| N threads access same bank (different words) | N-way conflict | Serialized, 1/N bandwidth |
-| Multiple threads access same word in same bank | Broadcast (no conflict) | Full bandwidth (read); One write wins (write) |
-| Stride of 1 word (4B) | No conflict | Ideal |
-| Stride of 2 words (8B) | 2-way conflict | 1/2 bandwidth |
-| Stride of 32 words (128B) | 32-way conflict (worst case) | 1/32 bandwidth |
-
-**Optimization tip:** Avoid strides that are multiples of 32 words. Padding shared memory arrays by 1 element can break bank conflicts (e.g., `__shared__ float tile[32][33]` instead of `[32][32]`).
-
----
-
-## 4. L1 / Shared Memory Configuration Modes
-
-The L1 data cache and shared memory share a unified on-chip memory block. The split is configurable per kernel.
-
-### H100 (Compute Capability 9.0)
-
-- **Total unified capacity:** 256 KB per SM
-- **Supported shared memory carveouts:** 0, 8, 16, 32, 64, 100, 132, 164, 196, 228 KB
-- **Max shared memory per block:** 227 KB (1 KB reserved for system)
-- **Static shared memory limit:** 48 KB (above requires dynamic shared memory + explicit opt-in)
-
-### Comparison Across Architectures
-
-| Architecture | Compute Cap. | Unified Size | Max Shared Mem/SM | Max Shared Mem/Block |
-|-------------|-------------|-------------|-------------------|---------------------|
-| Volta (V100) | 7.0 | 128 KB | 96 KB | 96 KB |
-| Turing | 7.5 | 96 KB | 64 KB | 64 KB |
-| Ampere (A100) | 8.0 | 192 KB | 164 KB | 163 KB |
-| Hopper (H100) | 9.0 | 256 KB | 228 KB | 227 KB |
-
-### Configuration API
-
-```cpp
-// Set preferred shared memory carveout (hint, not hard requirement)
-cudaFuncSetAttribute(kernel, cudaFuncAttributePreferredSharedMemoryCarveout, carveout);
-
-// Enable large dynamic shared memory (required for > 48 KB per block)
-cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, maxBytes);
-```
-
----
-
-## 5. Memory Coalescing Rules
-
-Global memory coalescing determines how efficiently a warp's memory requests are serviced.
-
-### Fundamental Rules (Compute Capability 6.0+)
-
-1. **Transaction size:** 32-byte aligned segments.
-2. **Coalescing:** A warp's accesses are grouped into the minimum number of 32-byte transactions covering all requested addresses.
-3. **Alignment:** Memory allocated via `cudaMalloc()` is guaranteed aligned to at least 256 bytes.
-4. **Word sizes:** Instructions support reading/writing 1, 2, 4, 8, or 16 bytes. Data must be naturally aligned for single-instruction access.
-
-### Access Pattern Performance Impact
-
-| Pattern | Transactions per Warp (32 threads, 4B each) | Efficiency |
-|---------|---------------------------------------------|------------|
-| Consecutive addresses (stride 1), aligned | 4 x 32B | 100% |
-| Consecutive addresses, misaligned by N bytes | 5 x 32B | ~80% |
-| Stride 2 (every other element) | 8 x 32B | 50% |
-| Stride 4 | 16 x 32B | 25% |
-| Stride 32 (one element per cache line) | 32 x 32B | 3.125% (worst case) |
-| Random / scattered | Up to 32 x 32B | Very low |
-
-### Best Practices for Coalescing
-
-- **Consecutive thread IDs should access consecutive memory addresses** (stride-1 pattern).
-- **Thread block width and array width should be multiples of warp size (32)** for 2D array access.
-- **Use `cudaMallocPitch()` / `cudaMalloc3D()`** for 2D/3D arrays to ensure proper alignment and padding.
-- **Use `__align__(8)` or `__align__(16)`** for struct types in global memory.
-- **Avoid array-of-structures (AoS); prefer structure-of-arrays (SoA)** for coalesced access.
-- **For strided access patterns, stage data through shared memory** to convert strided global reads into coalesced accesses.
-
----
-
-## 6. Global Memory Access Patterns and Performance
-
-### Caching Behavior
-
-| Memory Type | L1 Cached | L2 Cached | Notes |
-|-------------|-----------|-----------|-------|
-| Global | Yes (CC 6.0+, default) | Yes | Read-only data may use `__ldg()` for unified L1/texture cache |
-| Local | Yes | Yes | Used for register spills; coalesced across threads |
-| Constant | Constant cache | n/a | Broadcast to all threads if same address; serialized if different addresses |
-| Texture | Texture cache (unified with L1) | Yes | Optimized for 2D spatial locality |
-
-### L2 Cache Persistence Control (CC 8.0+)
-
-```cpp
-// Set aside L2 cache for persisting accesses
-cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, size);
-
-// Mark access window as persisting via stream attributes
-stream_attribute.accessPolicyWindow.hitProp = cudaAccessPropertyPersisting;
-```
-
-- H200 L2 cache: 50 MB total (same as H100; vs. A100's 40 MB).
-- Persisting accesses have priority for the set-aside L2 portion.
-- Tune `hitRatio` to avoid cache thrashing when persistent data exceeds set-aside size.
-
-### Local Memory
-
-- Resides in device memory (HBM) -- same latency/bandwidth as global memory.
-- Used for: register spills, large arrays with non-constant indices, large structures.
-- Coalesced automatically: consecutive 32-bit words accessed by consecutive thread IDs.
-- Cached in L1 and L2 (CC 6.0+).
-
----
-
-## 7. H100 Special Memory Features
-
-### Tensor Memory Accelerator (TMA)
-
-- Hardware unit for asynchronous transfer of 1D-5D tensors between global and shared memory.
-- Single-threaded programming model: one thread issues the TMA copy, all threads wait via `cuda::barrier`.
-- Avoids register usage for address generation; frees threads for computation.
-- Supports both global-to-shared and shared-to-global directions, plus reduction operations (add/min/max/and/or).
-
-### Distributed Shared Memory (DSMEM)
-
-- Available with Thread Block Clusters (CC 9.0).
-- All thread blocks in a cluster can directly load/store/atomic to each other's shared memory.
-- Total DSMEM = (blocks per cluster) x (shared memory per block).
-- ~7x faster than routing through global memory for inter-block data exchange.
-- Access should be coalesced and aligned to 32-byte segments for best performance.
-
-### Inline Compression (ILC)
-
-- Automatic hardware compression for global memory transfers.
-- Can increase effective bandwidth beyond raw HBM bandwidth.
-- Does not reduce memory footprint (allocation size unchanged).
-- Enabled per allocation via CUDA driver API (`CU_MEM_ALLOCATION_COMP_GENERIC`).
-
----
-
-## 8. Quick Decision Guide
-
-| Scenario | Recommended Memory | Why |
-|----------|-------------------|-----|
-| Thread-private accumulators, loop variables | Registers | Zero-latency access |
-| Tile of data reused across threads in a block | Shared memory | ~4x the bandwidth, ~15-30x lower latency than HBM (see the table above -- the often-quoted "~100x" matches neither) |
-| Read-only data with spatial locality | Texture / `__ldg()` | Texture cache optimized for 2D |
-| Frequently reused global data | L2 persistence hints | Keep hot data in L2 |
-| Data shared across blocks (within cluster) | DSMEM | Avoid global memory round-trip |
-| Large tensor loads into shared memory | TMA | Frees threads, hardware-managed |
-| Broadcast same value to all threads | Constant memory | Single read, broadcast to warp |
-
----
-
-*Sources: CUDA C++ Programming Guide 13.2, CUDA C++ Best Practices Guide 13.2, Hopper Tuning Guide 13.2, NVIDIA H100 Tensor Core Hopper Architecture Whitepaper.*

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/hardware/foundation/tensor-core-specs.md

@@ -1,211 +0,0 @@
----
-id: hw-foundation-tensor-core-specs
-title: Tensor Core Specs
-type: hardware
-vendor: nvidia
-architectures:
-- sm90
-- sm90a
-languages:
-- '*'
-tags:
-- cuda-cpp
-confidence: source-reported
-evidence_level: spec
-related: []
-sources: []
-aliases: []
----
-# Hopper (sm_90) Tensor Core Reference Card
-
-Quick-reference for tensor core instruction selection and kernel optimization.
-Covers 4th-generation Tensor Cores in H100/H200.
-
-## Tensor Core Overview
-
-- 4 Tensor Cores per SM, 528 total on H100/H200 SXM
-- Each Tensor Core performs matrix multiply-accumulate (MMA): D = A * B + C
-- 2x per-SM throughput vs A100 on all data types (clock-for-clock)
-- 4x per-SM throughput vs A100 on FP8 (new in Hopper)
-- Structured sparsity (2:4) doubles effective throughput
-
-## Instruction Hierarchy: Three Abstraction Levels
-
-### 1. WMMA (C++ API) -- Highest level, most portable
-
-| Aspect | Details |
-|---|---|
-| API | `nvcuda::wmma` namespace (C++ templates) |
-| Scope | Warp-level (32 threads) |
-| Data handling | `load_matrix_sync` / `store_matrix_sync` manage layout implicitly |
-| Fragment layout | Opaque, architecture-specific -- cannot assume mapping |
-| Portability | Works across sm_70+ (Volta through Hopper) |
-| Limitation | Fixed tile shapes, no explicit register control, no FP8 support via WMMA |
-| Best for | Prototyping, portable code that does not need peak performance |
-
-**WMMA Supported Shapes and Types:**
-
-| Input A/B | Accumulator | Tile Shapes (m-n-k) |
-|---|---|---|
-| FP16 (__half) | FP32 or FP16 | 16x16x16, 32x8x16, 8x32x16 |
-| BF16 (__nv_bfloat16) | FP32 | 16x16x16, 32x8x16, 8x32x16 |
-| TF32 (precision::tf32) | FP32 | 16x16x8 |
-| INT8 (u8/s8) | INT32 | 16x16x16, 32x8x16, 8x32x16 |
-| FP64 (double) | FP64 | 8x8x4 |
-
-### 2. MMA (PTX) -- Mid level, explicit register layout
-
-| Aspect | Details |
-|---|---|
-| Instruction | `mma.sync.aligned` (PTX) |
-| Scope | Warp-level (32 threads) |
-| Data handling | Programmer distributes matrix elements across thread registers explicitly |
-| Shapes | Asymmetric m16n8kX family (primary) plus legacy m8n8kX |
-| Sparsity | Supports structured sparse matrix A (2:4 pattern) |
-| Best for | Hand-tuned PTX kernels, libraries like CUTLASS |
-
-**MMA (PTX) Supported Shapes for Hopper (sm_90):**
-
-| Data Type | Dense Shapes | Sparse Shapes |
-|---|---|---|
-| FP64 | m8n8k4, m16n8k4, m16n8k8, m16n8k16 | -- |
-| FP16 | m8n8k4, m16n8k8, m16n8k16 | m16n8k16, m16n8k32 |
-| BF16 | m16n8k8, m16n8k16 | m16n8k16, m16n8k32 |
-| TF32 | m16n8k4, m16n8k8 | m16n8k8, m16n8k16 |
-| INT8 (u8/s8) | m8n8k16, m16n8k16, m16n8k32 | m16n8k32, m16n8k64 |
-| FP8 (e4m3/e5m2) | m16n8k32, m16n8k16 | m16n8k64 |
-| INT4 (u4/s4) | m8n8k32, m16n8k32, m16n8k64 | m16n8k64, m16n8k128 |
-
-**MMA Data Type Combinations:**
-
-| Input (A, B) | Accumulator (C, D) |
-|---|---|
-| .f16 | .f16 or .f32 |
-| .bf16 | .f32 |
-| .tf32 | .f32 |
-| .f64 | .f64 |
-| .e4m3 / .e5m2 | .f16 or .f32 |
-| .u8 / .s8 | .s32 |
-| .u4 / .s4 | .s32 |
-
-### 3. WGMMA (PTX) -- Lowest level, Hopper-only, highest throughput
-
-| Aspect | Details |
-|---|---|
-| Instruction | `wgmma.mma_async` (PTX) |
-| Scope | **Warpgroup-level** (4 warps = 128 threads) |
-| Execution | Asynchronous -- results via `wgmma.commit_group` / `wgmma.wait_group` |
-| Matrix A source | Registers **or** shared memory (via descriptor) |
-| Matrix B source | Shared memory only (via descriptor) |
-| Requires | `wgmma.fence` before issuing; `fence.proxy.async` for visibility |
-| Best for | Peak-throughput GEMM, CUTLASS 3.x, FlashAttention-style kernels |
-
-**WGMMA Supported Shapes (M is always 64):**
-
-| Data Type | K | N range | Shape Pattern |
-|---|---|---|---|
-| FP16 (dense) | 16 | 8 to 256, step 8 | m64n{8..256}k16 |
-| BF16 (dense) | 16 | 8 to 256, step 8 | m64n{8..256}k16 |
-| TF32 (dense) | 8 | 8 to 256, step 8 | m64n{8..256}k8 |
-| TF32 (sparse) | 16 | 8 to 256, step 8 | m64n{8..256}k16 |
-| FP8 e4m3/e5m2 (dense) | 32 | 8 to 256, step 8 | m64n{8..256}k32 |
-| INT8 u8/s8 (dense) | 32 | 8 to 256, step 16 | m64n{8..256}k32 |
-| FP16 (sparse) | 32 | 8 to 256, step 8 | m64n{8..256}k32 |
-| BF16 (sparse) | 32 | 8 to 256, step 8 | m64n{8..256}k32 |
-| FP8 (sparse) | 64 | 8 to 256, step 8 | m64n{8..256}k64 |
-| INT8 (sparse) | 64 | 8 to 256, step 16 | m64n{8..256}k64 |
-
-**WGMMA Data Type Combinations:**
-
-| Input (A, B) | Accumulator (D) |
-|---|---|
-| .f16 | .f16 or .f32 |
-| .bf16 | .f32 |
-| .tf32 | .f32 |
-| .e4m3 / .e5m2 | .f16 or .f32 |
-| .u8 / .s8 | .s32 |
-| .b1 | .s32 |
-
-## Per-SM Throughput Per Cycle
-
-Each SM has 4 Tensor Cores. At sm_90 boost clock (1830 MHz), throughput per SM per cycle:
-
-| Precision | Dense (FLOPs/cycle/SM) | With Sparsity |
-|---|---|---|
-| FP64 TC | 128 | -- |
-| TF32 | 512 | 1024 |
-| FP16 / BF16 | 1024 | 2048 |
-| FP8 (e4m3/e5m2) | 2048 | 4096 |
-| INT8 | 2048 | 4096 |
-
-Derivation: Peak TFLOPS / (num_SMs * boost_clock). Example: FP16 TC dense = 989.4e12 / (132 * 1.83e9) = 4096 ops/cycle/SM for the full chip, which is 1024 FMA-equivalent FLOPs/cycle/SM (each FMA = 2 ops).
-
-Note: The values in the table above represent FMA operations per cycle per SM (each FMA counts as 2 floating-point operations). To get peak TFLOPS, use: TFLOPS = ops_per_cycle_per_SM * 2 * SM_count * clock_GHz.
-
-## Data Layout Requirements
-
-### MMA (mma.sync) Register Mapping
-
-For the most commonly used `m16n8k16` FP16 shape:
-- **32 threads** in a warp cooperatively hold the fragments
-- **Matrix A (16x16):** Each thread holds 8 FP16 elements (4 registers of .b32, each packing 2xFP16)
-- **Matrix B (16x8, transposed as 8x16):** Each thread holds 4 FP16 elements (2 registers of .b32)
-- **Matrix C/D (16x8):** Each thread holds 4 FP32 elements (4 registers of .f32)
-- Thread-to-element mapping is defined by the PTX ISA (grouping of 4 threads covers 8 rows)
-
-### WGMMA Register/Shared Memory Layout
-
-For `wgmma.mma_async.m64nNk16` (FP16/BF16):
-- **Matrix A:** Can be in registers or shared memory
-  - In registers: distributed across 128 threads (4 warps). Each thread holds elements from a 64xK tile.
-  - In shared memory: described by a matrix descriptor (swizzle pattern configured via descriptor)
-- **Matrix B:** Always in shared memory, accessed via descriptor
-- **Matrix D (accumulator):** In registers, distributed across 128 threads of the warpgroup
-- Shared memory layout for descriptors: 128-byte aligned, uses swizzle modes (128B, 64B, 32B, or none)
-
-### Key Layout Constraints
-
-- Shared memory pointers for WGMMA descriptors must be 16-byte aligned
-- WMMA `load_matrix_sync` requires 256-bit (32-byte) aligned pointers
-- Leading dimension (ldm) must be a multiple of 16 bytes
-- Column-major A and row-major B is the natural layout for Tensor Cores
-- WGMMA supports transpose on .f16/.bf16 inputs only (via imm-trans-a/imm-trans-b flags)
-
-## Choosing the Right Abstraction
-
-| Criterion | WMMA | MMA (PTX) | WGMMA (PTX) |
-|---|---|---|---|
-| Portability | sm_70+ | sm_75+ | sm_90 only |
-| FP8 support | No | Yes | Yes |
-| Sparsity support | No | Yes | Yes |
-| Async execution | No | No | Yes |
-| Tile size control | Fixed small tiles | Fixed small tiles | Large m64 tiles, flexible N |
-| Integration with TMA | Manual | Manual | Native (descriptor-based) |
-| Peak throughput | ~60-80% | ~80-95% | ~95-100% |
-| Used by CUTLASS | 2.x (legacy) | 2.x | 3.x |
-
-## Sources
-
-- NVIDIA H100 Tensor Core GPU Architecture Whitepaper
-- PTX ISA Reference (CUDA Toolkit 13.2), Sections 9.7.14-9.7.15
-- CUDA C++ Programming Guide (CUDA Toolkit 13.2), Section 10.24
-
-## Tensor Core vs CUDA Core throughput
-
-How much a tensor core buys you is a division of two rows already in
-[h200-specs](h200-specs.md) -- there is no separate measurement, and
-the answer depends entirely on which pair you divide.
-
-| Comparison (H200, dense) | CUDA core | Tensor core | Factor |
-|---|---|---|---|
-| FP16 vs FP16 | 133.8 TFLOPS | 989.4 TFLOPS | **7.4x** |
-| FP32 (the usual baseline) vs FP16 TC | 66.9 TFLOPS | 989.4 TFLOPS | **14.8x** |
-| FP32 vs FP8 TC | 66.9 TFLOPS | 1,978.9 TFLOPS | **29.6x** |
-
-So the common phrasing "tensor cores are several to several-tens of times
-faster" is right as a range but useless as a target: **state which two rows you
-divided.** A GEMM that fails to reach the tensor path is not losing "10x", it is
-losing whichever of these applies to its dtype.
-
-⚠️ Doubling again with 2:4 structured sparsity -- see the sparsity section of
-[h200-specs](h200-specs.md).

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/hardware/foundation/warp-execution-model.md

@@ -1,252 +0,0 @@
----
-id: hw-foundation-warp-execution-model
-title: Warp Execution Model
-type: hardware
-vendor: nvidia
-architectures:
-- sm90
-- sm90a
-languages:
-- '*'
-tags:
-- cuda-cpp
-confidence: source-reported
-evidence_level: spec
-related: []
-sources: []
-aliases: []
----
-# CUDA Warp Execution Model Reference Card
-
-Quick-reference for kernel optimization decisions. Focus: NVIDIA H200 SXM (Hopper, compute capability sm_90a). The H200 shares the same GH100 compute die as the H100, with upgraded HBM3e memory (141 GB, 4.8 TB/s).
-
----
-
-## 1. Warp Fundamentals
-
-| Parameter | Value |
-|-----------|-------|
-| Warp size | 32 threads |
-| Max warps per SM | 64 |
-| Max threads per SM | 2048 |
-| Max thread blocks per SM | 32 |
-| Max threads per block | 1024 |
-
-- A warp is the fundamental scheduling and execution unit. All 32 threads execute the same instruction simultaneously (SIMT model).
-- Thread blocks are partitioned into warps by consecutive thread IDs: warp 0 = threads [0..31], warp 1 = threads [32..63], etc.
-- Total warps per block = ceil(threadsPerBlock / 32). Partial warps waste compute resources.
-- **Always choose block sizes that are multiples of 32** to avoid under-populated warps.
-
----
-
-## 2. Warp Scheduler Behavior
-
-### H100 (Compute Capability 9.0)
-
-- **4 warp schedulers per SM** (same as Volta, Ampere).
-- Warps are **statically distributed** among the 4 schedulers at block launch time.
-- Each scheduler can issue **1 instruction per cycle** from one of its assigned warps.
-- At every issue cycle, each scheduler picks a warp that has a ready instruction (no dependencies, not stalled).
-
-### Latency Hiding Requirement
-
-| Latency Source | Typical Latency | Warps Needed to Hide (per SM) |
-|---------------|----------------|-------------------------------|
-| Arithmetic (register-to-register) | ~4 cycles | 16 warps (4 schedulers x 4 cycles) |
-| Shared memory (no bank conflict) | ~20-30 cycles | 20-30 warps (or ILP within warps) |
-| Global memory (L2 hit) | ~200 cycles | Many warps + ILP |
-| Global memory (HBM) | ~400-600 cycles | Requires high occupancy + ILP |
-
-**Key formula:** To fully hide latency L with throughput T (instructions/cycle), need L x T active warps.
-- For 4 schedulers issuing 1 instr/cycle each: need 4L warps for latency L.
-- With instruction-level parallelism (ILP), fewer warps suffice because multiple independent instructions from one warp can issue back-to-back.
-
-### Independent Thread Scheduling (Volta+)
-
-- Since Volta (CC 7.0), each thread maintains its own program counter and call stack.
-- Threads within a warp can diverge and reconverge at sub-warp granularity.
-- A schedule optimizer groups active threads into SIMT units.
-- **Implication:** Do not assume warp-synchronous execution. Use explicit synchronization (`__syncwarp()`, `__syncthreads()`).
-
----
-
-## 3. Warp Divergence
-
-- When threads in a warp take different branch paths, the warp executes each path serially, disabling threads not on the active path.
-- **Branch divergence occurs only within a warp.** Different warps are completely independent.
-- If N threads take path A and (32-N) take path B, execution time = time(A) + time(B), not max(time(A), time(B)).
-
-### Optimization Tips
-
-- Structure conditions to align with warp boundaries: `if (threadIdx.x / 32 < threshold)` causes no divergence; `if (threadIdx.x % 2 == 0)` causes 100% divergence.
-- Minimize divergent code paths; keep both branches short when divergence is unavoidable.
-- Consider warp-level voting (`__ballot_sync`, `__any_sync`, `__all_sync`) to reorganize work.
-
----
-
-## 4. Warp Group Concept (Hopper-Specific)
-
-H100 introduces **warp groups** for cooperative matrix operations:
-
-| Parameter | Value |
-|-----------|-------|
-| Warp group size | 4 warps (128 threads) |
-| Purpose | Collaborative MMA (matrix multiply-accumulate) execution |
-
-- **Warp groups** enable MMA instructions that act on large matrices spanning 4 warps simultaneously.
-- Dynamic register capacity can be reassigned among warp groups to support larger matrix tiles.
-- Operand matrices can be accessed directly from shared memory (avoiding register staging).
-- This feature is exposed via inline PTX.
-
-### Warp Specialization
-
-- Hopper enables **warp specialization** within a thread block: some warps specialize in data movement (using TMA) while others focus on computation.
-- This decouples the data-feeding pipeline from the compute pipeline, improving overlapping.
-
----
-
-## 5. Register File
-
-### Specifications (H100)
-
-| Parameter | Value |
-|-----------|-------|
-| Register file size per SM | 65,536 x 32-bit registers (256 KB) |
-| Max registers per thread | 255 |
-| Max registers per block | 65,536 |
-| Register width | 32 bits (a `double` or `long long` uses 2 registers) |
-| Allocation granularity | 256 registers per warp (rounded up) |
-
-### Register Allocation Impact on Occupancy
-
-Registers are allocated per warp (not per thread). The total registers for a block = warps_per_block x registers_per_warp.
-
-**Example calculation (H100):**
-- Kernel uses 64 registers/thread, block size = 512 threads (16 warps).
-- Registers per warp = 64 x 32 = 2048, but allocated in multiples of 256, so 2048 (already aligned).
-- Registers per block = 16 x 2048 = 32,768.
-- Blocks per SM = floor(65,536 / 32,768) = 2 blocks = 32 warps = 50% occupancy.
-- Using 65 registers/thread: per warp = 65 x 32 = 2080, rounded up to 2304 (9 x 256).
-- Registers per block = 16 x 2304 = 36,864.
-- Blocks per SM = floor(65,536 / 36,864) = 1 block = 16 warps = 25% occupancy.
-
-**One extra register per thread can halve occupancy.** Use `--ptxas-options=-v` to check register usage and `-maxrregcount=N` or `__launch_bounds__()` to control it.
-
-### Controlling Register Usage
-
-```cpp
-// Per-kernel hint: max threads per block, min blocks per SM
-__global__ void __launch_bounds__(256, 4) kernel(...) { ... }
-
-// Compilation flag: cap registers at N per thread for the whole file
-// nvcc -maxrregcount=32
-
-// Per-kernel max register qualifier (PTX-level)
-__global__ void __maxnreg__(32) kernel(...) { ... }
-```
-
----
-
-## 6. Occupancy
-
-**Occupancy = active warps per SM / max warps per SM (64).**
-
-It is NOT always true that higher occupancy yields better performance, but low occupancy (<25%) typically cannot hide memory latency.
-
-### Limiting Factors
-
-| Resource | H100 Limit | How It Limits Occupancy |
-|----------|-----------|------------------------|
-| Registers per SM | 65,536 | More regs/thread -> fewer warps fit |
-| Shared memory per SM | Up to 228 KB | More smem/block -> fewer blocks fit |
-| Max blocks per SM | 32 | Even tiny blocks cap at 32 |
-| Max warps per SM | 64 (2048 threads) | Hard ceiling |
-
-### Occupancy Calculation Steps
-
-1. **Register limit:** max_blocks_by_regs = floor(65536 / (regs_per_thread x threads_per_block, rounded to allocation granularity))
-2. **Shared memory limit:** max_blocks_by_smem = floor(available_smem / smem_per_block)
-3. **Block count limit:** max_blocks_by_hw = 32
-4. **Effective blocks per SM:** min(max_blocks_by_regs, max_blocks_by_smem, max_blocks_by_hw)
-5. **Occupancy:** (effective_blocks x warps_per_block) / 64
-
-### Occupancy Calculator API
-
-```cpp
-int numBlocks;
-cudaOccupancyMaxActiveBlocksPerMultiprocessor(&numBlocks, kernel, blockSize, dynamicSmemSize);
-
-int blockSize, minGridSize;
-cudaOccupancyMaxPotentialBlockSize(&minGridSize, &blockSize, kernel, 0, 0);
-
-// For cluster-based kernels (Hopper)
-int maxActiveClusters;
-cudaOccupancyMaxActiveClusters(&maxActiveClusters, kernel, &launchConfig);
-```
-
-### Practical Guidelines
-
-- **128-256 threads per block** is a good starting range for experimentation.
-- Use several smaller blocks rather than one large block per SM if `__syncthreads()` is frequent.
-- Higher occupancy does not always help: with enough ILP, 50% occupancy can match 100%.
-- Low occupancy kernels have more registers/thread available, reducing spills to local memory.
-
----
-
-## 7. Warp Stall Reasons
-
-When profiling with Nsight Compute, warps report stall reasons. Categories relevant to optimization:
-
-| Stall Reason | Description | Optimization Strategy |
-|-------------|-------------|----------------------|
-| **Memory Dependency (Long Scoreboard)** | Waiting for global/local memory load to complete | Increase occupancy; prefetch data; use shared memory; improve coalescing |
-| **Memory Dependency (Short Scoreboard)** | Waiting for shared memory or L1 cache operation | Reduce bank conflicts; reduce shared memory pressure |
-| **Execution Dependency** | Waiting for result of prior arithmetic instruction (~4 cycles) | Increase ILP; reorder independent instructions |
-| **Synchronization (Barrier)** | Waiting at `__syncthreads()` or `__syncwarp()` | Reduce work imbalance between warps in block; use multiple blocks per SM |
-| **Memory Throttle** | Memory subsystem is saturated | Reduce memory requests; improve data reuse |
-| **Not Selected** | Warp is ready but scheduler picked another warp | Generally okay -- indicates good latency hiding |
-| **Instruction Fetch** | Waiting for instruction cache | Reduce code size; improve locality of code execution |
-| **Texture** | Waiting for texture memory fetch | Optimize texture access patterns |
-| **Sleeping** | Warp explicitly put to sleep (Hopper barrier wait) | Expected behavior with async barriers |
-| **Dispatch Stall** | Instruction dispatch pipeline is full | Reduce instruction mix pressure |
-| **Drain** | Warp completing final instructions before exit | Normal at end of kernel |
-| **IMC Miss** | Instruction cache miss | Keep hot loops compact |
-
-### Priority of Optimization
-
-1. **Long scoreboard stalls (global memory):** Usually the dominant bottleneck. Fix via coalescing, caching, shared memory tiling.
-2. **Barrier stalls:** Indicate load imbalance or over-synchronization. Consider async patterns, reduce sync points.
-3. **Short scoreboard stalls (shared memory):** Fix bank conflicts, reduce smem pressure.
-4. **Execution dependency:** Increase ILP or accept as inherent to the algorithm.
-
----
-
-## 8. H200/H100 SM Architecture Summary
-
-| Component | Count per SM |
-|-----------|-------------|
-| FP32 CUDA Cores | 128 (2x A100) |
-| FP64 CUDA Cores | 64 |
-| INT32 Cores | 64 |
-| Tensor Cores (4th gen) | 4 |
-| Warp Schedulers | 4 |
-| Special Function Units (SFU) | 16 |
-| Register File | 65,536 x 32-bit |
-| Unified L1/Shared Memory | 256 KB |
-| Constant Cache | Read-only, shared by all functional units |
-
-### H200 SXM GPU-Level Numbers
-
-| Parameter | Value |
-|-----------|-------|
-| Total SMs | 132 |
-| Total FP32 Cores | 16,896 |
-| Total Tensor Cores | 528 |
-| L2 Cache | 50 MB |
-| HBM3e | 141 GB, 4.8 TB/s |
-| NVLink (4th gen) | 900 GB/s bidirectional (18 links) |
-| Thread Block Cluster size | Up to 8 (portable), up to 16 (non-portable) |
-
----
-
-*Sources: CUDA C++ Programming Guide 13.2 (SIMT Architecture, Compute Capabilities), CUDA C++ Best Practices Guide 13.2, Hopper Tuning Guide 13.2, NVIDIA H100 Tensor Core Hopper Architecture Whitepaper.*