@kernaut/kernaut-resources 0.1.17-beta.1 → 0.1.18

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/memory/coalescing.md

@@ -1,217 +0,0 @@
----
-title: Global Memory Coalescing
-status: draft
-evidence_level: measured
-applies_to_pattern_class:
-- cuda-core
-applies_to_ops:
-- elementwise
-- reduction
-- normalization
-- scan
-- gather
-- scatter
-requires_sm: '>=6.0'
-requires_features: []
-single_kernel_useful: true
-cuda_version_tested: '12.9'
-driver_version_tested: 570.124.06
-toolchain: nvcc 12.9 + ptxas 12.9
-measured_on: H200-SXM
-source:
-- path: spec
-  anchor: Reference
-artifacts:
-  code: artifacts/experience/hw-probes/coalescing-stride/coalescing_probe.cu
-  build: nvcc -arch=sm_90a -O3 -std=c++17 -lineinfo -o coalescing_probe coalescing_probe.cu
-  introspection: ''
-  profile: ''
-related_apis:
-- __ldg
-- float4
-- float2
-- __restrict__
-related_skills:
-- vectorized-access
-- bank-conflict
-- warp-primitives
-id: skill-coalescing
-type: skill
-vendor: nvidia
-tags:
-- cuda-cpp
-- vectorized-loads
-- cache-policy
-- shared-memory-optimization
-- fused-kernel
-- attention
-applies_to:
-- general
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L1379-L1411
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L539-L542
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L545-L567
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L606-L634
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L1446-L1448
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L22350
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- vectorized-loads
-- cache-policy
-- shared-memory-optimization
-kernel_types:
-- fused-kernel
-- attention
-confidence: experimental
-artifact_dir: artifacts/experience/hw-probes/coalescing-stride
----
-## What
-
-Global memory coalescing is the hardware mechanism by which a warp's individual per-thread memory requests are combined into the minimum number of memory transactions. On compute capability 6.0 and above (including H200 / sm_90a), global memory is accessed via **32-byte transactions**. When consecutive threads in a warp access consecutive 4-byte words in a 32-byte aligned region, the warp's 32 requests (128 bytes total) are serviced by exactly **four 32-byte transactions** -- this is a perfectly coalesced access.
-
-The coalescing unit operates at warp granularity. The hardware inspects the addresses requested by all 32 threads in the warp, computes which 32-byte segments are touched, and issues one transaction per segment. The key insight: it does not matter which thread accesses which word within a segment -- what matters is how many distinct segments the warp touches.
-
-**Transaction math** (for 4-byte words, compute capability >= 6.0):
-- Coalesced (stride-1): 32 threads x 4 bytes = 128 bytes, spanning 4 segments. 4 transactions, 100% utilization.
-- Stride-2: 32 threads touch 8 segments. 8 transactions, 50% utilization.
-- Stride-32: 32 threads touch 32 segments. 32 transactions, 12.5% utilization -- the pathological worst case.
-
-On devices with L1 caching enabled (which is the default on sm_60+), the data access unit is 32 bytes regardless of whether global loads are cached in L1 or not (best-practices guide L549).
-
-## Why
-
-Memory coalescing is the single most impactful optimization for memory-bound CUDA-core kernels. The programming guide (L1411) states: "Ensuring proper coalescing of global memory accesses is one of the most important performance considerations for writing performant CUDA kernels."
-
-On H200, our microbenchmark probe measured a **2.85x slowdown** for stride-32 access compared to stride-1 access (see Measured Characteristics below). This means that a kernel with non-coalesced access is leaving approximately 65% of its potential memory bandwidth on the table.
-
-The cost of non-coalesced access arises from two effects:
-1. **Wasted bandwidth**: each 32-byte transaction fetches a full sector, but only a fraction of the bytes are used.
-2. **Increased transaction count**: more transactions compete for the memory subsystem's finite transaction throughput.
-
-## When to use
-
-Coalescing awareness should be applied to **every** kernel that accesses global memory. Specific scenarios where explicit attention is needed:
-
-- **Elementwise kernels** (vecadd, activation, etc.): ensure thread indexing is `tid = blockIdx.x * blockDim.x + threadIdx.x` and array access is `a[tid]` (stride-1).
-
-- **Reduction kernels**: the input-load phase should be coalesced. Even when the reduction itself uses shared memory or warp shuffles, the initial global-to-register load pattern matters.
-
-- **Matrix operations with 2D thread blocks**: when `threadIdx.x` is the fastest-moving dimension, ensure it maps to the contiguous dimension of the matrix (the column index in row-major storage).
-
-- **SoA vs AoS layout**: for data with multiple fields per element (e.g., RGB pixels, 3D coordinates), use Structure-of-Arrays layout so that all threads read the same field contiguously, rather than Array-of-Structures where consecutive threads stride across multiple fields.
-
-- **Vectorized loads** (float4, float2): when each thread processes multiple elements, casting the pointer to `float4*` and loading one `float4` per thread generates a single 128-bit (16-byte) load instruction. A warp of 32 threads loading `float4` values contiguously issues 32 x 16 = 512 bytes in perfectly coalesced 128-bit transactions. This is the "next level" of coalescing -- see the vectorized-access skill.
-
-- **Read-only data**: marking input pointers with `const ... __restrict__` or using `__ldg()` routes loads through the read-only data cache (L1/Tex cache), which can improve effective bandwidth for data read by many warps.
-
-## When NOT to use
-
-Coalescing is an inherent hardware behavior, not an optional optimization. However, there are situations where achieving perfect coalescing is not straightforward or not the primary concern:
-
-- **Gather/scatter patterns**: when the access pattern is determined by an index array (indirect addressing), coalescing depends on the index distribution. If the indices are random, coalescing is poor and no simple code change fixes it -- the algorithmic design must change (e.g., sorting by key, binning).
-
-- **Matrix transpose**: writing the transposed output inherently requires non-coalesced writes (the write index strides by the leading dimension). The standard fix is to use shared memory as a staging buffer: load a tile with coalesced reads, transpose in shared memory, then write with coalesced writes. This is documented in the programming guide at L1413-L1448.
-
-- **Small data per thread**: if each thread accesses only 1 byte and the warp accesses 32 bytes total, one 32-byte transaction suffices regardless of layout. Coalescing only matters when the total warp footprint exceeds one segment.
-
-## Classical example: coalesced vs non-coalesced copy
-
-The following pair of kernels demonstrates the difference between coalesced and non-coalesced access (adapted from the best-practices guide L584-L617):
-
-```cuda
-// GOOD: coalesced -- consecutive threads read consecutive elements
-__global__ void coalescedCopy(float* odata, const float* idata, int n) {
-    int tid = blockIdx.x * blockDim.x + threadIdx.x;
-    if (tid < n)
-        odata[tid] = idata[tid];
-}
-
-// BAD: strided -- consecutive threads read elements stride apart
-__global__ void stridedCopy(float* odata, const float* idata,
-                            int stride, int n) {
-    int tid = blockIdx.x * blockDim.x + threadIdx.x;
-    int idx = tid * stride;
-    if (idx < n)
-        odata[idx] = idata[idx];
-}
-```
-
-With `stride = 32`, the strided kernel touches 32 distinct 32-byte segments per warp, fetching 1024 bytes but using only 128 bytes -- a 12.5% memory utilization rate.
-
-## Shared memory staging for transpose
-
-When an algorithm requires non-coalesced writes (such as matrix transpose), the canonical solution is to use shared memory as a coalescing buffer:
-
-```cuda
-__global__ void transpose_coalesced(float*       odata,
-                                    const float* idata,
-                                    int width) {
-    __shared__ float tile[32][33];  // +1 padding avoids bank conflicts
-    int xIdx = blockIdx.x * 32 + threadIdx.x;
-    int yIdx = blockIdx.y * 32 + threadIdx.y;
-
-    // Coalesced read from input (threadIdx.x is contiguous dimension)
-    if (xIdx < width && yIdx < width)
-        tile[threadIdx.y][threadIdx.x] = idata[yIdx * width + xIdx];
-    __syncthreads();
-
-    // Coalesced write to output (threadIdx.x maps to contiguous output)
-    xIdx = blockIdx.y * 32 + threadIdx.x;
-    yIdx = blockIdx.x * 32 + threadIdx.y;
-    if (xIdx < width && yIdx < width)
-        odata[yIdx * width + xIdx] = tile[threadIdx.x][threadIdx.y];
-}
-```
-
-The read of `idata` is coalesced because `threadIdx.x` maps to the column index. The write of `odata` is also coalesced because, after the shared memory transpose, `threadIdx.x` again maps to the contiguous output dimension. The `[32][33]` padding avoids shared-memory bank conflicts (see the bank-conflict skill).
-
-## Measured Characteristics
-
-- coalescing-stride bandwidth probe: On H200 (sm_90a, CUDA 12.9), stride-1 (coalesced) access achieved an effective bandwidth of 530.66 GB/s (median latency 0.0079 ms), while stride-32 (non-coalesced) access achieved only 186.18 GB/s (median latency 0.0225 ms). The strided access was **2.85x slower** than coalesced access. With stride-32, each warp issues 32 separate 32-byte transactions for 128 bytes of useful data, yielding 12.5% memory utilization.
-
-## Health thresholds
-
-`l1tex__average_t_sectors_per_request_pipe_lsu_mem_global_op_ld.ratio` is the
-single number that says whether a load is coalesced.
-
-**The ideal value is 4, and it is a derivation, not a measurement:** a fully
-coalesced warp-wide load asks for one 128-byte line, the L1 sector is 32 bytes,
-so 128 / 32 = **4 sectors per request**. Nothing about the GPU model enters
-that arithmetic.
-
-| Observed ratio | Reads as |
-|---|---|
-| ~4 | fully coalesced |
-| 8-16 | partially coalesced — misalignment or a stride of 2-4 elements |
-| ~32 | each lane touched a different line; the warp paid 32 transactions for 128 useful bytes |
-
-The 32-sector case is the same access pattern as the stride-32 probe in
-*Measured Characteristics* above: 12.5% memory utilisation, 2.85x slower.
-
-### Superseded: the 128-byte-transaction model
-
-Older material describes a coalesced warp load as **one 128-byte transaction**
-and gives the ideal as **1 transaction per request**. That is the pre-Pascal
-model. On sm_60 and later the L1 sector is 32 bytes and a fully coalesced
-warp-wide load is **four** of them, which is why the ideal ratio above is 4 and
-not 1. Both statements describe the same hardware at different granularities;
-only the second matches what NCU reports.
-
-If you meet the "ideal is 1" phrasing in an inherited document, it is not a
-different opinion -- it is a stale one.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/memory/l2-access-policy/apis.md

@@ -1,203 +0,0 @@
----
-title: L2 Access Policy APIs
-status: draft
-source:
-- path: spec
-  anchor: Reference
-apis:
-- func_name: cudaStreamSetAttribute
-  namespace: cuda-runtime
-  kind: stream-attribute-setter
-  signature: cudaError_t cudaStreamSetAttribute(cudaStream_t stream, cudaLaunchAttributeID id, const cudaLaunchAttributeValue* value);
-  notes: Attach an attribute (e.g. accessPolicyWindow) to a stream; applies to every kernel launched on that stream until overridden. id = cudaLaunchAttributeAccessPolicyWindow selects the L2 window. value points to a cudaStreamAttrValue whose accessPolicyWindow member carries the policy.
-- func_name: cudaStreamGetAttribute
-  namespace: cuda-runtime
-  kind: stream-attribute-getter
-  signature: cudaError_t cudaStreamGetAttribute(cudaStream_t stream, cudaLaunchAttributeID id, cudaLaunchAttributeValue* value_out);
-  notes: Read back the current accessPolicyWindow attached to a stream. Useful when threading multiple policies through the same stream.
-- func_name: cudaAccessPolicyWindow
-  namespace: cuda-runtime
-  kind: struct
-  signature: struct cudaAccessPolicyWindow { void* base_ptr; size_t num_bytes; float hitRatio; cudaAccessProperty hitProp; cudaAccessProperty missProp; };
-  notes: Descriptor consumed by stream/graph/launch attribute setters. num_bytes must be <= cudaDeviceProp::accessPolicyMaxWindowSize. hitRatio ∈ [0,1]. Set num_bytes = 0 to disable.
-- func_name: cudaAccessProperty
-  namespace: cuda-runtime
-  kind: enum
-  notes: 'cudaAccessPropertyNormal (default), cudaAccessPropertyStreaming (evict-first), cudaAccessPropertyPersisting (evict-last / sticky). Typical pairing: hitProp=Persisting + missProp=Streaming.'
-- func_name: cudaDeviceSetLimit
-  namespace: cuda-runtime
-  kind: device-limit-setter
-  signature: cudaError_t cudaDeviceSetLimit(cudaLimit limit, size_t value);
-  notes: Call with limit=cudaLimitPersistingL2CacheSize to reserve the set-aside portion of L2. Silent no-op under MIG (legacy P3). Value is clamped to cudaDeviceProp::persistingL2CacheMaxSize (37.5 MiB on H200).
-- func_name: cudaDeviceGetLimit
-  namespace: cuda-runtime
-  kind: device-limit-getter
-  signature: cudaError_t cudaDeviceGetLimit(size_t* value_out, cudaLimit limit);
-  notes: Read the currently reserved set-aside size. Use to verify that cudaDeviceSetLimit actually took effect (MIG silent-failure detection).
-- func_name: cudaLimitPersistingL2CacheSize
-  namespace: cuda-runtime
-  kind: cudaLimit-enum-value
-  notes: Enum selector for cudaDeviceSetLimit / cudaDeviceGetLimit to address the persisting-L2 set-aside. Also accessible as cudaLimit::cudaLimitPersistingL2CacheSize.
-- func_name: cudaCtxResetPersistingL2Cache
-  namespace: cuda-runtime
-  kind: persisting-cache-reset
-  signature: cudaError_t cudaCtxResetPersistingL2Cache(void);
-  notes: 'Evict all persisting-tagged lines. Call between distinct hot regions to avoid stale pins (legacy P2). Also: set the window num_bytes=0 first, then call this.'
-- func_name: cudaLaunchKernelEx
-  namespace: cuda-runtime
-  kind: extended-launch
-  signature: cudaError_t cudaLaunchKernelEx(const cudaLaunchConfig_t* config, F kernel, Args... args);
-  notes: Per-launch attribute variant. config->attrs[] can include cudaLaunchAttributeAccessPolicyWindow for per-launch L2 policy, independent of stream defaults.
-- func_name: cudaLaunchConfig_t
-  namespace: cuda-runtime
-  kind: struct
-  notes: 'Launch config consumed by cudaLaunchKernelEx. Fields: gridDim, blockDim, dynamicSmemBytes, stream, attrs, numAttrs. attrs is an array of cudaLaunchAttribute whose id can be cudaLaunchAttributeAccessPolicyWindow.'
-- func_name: cudaLaunchAttributeAccessPolicyWindow
-  namespace: cuda-runtime
-  kind: launch-attribute-id
-  notes: ID constant selecting the access-policy-window attribute. Unified across stream (cudaStreamSetAttribute), graph node (cudaGraphKernelNodeSetAttribute), and launch (cudaLaunchAttribute) entry points.
-- func_name: cudaStreamAttributeAccessPolicyWindow
-  namespace: cuda-runtime
-  kind: stream-attribute-id
-  notes: Alias for cudaLaunchAttributeAccessPolicyWindow used in cudaStreamSetAttribute signatures. Same underlying ID.
-- func_name: cudaKernelNodeAttributeAccessPolicyWindow
-  namespace: cuda-runtime
-  kind: graph-node-attribute-id
-  notes: Alias for cudaLaunchAttributeAccessPolicyWindow used in cudaGraphKernelNodeSetAttribute signatures. Per-kernel-node L2 policy inside a CUDA Graph.
-- func_name: cudaGraphKernelNodeSetAttribute
-  namespace: cuda-runtime
-  kind: graph-node-attribute-setter
-  signature: cudaError_t cudaGraphKernelNodeSetAttribute(cudaGraphNode_t node, cudaKernelNodeAttrID id, const cudaKernelNodeAttrValue* value);
-  notes: Per-graph-node L2 policy variant. Semantically equivalent to the stream variant; picks whichever submission model the caller uses (legacy S4).
-- func_name: cudaGraphKernelNodeGetAttribute
-  namespace: cuda-runtime
-  kind: graph-node-attribute-getter
-  signature: cudaError_t cudaGraphKernelNodeGetAttribute(cudaGraphNode_t node, cudaKernelNodeAttrID id, cudaKernelNodeAttrValue* value_out);
-  notes: Read-back companion. Useful when composing reusable graph templates.
-- func_name: cudaStreamCopyAttributes
-  namespace: cuda-runtime
-  kind: stream-attribute-copy
-  signature: cudaError_t cudaStreamCopyAttributes(cudaStream_t dst, cudaStream_t src);
-  notes: Copies all attributes (including accessPolicyWindow) from src to dst. Convenient when cloning a stream's policy without re-specifying the struct.
-- func_name: cudaDeviceProp::l2CacheSize
-  namespace: cuda-runtime
-  kind: device-property
-  notes: 'Total L2 in bytes. H200: 62914560 (60 MiB).'
-- func_name: cudaDeviceProp::persistingL2CacheMaxSize
-  namespace: cuda-runtime
-  kind: device-property
-  notes: 'Maximum set-aside for persisting accesses. H200: 39321600 (37.5 MiB). Hardware-fixed upper bound for cudaLimitPersistingL2CacheSize.'
-- func_name: cudaDeviceProp::accessPolicyMaxWindowSize
-  namespace: cuda-runtime
-  kind: device-property
-  notes: 'Maximum num_bytes accepted in cudaAccessPolicyWindow. H200: 134217728 (128 MiB). Exceeding this value is silently clamped (legacy P5).'
-- func_name: createpolicy.fractional.L2::evict_last.L2::evict_first.b64
-  namespace: ptx
-  kind: ptx-l2-policy-create
-  signature: createpolicy.fractional.L2::evict_last.L2::evict_first.b64 cache_policy, fraction;
-  notes: PTX-level creation of a 64-bit L2 cache policy descriptor. Mirrors the runtime hitProp=Persisting + missProp=Streaming + hitRatio=fraction combination, but applies per-instruction via .L2::cache_hint load/store modifiers instead of per-stream.
-- func_name: ld.global.L2::cache_hint
-  namespace: ptx
-  kind: ptx-load-with-policy
-  signature: ld.global.L2::cache_hint.type d, [a], cache_policy;
-  notes: Issue a load using the policy descriptor from createpolicy. Per-instruction form; complementary to the per-stream runtime window. See the `cache-load-hints` skill (pending).
-- func_name: st.global.L2::cache_hint
-  namespace: ptx
-  kind: ptx-store-with-policy
-  signature: st.global.L2::cache_hint.type [a], b, cache_policy;
-  notes: Store variant of the above.
-- func_name: applypriority.global.L2
-  namespace: ptx
-  kind: ptx-l2-priority
-  notes: Apply eviction priority (evict_normal / evict_first / evict_last / no_allocate) to a span of L2 lines. Orthogonal to the runtime window — a single-shot hint, not a persistent tag.
-- func_name: discard.global.L2
-  namespace: ptx
-  kind: ptx-l2-discard
-  notes: Drop L2 lines (no write-back). Dangerous on dirty lines; intended for scratch regions the kernel has finished with.
-- func_name: cp.async.bulk.prefetch
-  namespace: ptx
-  kind: ptx-l2-prefetch
-  notes: Bulk async prefetch into L2. Pairs with the persisting window when you want to prime the cache before the hot phase.
-- func_name: cp.async.bulk.prefetch.tensor
-  namespace: ptx
-  kind: ptx-l2-prefetch-tensor
-  notes: TMA variant of the above; prefetches tensor tiles into L2 ahead of a cp.async.bulk.tensor load into smem.
-id: api-l2-access-policy-ref
-type: api-definition
-vendor: nvidia
-func_name: L2 Access Policy APIs
-namespace: runtime
-header: cuda_runtime.h
-signature: See documentation
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L4060-L4170
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1561-L1610
-architectures:
-- sm90
-- sm90a
-languages:
-- ptx
-- cuda-cpp
-hardware_features:
-- tma
-- cluster
-techniques:
-- persistent-kernel
-- cache-policy
-- shared-memory-optimization
-confidence: inferred
-evidence_level: inferred
-tags:
-- tma
-- cluster
-- persistent-kernel
-- cache-policy
-- shared-memory-optimization
-- ptx
-- cuda-cpp
----
-## Core APIs
-
-| API / Instruction | Layer | Purpose |
-|-------------------|-------|---------|
-| `cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, ...)` | Runtime | Attach L2 policy to a stream |
-| `cudaGraphKernelNodeSetAttribute(node, cudaKernelNodeAttributeAccessPolicyWindow, ...)` | Runtime | Attach L2 policy to a graph kernel node |
-| `cudaLaunchKernelEx(config, ...)` + `cudaLaunchAttributeAccessPolicyWindow` | Runtime | Attach L2 policy per launch |
-| `cudaAccessPolicyWindow` | Runtime | Struct: base_ptr / num_bytes / hitRatio / hitProp / missProp |
-| `cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, n)` | Runtime | Reserve the set-aside portion of L2 |
-| `cudaCtxResetPersistingL2Cache()` | Runtime | Evict all persisting-tagged lines |
-| `cudaDeviceProp::{l2CacheSize, persistingL2CacheMaxSize, accessPolicyMaxWindowSize}` | Runtime | Device-specific limits |
-
-## Policy-value enum
-
-| `cudaAccessProperty` | Meaning |
-|---|---|
-| `Normal` | Default L2 LRU; no tagging |
-| `Streaming` | Evict-first; this line should not stay in L2 |
-| `Persisting` | Evict-last; prefer to keep this line under pressure |
-
-## PTX-level complement (per-instruction hints)
-
-| Instruction | Purpose |
-|-------------|---------|
-| `createpolicy.fractional.L2::evict_last.L2::evict_first.b64` | Build a 64-bit policy descriptor |
-| `ld.global.L2::cache_hint.type` | Load using the descriptor |
-| `st.global.L2::cache_hint.type` | Store using the descriptor |
-| `applypriority.global.L2` | Per-span eviction priority (one-shot) |
-| `discard.global.L2` | Drop L2 lines (scratch release) |
-| `cp.async.bulk.prefetch[.tensor]` | Prefetch into L2 ahead of consume |
-
-## Cross-references
-
-- **`cache-load-hints` skill** (pending): `__ldcs` / `__ldca` / `.cg` / `.cs` are the per-instruction cousins of the runtime-level window. The two compose — a stream with `hitProp=Persisting` together with `__ldcs` loads inside the kernel signals both "keep this region hot at L2" and "do not pollute L1 with this access".
-- **`async-copy` skill**: `cp.async.bulk.prefetch` + persisting window is the canonical "prime L2, then stream through L1/smem" recipe.
-- **`coalescing` skill**: neither the window nor the PTX hints change memory-access *coalescing*. Fix coalescing first; L2 policy is downstream.
-- **`wiki/nvidia/hardware/thread-block-cluster/`** (pending bucket F): cluster-scope kernels share an L2 across blocks; the window semantics generalize but are not re-measured.
-
-## Related Probes
-
-- sources/experience/hw-probes/l2-residency.md — 3 WS × 3 policy sweep on H200. Load-bearing result: `hitRatio = set_aside / WS` at WS = 80 MiB gives +17.7% effective BW.

package/resources/kop-2026-10-23-0.1.17-beta.1/product_capabilities/kernel_optflow/skills/KernelWiki-private/wiki/nvidia/foundations/memory/l2-access-policy/pitfalls.md

@@ -1,98 +0,0 @@
----
-title: L2 Access Policy — Pitfalls
-status: verified
-related_skill: wiki/nvidia/foundations/memory/l2-access-policy/skill.md
-source:
-- path: spec
-  anchor: Reference
-experience_refs:
-- sources/experience/hw-probes/l2-residency.md
-id: pitfall-l2-access-policy
-type: pitfall
-vendor: nvidia
-source_refs:
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-programming-guide/cuda_cuda-programming-guide_index.html.md
-  anchor: L4094-L4170
-- source_id: cuda-official/toolkit-docs-13.2
-  path: CUDA Programming Guides/cuda-c-best-practices-guide/cuda_cuda-c-best-practices-guide_index.html.md
-  anchor: L1561-L1610
-architectures:
-- sm90
-- sm90a
-languages:
-- cuda-cpp
-techniques:
-- persistent-kernel
-- cache-policy
-confidence: inferred
-evidence_level: inferred
-tags:
-- persistent-kernel
-- cache-policy
-- cuda-cpp
----
-Legacy P1–P5 came from L3-sandbox verification on pre-H200 hardware (A100 unless noted). P6–P10 are L3-sandbox findings copied from the legacy KB and annotated with the current H200 measurement where the probe touches them. "**Measured on H200 sm_9.0a**" tags indicate pitfalls validated by sources/experience/hw-probes/l2-residency/.
-
-## P1. `hitRatio = 1.0` thrashes silently when `num_bytes > set_aside`
-
-**Symptom (legacy, A100)**: Performance drops ~10 % when the persistent data region exceeds L2 set-aside size with `hitRatio=1.0`. **Symptom (measured on H200 sm_9.0a, 2026-04-23)**: wall-clock is **identical** to no policy (WS=80 MiB / set_aside=37.5 MiB: 1.8694 ms both). The H200 L2 controller degrades gracefully to "nothing net-sticks" rather than evicting into DRAM. This is a **silent null**, not a catastrophic regression — but still the wrong tuning. **Detection**: measure wall-clock with and without the window; if they're equal, the window is a no-op. NCU L2 hit-rate metrics are unreliable here (see probe record — NCU replay artifacts). **Fix**: set `hitRatio = min(1.0, set_aside / WS)`. Measured: this recovers +17.7 % at WS=80 MiB on H200. **Source**: PG §4.13.3 (L4094-L4130); probe 2026-04-23.
-
-## P2. Stale persisting lines reduce L2 for subsequent kernels
-
-**Symptom**: Kernel B runs slower after Kernel A used L2 persistence, even though Kernel B has its own data. **Detection**: L2 hit rate for Kernel B is lower than baseline. Persisting lines from Kernel A still occupy set-aside slots. **Fix**:
-
-```cuda
-cudaStreamAttrValue attr = {};
-attr.accessPolicyWindow.num_bytes = 0;
-cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &attr);
-cudaCtxResetPersistingL2Cache();
-```
-
-**Not re-measured on H200** — the current probe is single-kernel so stale carry-over does not occur in the harness. Retained as legacy- anecdotal pending a multi-kernel follow-up probe. **Source**: PG §4.13.5 (reset L2 access to normal).
-
-## P3. MIG / MPS silently disables the set-aside
-
-**Symptom**: `cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, n)` returns `cudaSuccess` but nothing is reserved; subsequent policies never persist. **Detection**: `cudaDeviceGetLimit(&v, cudaLimitPersistingL2CacheSize)` returns 0 after the set call, **or** `nvidia-smi -i 0 -q | grep MIG` confirms MIG mode. **Fix**: L2 set-aside is not available in MIG or on MPS servers you do not administer. Fall back to per-instruction cache hints via the `cache-load-hints` skill (pending). On MPS, the set-aside is a server-start-time option controlled by the administrator. **Source**: PG §4.13.1 (L2 Cache Set-Aside for Persisting Accesses).
-
-## P4. Concurrent streams compete for the set-aside
-
-**Symptom**: Two concurrent streams with persisting windows evict each other's data. **Detection**: Both kernels show lower L2 hit rates than when run alone. **Fix**: Reduce `hitRatio` for each stream (e.g. 0.5 and 0.5 for two equal streams) so the **sum** of pinned footprints fits the set-aside. Alternatively, schedule them serially on one stream. **Not re-measured on H200** — the current probe is single-stream. Follow-up: `sources/experience/hw-probes/l2-residency-contended/` (open). **Source**: PG §4.13.6 (Manage Utilization of L2 Set-Aside Cache).
-
-## P5. `num_bytes` exceeds `accessPolicyMaxWindowSize`
-
-**Symptom**: Values of `num_bytes` beyond the device limit are silently clamped; effective window is smaller than requested. **Detection**: Compare `attr.accessPolicyWindow.num_bytes` against `cudaDeviceProp::accessPolicyMaxWindowSize`. H200: 128 MiB. **Fix**: `num_bytes = std::min(desired, prop.accessPolicyMaxWindowSize);`. For a hot region larger than 128 MiB, window covers only a prefix; the remaining tail uses default caching — this is almost never what the caller intends. **Source**: PG §4.13.7 (Query L2 Cache Properties).
-
-## P6. Applying the policy to already-well-cached data
-
-**Symptom (legacy, pre-H200)**: "Dramatically increases DRAM traffic and hurts performance." **Symptom (measured on H200 sm_9.0a, 2026-04-23)**: Downgraded to a **soft null** — WS=4 MiB and WS=40 MiB cases all land within 0.2 % of no-policy wall-clock (4524/4357/4360 GB/s effective). The H200 L2 controller does **not** punish the attempt; it just wastes a few cycles on policy metadata. Net measured impact: essentially 0. **Fix**: Only apply the window when WS approaches or exceeds the set-aside **and** other memory pressure exists. If the hot region is small enough that normal LRU keeps it resident, skip the window. **Source**: Level-3 sandbox verification (2026-04-05); downgraded on H200 by probe 2026-04-23.
-
-## P7. L2 policy on a bandwidth-saturated kernel
-
-**Symptom (legacy)**: "Kernels already at peak memory bandwidth may see single-kernel latency hurt slightly by policy enforcement overhead." **Symptom (measured on H200 sm_9.0a, 2026-04-23)**: Not observed. The WS=40 MiB case runs at 4357 GB/s regardless of policy — no measurable overhead. Legacy claim retained for completeness but demoted from "measured" to "inferred" on H200. **Fix**: If wall-clock noise floor hides your expected benefit, you do not have a qualifying use case; drop the window. **Source**: Level-3 sandbox verification (2026-04-05).
-
-## P8. Single-kernel microbench cannot demonstrate the "pinning" benefit
-
-**Symptom**: Probe or benchmark shows no wall-clock difference between "policy on" and "policy off" even when WS ≤ set_aside. **Detection**: Your harness has no concurrent workload to create eviction pressure on the hot buffer. LRU already keeps it resident for the duration of a single kernel. **Fix**: Either (a) accept that the window is a pure insurance policy in single-kernel mode (no measurable benefit but no cost either below set-aside), or (b) design a multi-kernel harness that touches competing memory between passes. Follow-up probe `sources/experience/hw-probes/l2-residency-contended/` (open). **Measured on H200 sm_9.0a**: WS=40 MiB soft-null directly confirms this pitfall — legacy P8 upgraded from inferred to measured. **Source**: Level-3 sandbox verification (2026-04-05); probe 2026-04-23.
-
-## P9. `num_bytes` exceeding the reservable set-aside silently no-ops
-
-**Symptom (legacy)**: "API succeeds but no persistence actually occurs." **Measured context on H200 sm_9.0a**: With WS=80 MiB and set_aside=37.5 MiB (H200 cap), `hitRatio=1.0` window ≈ `none` wall-clock — confirming the "silent no-op at hitRatio=1.0 when the region overflows set-aside" variant. The fix is P1's — use tuned hitRatio. **Fix**: Same as P1 (tuned hitRatio). Equivalent to P1 under a different phrasing; kept separate from legacy P1 because legacy P1 focused on the *performance loss* and legacy P9 on the *silent success* — the fix is the same but the diagnostic is different. **Source**: Level-3 sandbox verification (2026-04-05); probe 2026-04-23.
-
-## P10. "This skill is just runtime config — not a kernel optimization"
-
-**Symptom (legacy)**: Verification agent initially categorized the skill as "category error — kernel code doesn't change." **Resolution**: The observation is correct mechanically (kernel source is untouched) but misleading as a verdict. The policy *does* change observable wall-clock bandwidth when applied in the right regime — measured +17.7 % at WS=80 MiB on H200. Treat this skill as a **runtime-level tuning** tool, not a kernel-code optimization; both belong in the perf engineer's toolbox. **Source**: Level-3 sandbox verification (2026-04-05); probe 2026-04-23.
-
-## P11. Forgetting to reserve the set-aside (NEW — discovered during this probe)
-
-**Symptom**: `cudaStreamSetAttribute` with `hitProp=cudaAccessPropertyPersisting` + `hitRatio=1.0` returns `cudaSuccess`, but the policy has zero effect at any WS. **Cause**: The default set-aside on most platforms is **0 bytes** until `cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, n)` is called. Until then, "persisting" has no capacity to persist into. **Detection**: `cudaDeviceGetLimit(&v, cudaLimitPersistingL2CacheSize); assert(v > 0);`. Skipping this check is the most common "my window doesn't do anything" symptom on a correctly-configured H200. **Fix**: Always call `cudaDeviceSetLimit` before the first stream setup. Probe's pattern:
-
-```cuda
-cudaDeviceProp prop; cudaGetDeviceProperties(&prop, 0);
-size_t set_aside = std::min<size_t>(
-    (size_t)(prop.l2CacheSize * 0.75),
-    prop.persistingL2CacheMaxSize);
-cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, set_aside);
-```
-
-**Source**: Discovered while writing the probe harness 2026-04-23. PG §4.13.1 documents the reservation step but does not call out the default-zero behavior as a distinct failure mode — this pitfall makes it explicit.