math-skill 2.0.1 → 3.0.0

package/design-patterns/compression/low-rank-kv-cache.en.md

@@ -0,0 +1,60 @@
+# Low-Rank KV-Cache
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Target Problem
+Use when KV-Cache memory consumption becomes the bottleneck during LLM inference: long-context inference ($L > 8K$), multi-turn conversation accumulation, edge deployment, speculative decoding / beam search. Core objective: **compress the KV-Cache from $O(Ld)$ to $O(kd)$ with $k \ll L$, under controlled information loss**.
+
+## Mathematical Foundations
+- Lenses: lenses/spectral.md (spectral component identification and truncation), lenses/variational.md (Pareto trade-off between compression ratio and reconstruction error), lenses/duality.md (nuclear norm--spectral norm duality)
+- Knowledge: knowledge-base/matrix-analysis/low-rank-approximation.md (Eckart--Young optimal approximation, randomized SVD), knowledge-base/matrix-analysis/projection.md (orthogonal projection onto principal subspace), knowledge-base/matrix-analysis/matrix-perturbation.md (Weyl perturbation bound)
+
+## Required Mathematical Background
+- **Eckart--Young--Mirsky Theorem**: $A_k = U_k \Sigma_k V_k^H$, $\|A - A_k\|_F = \sqrt{\sum_{i>k} \sigma_i^2}$; truncated SVD = optimal rank-$k$ approximation
+- **Randomized SVD**: $Y = A\Omega$ ($\Omega$ random Gaussian), $Y = QR$, $B = Q^H A$, perform SVD on $B$; complexity $O(Ldk)$, dominated by matrix multiplications
+- **Weyl Perturbation Bound**: $\|A - A_k\|_2 = \sigma_{k+1}$; maximum deviation of attention scores after compression $\leq \sigma_{k+1}$
+- **Effective Rank**: $r_{\text{eff}}(K) = \|K\|_F^2 / \|K\|_2^2$, guides adaptive selection of $k$
+
+## AI Module Specification
+```
+Module: LowRankKVCompressor
+Input: K ∈ R^{L×d}, V ∈ R^{L×d}    Parameters: target rank k << L, update frequency M
+
+Method 1 - Offline periodic compression (most practical):
+  Omega = randn(d, k+p)                    // random projection, p=5 oversampling
+  Q_k = qr(K @ Omega)[0]                   // L×(k+p) orthonormal basis (GEMM + QR)
+  B_k = Q_k^T @ K                          // (k+p)×d small matrix GEMM
+  U_r, S_r, Vt_r = svd(B_k)               // small matrix SVD
+  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d compressed Key
+  // Attention: softmax(Q @ K_comp^T / √d) @ V_comp, sequence dimension L → k
+
+Method 2 - Streaming incremental compression (low latency):
+  Maintain basis (U_basis ∈ R^{k×d}), on new token arrival:
+    residual p = k_new - U_basis^T @ (U_basis @ k_new)
+    if ‖p‖ > τ: brand_update + truncate_to_rank(k)  // expand basis
+    else: coeff = U_basis @ k_new                    // project onto existing basis
+
+Method 3 - Layer-wise adaptive: allocate per-layer, per-head k based on effective rank r_eff[l,h]
+```
+
+## Implementable Architectures
+- **Periodic compression layer**: trigger randomized SVD every $M=64$ steps, $L \times d \to k \times d$
+- **Double buffering**: compressed basis + recent $w$ raw tokens, balancing accuracy and compression ratio
+- **Shared basis**: multi-head sharing of the Key column-space basis, each head stores only coefficients
+- **Quantized basis**: further INT8/FP8 quantization after compression, achieving dual compression
+
+## GPU Feasibility
+- Tensorization / GEMM: randomized SVD = 3 GEMMs + 1 small SVD, maps perfectly onto Tensor Cores
+- Complexity: $O(Ldk)$ is far superior to $O(Ld^2)$ full SVD; overhead negligible for $k \sim 256$
+- Memory: KV-Cache reduced from $O(Ld)$ to $O(kd)$; $k/L \sim 1/16$ yields 16x compression
+- Low precision: SVD recommended in fp32 (acceptable for small matrices); compressed KV can be stored back in bf16
+- Parallelism: compression across layers / heads is fully independent; incremental update $O(kd)$ with very low latency
+- Operator fusion: $K\Omega$ + QR can be partially fused; the QK^T dimension in attention is already reduced
+
+## Paper-Worthy Formulation
+"Building on the Eckart--Young--Mirsky theorem, we employ randomized SVD to project the KV-Cache onto the optimal rank-$k$ subspace, achieving $L/k$-fold memory compression at $O(Ldk)$ complexity, with the Weyl perturbation bound guaranteeing that attention score deviations do not exceed $\sigma_{k+1}$."
+
+## Risks
+- **Improper rank selection**: $k$ too small causes $\sigma_{k+1}$ to be non-negligible, degrading long-range recall; singular value decay curves must be monitored
+- **Incremental SVD error accumulation**: repeated Brand updates drift away from the true SVD, requiring periodic re-compression for correction
+- **Positional encoding distortion**: RoPE is coupled with Keys, and compression may disrupt relative positional information; separate handling or re-injection is needed
+- **Double-buffer seam**: attention score scales differ between the compressed region and the raw region, requiring unified normalization

package/design-patterns/compression/low-rank-kv-cache.md

@@ -0,0 +1,60 @@
+# Low-Rank KV-Cache（低秩 KV 缓存压缩）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当 LLM 推理的 KV-Cache 显存占用成为瓶颈时使用：长上下文推理（$L > 8K$）、多轮对话累积、边缘部署、投机解码/束搜索。核心诉求：**以可控信息损失将 KV-Cache 从 $O(Ld)$ 压缩到 $O(kd)$，$k \ll L$**。
+
+## 数学思想来源
+- 透镜：lenses/spectral.md（谱分量识别与截断）、lenses/variational.md（压缩率与重构误差的帕累托取舍）、lenses/duality.md（核范数-谱范数对偶）
+- 知识：knowledge-base/matrix-analysis/low-rank-approximation.md（Eckart-Young 最优逼近、随机化 SVD）、knowledge-base/matrix-analysis/projection.md（正交投影到主子空间）、knowledge-base/matrix-analysis/matrix-perturbation.md（Weyl 扰动界）
+
+## 需要的数学知识
+- **Eckart-Young-Mirsky 定理**：$A_k = U_k \Sigma_k V_k^H$，$\|A - A_k\|_F = \sqrt{\sum_{i>k} \sigma_i^2}$，截断 SVD = 最优秩-$k$ 逼近
+- **随机化 SVD**：$Y = A\Omega$（$\Omega$ 随机高斯），$Y = QR$，$B = Q^H A$，对 $B$ 做 SVD；复杂度 $O(Ldk)$，核心全是 matmul
+- **Weyl 扰动界**：$\|A - A_k\|_2 = \sigma_{k+1}$，压缩后 attention score 最大偏差 $\leq \sigma_{k+1}$
+- **有效秩**：$r_{\text{eff}}(K) = \|K\|_F^2 / \|K\|_2^2$，指导 $k$ 的自适应选取
+
+## AI 模块形式
+```
+模块：LowRankKVCompressor
+输入：K ∈ R^{L×d}, V ∈ R^{L×d}    参数：目标秩 k << L，更新频率 M
+
+方法1 - 离线周期压缩（最实用）：
+  Omega = randn(d, k+p)                    // 随机投影，p=5 过采样
+  Q_k = qr(K @ Omega)[0]                   // L×(k+p) 正交基底（GEMM + QR）
+  B_k = Q_k^T @ K                          // (k+p)×d 小矩阵 GEMM
+  U_r, S_r, Vt_r = svd(B_k)               // 小矩阵 SVD
+  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d 压缩 Key
+  // Attention: softmax(Q @ K_comp^T / √d) @ V_comp，序列维度 L → k
+
+方法2 - 流式增量压缩（低延迟）：
+  维护基底 (U_basis ∈ R^{k×d})，新 token 到来：
+    残差 p = k_new - U_basis^T @ (U_basis @ k_new)
+    if ‖p‖ > τ: brand_update + truncate_to_rank(k)  // 扩展基底
+    else: coeff = U_basis @ k_new                    // 投影到现有基底
+
+方法3 - 分层自适应：按有效秩 r_eff[l,h] 分配各层各头的 k
+```
+
+## 可实现结构
+- **周期压缩层**：每 M=64 步触发随机化 SVD，$L \times d \to k \times d$
+- **双缓冲**：压缩基底 + 近期 $w$ 个原始 token，兼顾精度与压缩率
+- **共享基底**：多头共享 Key 列空间基底，各头只存系数
+- **量化基底**：压缩后进一步 INT8/FP8 量化，双重压缩
+
+## GPU 可行性
+- 张量化/GEMM：随机化 SVD = 3 次 GEMM + 1 次小 SVD，完美映射 Tensor Core
+- 复杂度：$O(Ldk)$ 远优于 $O(Ld^2)$ 完整 SVD；$k \sim 256$ 时开销可忽略
+- 显存：KV-Cache 从 $O(Ld)$ 到 $O(kd)$，$k/L \sim 1/16$ 可达 16x 压缩
+- 低精度：SVD 建议 fp32（小矩阵可接受）；压缩后 KV 可回 bf16 存储
+- 并行：多层/多头压缩完全独立；增量更新 $O(kd)$ 极低延迟
+- 算子融合：$K\Omega$ + QR 可部分融合；attention 的 QK^T 维度已缩小
+
+## 论文表述方式
+"基于 Eckart-Young-Mirsky 定理，采用随机化 SVD 将 KV-Cache 投影到最优秩-$k$ 子空间，以 $O(Ldk)$ 复杂度实现 $L/k$ 倍显存压缩，Weyl 扰动界保证 attention score 偏差不超过 $\sigma_{k+1}$。"
+
+## 风险
+- **秩选取不当**：$k$ 过小导致 $\sigma_{k+1}$ 不可忽略，长距离 recall 下降；需监控奇异值衰减曲线
+- **增量 SVD 误差累积**：Brand 更新多次后偏离真实 SVD，需定期重压缩校正
+- **位置编码失真**：RoPE 与 Key 耦合，压缩可能破坏相对位置；需单独处理或重新注入
+- **双缓冲接缝**：压缩区与原始区的 attention score 尺度不一致，需统一归一化

package/design-patterns/compression/spectral-token-pruning.en.md

@@ -0,0 +1,61 @@
+# Spectral Token Pruning
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Target Problem
+Use when pruning must be based on the structural importance of tokens (rather than raw attention scores alone): KV-Cache eviction, long-document summarization, inference acceleration (reducing $O(L^2)$), multimodal vision token compression. Core objective: **quantify the structural importance of each token via spectral methods, achieving pruning with minimal information loss**.
+
+## Mathematical Foundations
+- Lenses: lenses/spectral.md (identifying dominant spectral components, discarding redundant ones), lenses/algorithmic.md (complexity classification and approximation algorithms), lenses/perturbation.md (pruning = sparse perturbation, Weyl bound for spectral drift estimation)
+- Knowledge: knowledge-base/matrix-analysis/spectral-decomposition.md (spectral radius, eigenvector centrality), knowledge-base/matrix-analysis/matrix-perturbation.md (Geršgorin discs, perturbation bounds), knowledge-base/matrix-analysis/positive-semidefinite.md (Gram matrix PSD structure)
+
+## Required Mathematical Background
+- **Eigenvector Centrality**: the principal eigenvector of the attention matrix $A$ satisfies $Ax = \lambda_1 x$; component $x_i$ quantifies the global influence of token $i$ (Perron--Frobenius guarantees non-negativity)
+- **Spectral Gap**: $\Delta = \lambda_1 - \lambda_2$ governs the rate of information diffusion; large $\Delta \Rightarrow$ a few tokens dominate $\Rightarrow$ safe to prune
+- **Fiedler Vector**: the second-smallest eigenvector of the Laplacian $L_{\text{sym}}$ yields the optimal bipartition; small magnitude = partition boundary = important
+- **Weyl Perturbation Bound**: pruning = removing rows/columns = rank-1 perturbation $E$, $|\lambda_i(A) - \lambda_i(A_{\text{pruned}})| \leq \|E\|_2$
+
+## AI Module Specification
+```
+Module: SpectralTokenPruner
+Input: K ∈ R^{L×d}    Parameters: retention ratio ρ ∈ (0,1]
+
+Method 1 - Spectral centrality pruning (power iteration):
+  A = softmax(K @ K^T / √d)                  // L×L similarity graph
+  v = ones(L) / √L
+  for t in range(5): v = A @ v; v = v / ‖v‖  // power iteration O(L²·T)
+  indices = topk(v, ceil(ρ * L))              // retain tokens with highest centrality
+
+Method 2 - Geršgorin cheap pruning (zero iterations):
+  A = softmax(K @ K^T / √d)
+  gersh_score = |diag(A)| + sum(|A|, dim=1)    // disc upper bound, O(L²) elementwise
+  indices = topk(gersh_score, ceil(ρ * L))
+
+Method 3 - Differentiable spectral pruning (end-to-end):
+  v = power_iteration(A, T=5)
+  gate = sigmoid(v @ W_gate / τ)               // soft gating, τ annealing
+  K_gated = gate * K                            // per-token scaling
+  L_sparse = ‖gate‖_1 / L                       // sparsity regularization
+```
+
+## Implementable Architectures
+- **Power iteration centrality**: 5-step matvec to estimate the principal eigenvector, $O(L^2 \cdot 5)$, suitable for moderate-length sequences
+- **Sampling approximation**: for $L > 4096$, sample $m$ anchor points and construct an $m \times m$ submatrix for spectral analysis
+- **Multi-head fusion**: average the attention graphs across different heads before performing spectral analysis
+- **Progressive pruning**: incrementally increase the pruning ratio across layers (light pruning in shallow layers, heavy pruning in deep layers)
+
+## GPU Feasibility
+- Tensorization / GEMM: $A = KK^T$ is a GEMM; power iteration is a chain of matvecs; Geršgorin is elementwise
+- Complexity: power iteration $O(L^2 T)$, $T \leq 10$; Geršgorin $O(L^2)$ elementwise, zero iterations
+- Memory: the $L \times L$ similarity matrix exceeds 256 MB for $L > 8K$, requiring chunking or sampling
+- Low precision: power iteration is stable in bf16 (normalization prevents overflow); Geršgorin is purely elementwise with no precision concerns
+- Parallelism: spectral analysis across heads / layers is independently parallel; matvec is highly parallelizable
+- Operator fusion: $KK^T$ + row-sum + topk can be fused into a single kernel
+
+## Paper-Worthy Formulation
+"We cast token pruning as spectral sparsification of a directed graph: leveraging the Perron--Frobenius principal eigenvector of the attention matrix to quantify global centrality, with the Weyl perturbation bound guaranteeing that post-pruning spectral drift does not exceed the $\ell_2$ norm of the removed tokens, while Geršgorin discs provide an $O(L^2)$ inexpensive alternative."
+
+## Risks
+- **$L \times L$ matrix memory bottleneck**: for long sequences the similarity matrix itself may exceed available memory, necessitating sampling or chunking
+- **Slow power iteration convergence**: when $\lambda_1 / \lambda_2 \approx 1$, $O(1/\Delta)$ iterations are required, reducing efficiency
+- **Semantics $\neq$ spectral importance**: certain tokens (e.g., punctuation) have low spectral centrality yet are semantically critical; purely spectral methods may prune them erroneously
+- **Hard pruning is non-differentiable**: top-k blocks gradients; end-to-end training requires softmax relaxation or Gumbel-topk

package/design-patterns/compression/spectral-token-pruning.md

@@ -0,0 +1,61 @@
+# Spectral Token Pruning（谱 Token 剪枝）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要基于 token 的结构性重要性（而非单纯 attention score）进行剪枝时使用：KV-Cache 驱逐、长文档摘要、推理加速（降低 $O(L^2)$）、多模态视觉 token 压缩。核心诉求：**用谱方法量化每个 token 的结构性重要性，实现信息损失最小的剪枝**。
+
+## 数学思想来源
+- 透镜：lenses/spectral.md（识别主导谱分量、丢弃冗余分量）、lenses/algorithmic.md（复杂度分类与近似算法）、lenses/perturbation.md（剪枝 = 稀疏扰动，Weyl 界估谱漂移）
+- 知识：knowledge-base/matrix-analysis/spectral-decomposition.md（谱半径、特征向量中心性）、knowledge-base/matrix-analysis/matrix-perturbation.md（Geršgorin 圆盘、扰动界）、knowledge-base/matrix-analysis/positive-semidefinite.md（Gram 矩阵 PSD 结构）
+
+## 需要的数学知识
+- **特征向量中心性**：attention 矩阵 $A$ 的主特征向量 $Ax = \lambda_1 x$，分量 $x_i$ 量化 token $i$ 的全局影响力（Perron-Frobenius 保证非负）
+- **谱间隙**：$\Delta = \lambda_1 - \lambda_2$ 决定信息扩散速度，$\Delta$ 大 $\Rightarrow$ 少数 token 主导 $\Rightarrow$ 安全剪枝
+- **Fiedler 向量**：Laplacian $L_{\text{sym}}$ 的第二小特征向量给出最优二分割，幅值小 = 分割边界 = 重要
+- **Weyl 扰动界**：剪枝 = 删除行/列 = 秩-1 扰动 $E$，$|\lambda_i(A) - \lambda_i(A_{\text{pruned}})| \leq \|E\|_2$
+
+## AI 模块形式
+```
+模块：SpectralTokenPruner
+输入：K ∈ R^{L×d}    参数：保留比例 ρ ∈ (0,1]
+
+方法1 - 谱中心性剪枝（幂迭代）：
+  A = softmax(K @ K^T / √d)                  // L×L 相似度图
+  v = ones(L) / √L
+  for t in range(5): v = A @ v; v = v / ‖v‖  // 幂迭代 O(L²·T)
+  indices = topk(v, ceil(ρ * L))              // 保留中心性最高的 token
+
+方法2 - Geršgorin 廉价剪枝（零迭代）：
+  A = softmax(K @ K^T / √d)
+  gersh_score = |diag(A)| + sum(|A|, dim=1)    // 圆盘上界，O(L²) elementwise
+  indices = topk(gersh_score, ceil(ρ * L))
+
+方法3 - 可微谱剪枝（端到端）：
+  v = power_iteration(A, T=5)
+  gate = sigmoid(v @ W_gate / τ)               // 软门控，τ 退火
+  K_gated = gate * K                            // 逐 token 缩放
+  L_sparse = ‖gate‖_1 / L                       // 稀疏正则
+```
+
+## 可实现结构
+- **幂迭代中心性**：5 步 matvec 估计主特征向量，$O(L^2 \cdot 5)$，适合中等序列
+- **采样近似**：$L > 4096$ 时采样 $m$ 个锚点，构造 $m \times m$ 子矩阵做谱分析
+- **多头融合**：不同头的 attention 图取平均后再做谱分析
+- **渐进式剪枝**：逐层递增剪枝比例（浅层少剪、深层多剪）
+
+## GPU 可行性
+- 张量化/GEMM：$A = KK^T$ 为 GEMM；幂迭代为 matvec 链；Geršgorin 为 elementwise
+- 复杂度：幂迭代 $O(L^2 T)$，$T \leq 10$；Geršgorin $O(L^2)$ elementwise，零迭代
+- 显存：$L \times L$ 相似度矩阵在 $L > 8K$ 时超 256MB，需分块或采样
+- 低精度：幂迭代在 bf16 下稳定（归一化防溢出）；Geršgorin 纯 elementwise 无精度问题
+- 并行：多头/多层谱分析独立并行；matvec 高度并行
+- 算子融合：$KK^T$ + row-sum + topk 可融合为单一 kernel
+
+## 论文表述方式
+"将 token 剪枝建模为有向图的谱稀疏化：利用 attention 矩阵的 Perron-Frobenius 主特征向量量化全局中心性，Weyl 扰动界保证剪枝后谱漂移不超过被移除 token 的 $\ell_2$ 范数，Geršgorin 圆盘提供 $O(L^2)$ 廉价替代。"
+
+## 风险
+- **$L \times L$ 矩阵显存瓶颈**：长序列下相似度矩阵本身可能超出显存，必须采样或分块
+- **幂迭代收敛慢**：$\lambda_1 / \lambda_2 \approx 1$ 时需 $O(1/\Delta)$ 步，效率下降
+- **语义 ≠ 谱重要性**：某些 token（如标点）谱中心性低但语义关键，纯谱方法可能误剪
+- **硬剪枝不可微**：top-k 阻断梯度，端到端训练需 softmax 松弛或 Gumbel-topk

package/design-patterns/compression/topology-preserving-compression.en.md

@@ -0,0 +1,64 @@
+# Topology-Preserving Compression
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Target Problem
+Use when compressing representations while homologically preserving the intrinsic topological structure of the data (connected components, loops, cavities): latent-space compression (a toroidal manifold must not collapse into a line segment), knowledge distillation (student--teacher homological equivalence), 3D mesh simplification (genus invariance), KV-Cache semantic preservation (cluster structure must not collapse). Core objective: **compress dimensions or counts while ensuring that changes in the persistence diagram of persistent homology remain controlled**.
+
+## Mathematical Foundations
+- Lenses: lenses/topological.md (topological invariants -- connected components and hole counts are invariant under continuous deformation), lenses/spectral.md (Gauss--Bonnet linking curvature and Euler characteristic), lenses/variational.md (constrained variational of compression ratio vs. topological fidelity)
+- Knowledge: knowledge-base/topology/persistent-homology.md (persistent homology, Vietoris--Rips filtration, bottleneck distance), knowledge-base/topology/euler-characteristic.md (Euler characteristic for rapid topological diagnostics), knowledge-base/matrix-analysis/matrix-perturbation.md (Davis--Kahan subspace perturbation bound)
+
+## Required Mathematical Background
+- **Stability Theorem of Persistent Homology**: $d_B(D(X), D(Y)) \leq d_{GH}(X, Y)$; the Hausdorff distance bounds the change in persistence diagrams
+- **Euler Characteristic Curve**: $\chi(\epsilon) = \sum_k (-1)^k \beta_k(\epsilon)$, richer in information than a single $\chi$, computable in $O(N^2)$
+- **Mapping Cylinder Isomorphism**: if $f: X \to Y$ is an $\epsilon$-isometry, then $f_*$ induces an isomorphism on features with persistence intervals $> 2\epsilon$
+- **Landmark Approximation**: build a witness complex on $m$ landmarks, $O(m^3)$ replacing $O(N^3)$
+
+## AI Module Specification
+```
+Module: TopologyPreservingCompressor
+Input: X ∈ R^{N×d}    Parameters: topological weight λ_topo, target dimension r < d
+
+Method 1 - Euler curve matching (most practical, differentiable):
+  Z = encoder(X)                              // R^{N×r}
+  D_orig = cdist(X, X); D_comp = cdist(Z, Z)  // N×N distance matrices
+  chi_orig = euler_curve(D_orig, eps_grid)      // |eps_grid|-dimensional vector
+  chi_comp = euler_curve(D_comp, eps_grid)
+  L_topo = ‖chi_orig - chi_comp‖_2²            // topological matching loss (differentiable)
+  L_total = L_recon + λ_topo · L_topo
+
+Method 2 - Persistent homology regularization (exact but expensive):
+  D_orig = persistent_homology(X, max_dim=1)    // H_0 + H_1 barcode
+  D_comp = persistent_homology(Z, max_dim=1)
+  // differentiable surrogate: persistence landscape/image
+  L_topo = ‖landscape(D_orig) - landscape(D_comp)‖_2²
+
+Method 3 - Topology monitoring + adaptive compression ratio (at inference time):
+  Z = compress(X, ratio=ρ)
+  if count_components(Z, τ) < 0.8 * count_components(X, τ):
+    ρ *= 1.2; Z = compress(X, ratio=ρ)          // topological collapse → reduce compression ratio
+```
+
+## Implementable Architectures
+- **Euler curve matching layer**: $\chi(\epsilon)$ requires only a distance matrix + threshold counting; the $\epsilon$ sweep is parallelizable
+- **Landmark sampler**: FPS (Farthest Point Sampling) in $O(Nm)$, guaranteeing coverage
+- **Topology-aware distillation**: the persistence image discrepancy between teacher and student representations serves as an auxiliary distillation objective
+- **Topological diagnostics dashboard**: real-time plotting of $\beta_0(\epsilon), \beta_1(\epsilon)$ curves during training
+
+## GPU Feasibility
+- Tensorization / GEMM: the distance matrix `cdist` is GEMM-dominated; $\chi(\epsilon)$ involves threshold counting + cumulative sum
+- Complexity: full persistent homology $O(N^3)$ is infeasible; Euler curve $O(N^2 |\epsilon|)$ is feasible; Landmark $O(m^3)$
+- Memory: the $N \times N$ distance matrix requires chunking or landmark reduction for $N > 8K$
+- Low precision: distance computation is stable in bf16 (positive-number addition); $\chi$ involves integer arithmetic with no precision concerns
+- Parallelism: each threshold in the $\epsilon$ sweep is independently parallel; landmark selection can be batch-parallelized
+- Operator fusion: cdist + threshold + count can be fused to avoid materializing the large distance matrix
+
+## Paper-Worthy Formulation
+"Grounded in the Bottleneck stability theorem of persistent homology, we achieve $O(N^2)$-complexity topology-preserving regularization via Euler characteristic curve matching, ensuring that post-compression deviations in Betti numbers over persistence intervals are controlled by the Hausdorff distance."
+
+## Risks
+- **Persistent homology computational bottleneck**: exact boundary matrix reduction is highly serial ($O(N^3)$), necessitating reliance on Euler curve or landmark approximations
+- **Information loss in the Euler curve**: $\chi = \sum(-1)^k \beta_k$ collapses multi-order Betti numbers into a single scalar; distinct topologies may share the same $\chi$
+- **Topology $\neq$ semantics**: topological preservation does not imply semantic preservation -- two semantically distinct spaces may be topologically isomorphic
+- **Scale parameter sensitivity**: the $\epsilon$ range for the filtration must be set manually and varies significantly across datasets
+- **Landmark sampling bias**: FPS may produce non-uniform coverage in high-dimensional spaces, leading to biased topological estimates

package/design-patterns/compression/topology-preserving-compression.md

@@ -0,0 +1,64 @@
+# Topology-Preserving Compression（拓扑保持压缩）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当压缩表示时需要同构保持数据的本质拓扑结构（连通性、环、空腔）时使用：隐空间压缩（环形流形不能压成线段）、知识蒸馏（学生-教师同调等价）、3D 网格简化（genus 不变）、KV-Cache 语义保持（聚类结构不坍缩）。核心诉求：**压缩维度或数量，保证持续同调的 persistence diagram 变化可控**。
+
+## 数学思想来源
+- 透镜：lenses/topological.md（拓扑不变量——连通性、洞数在连续变形下不变）、lenses/spectral.md（Gauss-Bonnet 连接曲率与欧拉示性数）、lenses/variational.md（压缩率 vs. 拓扑保真度的约束优化）
+- 知识：knowledge-base/topology/persistent-homology.md（持续同调、Vietoris-Rips 滤流、Bottleneck 距离）、knowledge-base/topology/euler-characteristic.md（欧拉示性数快速拓扑诊断）、knowledge-base/matrix-analysis/matrix-perturbation.md（Davis-Kahan 子空间扰动界）
+
+## 需要的数学知识
+- **持续同调稳定性定理**：$d_B(D(X), D(Y)) \leq d_{GH}(X, Y)$，Hausdorff 距离界定 persistence diagram 变化
+- **欧拉示性数曲线**：$\chi(\epsilon) = \sum_k (-1)^k \beta_k(\epsilon)$，比单一 $\chi$ 信息更丰富，计算 $O(N^2)$
+- **映射柱同构**：若 $f: X \to Y$ 是 $\epsilon$-等距，则 $f_*$ 在持续区间 $> 2\epsilon$ 的特征上同构
+- **Landmark 近似**：用 witness complex 在 $m$ 个 landmark 上构建，$O(m^3)$ 替代 $O(N^3)$
+
+## AI 模块形式
+```
+模块：TopologyPreservingCompressor
+输入：X ∈ R^{N×d}    参数：拓扑权重 λ_topo，目标维度 r < d
+
+方法1 - Euler curve 匹配（最实用，可微）：
+  Z = encoder(X)                              // R^{N×r}
+  D_orig = cdist(X, X); D_comp = cdist(Z, Z)  // N×N 距离矩阵
+  chi_orig = euler_curve(D_orig, eps_grid)      // |eps_grid| 维向量
+  chi_comp = euler_curve(D_comp, eps_grid)
+  L_topo = ‖chi_orig - chi_comp‖_2²            // 拓扑匹配损失（可微）
+  L_total = L_recon + λ_topo · L_topo
+
+方法2 - 持续同调正则（精确但昂贵）：
+  D_orig = persistent_homology(X, max_dim=1)    // H_0 + H_1 barcode
+  D_comp = persistent_homology(Z, max_dim=1)
+  // 可微代理：persistence landscape/image
+  L_topo = ‖landscape(D_orig) - landscape(D_comp)‖_2²
+
+方法3 - 拓扑监控 + 自适应压缩率（推理时）：
+  Z = compress(X, ratio=ρ)
+  if count_components(Z, τ) < 0.8 * count_components(X, τ):
+    ρ *= 1.2; Z = compress(X, ratio=ρ)          // 拓扑坍缩 → 降低压缩比
+```
+
+## 可实现结构
+- **Euler curve 匹配层**：$\chi(\epsilon)$ 只需距离矩阵 + 阈值计数，$\epsilon$ 扫描可并行
+- **Landmark 采样器**：FPS（最远点采样）$O(Nm)$，保证覆盖
+- **拓扑感知蒸馏**：教师-学生表示的 persistence image 差异作为附加蒸馏目标
+- **拓扑诊断仪表板**：训练中实时绘制 $\beta_0(\epsilon), \beta_1(\epsilon)$ 曲线
+
+## GPU 可行性
+- 张量化/GEMM：距离矩阵 `cdist` 核心为 GEMM；$\chi(\epsilon)$ 为阈值计数 + 累积和
+- 复杂度：完整持续同调 $O(N^3)$ 不可行；Euler curve $O(N^2 |\epsilon|)$ 可行；Landmark $O(m^3)$
+- 显存：$N \times N$ 距离矩阵在 $N > 8K$ 时需分块或 Landmark 降维
+- 低精度：距离计算在 bf16 下稳定（正数加法）；$\chi$ 为整数运算无精度问题
+- 并行：$\epsilon$ 扫描的每个阈值独立并行；Landmark 选择可批并行
+- 算子融合：cdist + threshold + count 可融合避免物化大距离矩阵
+
+## 论文表述方式
+"以持续同调的 Bottleneck 稳定性定理为理论基础，通过 Euler characteristic curve 匹配实现 $O(N^2)$ 复杂度的拓扑保真正则化，保证压缩后 Betti 数在持续区间上的偏差受 Hausdorff 距离控制。"
+
+## 风险
+- **持续同调计算瓶颈**：精确边界矩阵约化高度串行（$O(N^3)$），必须依赖 Euler curve 或 Landmark 近似
+- **Euler curve 信息退化**：$\chi = \sum(-1)^k \beta_k$ 将多阶 Betti 数压成单值，不同拓扑可共享同一 $\chi$
+- **拓扑 ≠ 语义**：拓扑保持不等于语义保持——两个语义不同的空间可能拓扑同构
+- **尺度参数敏感**：滤流的 $\epsilon$ 范围需手动设定，不同数据集差异大
+- **Landmark 采样偏差**：FPS 在高维空间中可能不均匀，导致拓扑估计偏差

package/design-patterns/loss/constraint-penalty.en.md

@@ -0,0 +1,67 @@
+# Constraint Penalty
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the design involves hard constraints (e.g., probability simplex, orthogonality, capacity limits, load balancing) but end-to-end training is required. Typical scenarios: (1) MoE routing probabilities must lie on the $K$-simplex with load balancing; (2) Expert activation count is constrained (top-k); (3) Subspace projection matrices must satisfy orthogonality $W^T W = I$; (4) Feature norms are bounded $\|z\| \leq R$. Core objective: **transform mathematical constraints into differentiable penalty terms integrated into gradient-based variational**.
+
+## Mathematical Inspiration
+- Lenses: lenses/variational.md (constrained variational, Lagrangian duality, KKT conditions), lenses/geometric.md (manifold projection)
+- Knowledge: knowledge-base/optimization/lagrangian-duality.md (augmented Lagrangian method, penalty function method), knowledge-base/matrix-analysis/projection.md (projection operators, constraint sets)
+
+## Required Mathematical Knowledge
+- **Penalty Function Method**: $\min f(x)$ s.t. $g(x)=0 \to \min f(x) + \rho/2 \cdot \|g(x)\|^2$ -- $\rho$ is gradually increased (exterior point method), driving constraint violation $\|g(x)\| \to 0$
+- **Augmented Lagrangian Method**: $\min f(x) + \lambda^T g(x) + \rho/2 \cdot \|g(x)\|^2$ -- introduces dual variable $\lambda$, alternating between updating $\lambda \leftarrow \lambda + \rho \cdot g(x)$ and optimizing $x$; converges better than pure penalty methods
+- **Projected Gradient Method**: $x_{k+1} = \text{Proj}_C(x_k - \alpha \nabla f(x_k))$ -- for simple constraint sets $C$ (e.g., simplex, sphere), closed-form projection formulas exist
+- **Barrier Function Method**: $\min f(x) - \mu \sum \log(-g_i(x))$ for inequality constraints $g_i(x) \leq 0$ -- as $\mu \to 0$, approaches the constrained optimum
+
+## AI Module Form
+```
+Module: ConstraintPenalty
+Input: constraint violations g(x) in R^m (equality constraints), h(x) in R^p (inequality constraints)
+
+Method 1 - Adaptive Penalty Function (most commonly used):
+  L_penalty = Sum_i rho_i/2 * g_i(x)^2  +  Sum_j rho_j/2 * max(0, h_j(x))^2
+  // rho updated dynamically: each epoch rho_i *= gamma (gamma=2~10) until constraints are satisfied
+  // Different constraints can have different rho values, adapted to violation severity
+
+Method 2 - Augmented Lagrangian Method (ALM):
+  L_ALM = lambda^T * g(x) + rho/2 * ||g(x)||^2
+  // lambda is a learnable parameter (nn.Parameter), updated via gradient ascent:
+  lambda.data += rho * g(x).detach()   // dual ascent step
+  // Converges faster than pure penalty, avoids rho -> infinity
+
+Method 3 - Softmax Projection onto Simplex (load balancing special case):
+  p = softmax(logits / tau)           // project onto Delta^{K-1}
+  L_balance = ||p - 1/K||^2           // uniformity penalty
+  // Or Switch Transformer auxiliary loss:
+  L_aux = K * Sum_k f_k * P_k         // f_k = allocation fraction, P_k = average probability
+
+Method 4 - Orthogonal Constraint Projection:
+  W_proj = W * (W^T W)^{-1/2}       // project onto Stiefel manifold via matrix square root inverse
+  // Or parameterize via Cayley transform: W = (I-A)(I+A)^{-1} * W_0, A is skew-symmetric
+```
+
+## Implementable Architectures
+- **Loss Wrapper**: ConstraintLoss(base_loss, constraints, rho_schedule) -- forward computes base_loss + Sum constraint.penalty()
+- **Dual Variable Management**: nn.Parameter stores lambda; negative learning rate set in the optimizer to achieve gradient ascent
+- **Warm-up Strategy**: Optimize only base_loss for the first $N$ steps, then progressively activate constraint penalties
+- **Constraint Monitoring**: Record $\|g(x)\|$ at each step for visualization and adaptive rho adjustment
+
+## GPU Feasibility
+- **Tensorization**: Constraint violations are vector/matrix operations; penalty terms are element-wise squared sums
+- **GEMM-mappability**: Load balancing $f_k$, $P_k$ computation uses softmax + reduce_sum; orthogonal constraint uses matmul
+- **Complexity**: Penalty computation is $O(m)$ or $O(m^2)$, far smaller than the main network forward pass; negligible
+- **Memory & KV-Cache**: Only additional storage for lambda ($m$ dimensions) and rho ($m$ dimensions); minimal overhead
+- **Low Precision Stability**: Penalty terms are squaring operations, safe under fp16; ALM lambda updates are recommended in fp32 to avoid accumulated errors
+- **Parallelism & Communication**: Constraints are independently computable and parallelizable; on multiple GPUs, lambda updates require all-reduce of $g(x)$
+- **Sparse Structure**: $\max(0, h(x))^2$ has zero gradient when constraints are satisfied, naturally sparse activation
+- **Operator Fusion**: Constraint computation + weighted summation + merging with base_loss can be fused into a single kernel
+
+## Paper Phrasing
+"We employ the augmented Lagrangian method to convert hard constraints $g(x)=0$ into differentiable penalty terms $\lambda^T g(x) + \rho/2 \|g(x)\|^2$. Through alternating dual ascent updates of $\lambda$, constraint satisfaction is guaranteed to converge at rate $O(1/\rho)$ without requiring $\rho \to \infty$, reducing constraint violations by 3x compared to pure penalty methods in experiments."
+
+## Risks
+- Excessively fast rho growth leads to ill-conditioned variational landscapes (deteriorating condition numbers), causing gradient vanishing or explosion
+- ALM lambda update frequency and step size require tuning; too fast causes oscillation, too slow impedes convergence
+- When multiple constraints coexist, the relative ratios of their rho values influence the variational trajectory
+- Projection operations (e.g., matrix square root inverse) are computationally expensive and numerically unstable

package/design-patterns/loss/constraint-penalty.md

@@ -0,0 +1,78 @@
+# Constraint Penalty（约束惩罚损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当设计中有硬约束（如概率单纯形、正交性、容量限制、负载均衡）但需要端到端训练时使用。
+典型场景：(1) MoE 路由概率必须在 K-simplex 上且负载均衡；(2) 专家激活数量受限（top-k）；
+(3) 子空间投影矩阵必须满足正交性 W^TW = I；(4) 特征范数有界 ‖z‖ ≤ R。
+核心诉求：**将数学约束转化为可微惩罚项，融入梯度优化流程**。
+
+## 数学思想来源
+- 透镜：lenses/variational.md（约束优化、拉格朗日对偶、KKT 条件）、lenses/geometric.md（流形投影）
+- 知识：knowledge-base/optimization/lagrangian-duality.md（增广拉格朗日法、惩罚函数法）、
+  knowledge-base/matrix-analysis/projection.md（投影算子、约束集）
+
+## 需要的数学知识
+- **罚函数法**：min f(x) s.t. g(x)=0 → min f(x) + ρ/2 · ‖g(x)‖²
+  ρ 逐步增大（外点法），约束违反量 ‖g(x)‖ → 0
+- **增广拉格朗日法**：min f(x) + λ^T g(x) + ρ/2 · ‖g(x)‖²
+  引入对偶变量 λ，交替更新 λ ← λ + ρ·g(x) 和优化 x，比纯罚函数收敛更好
+- **投影梯度法**：x_{k+1} = Proj_C(x_k - α∇f(x_k))
+  对简单约束集 C（如 simplex、球面）有闭合投影公式
+- **屏障函数法**：min f(x) - μ Σ log(-g_i(x))  对不等式约束 g_i(x) ≤ 0
+  μ → 0 时逼近约束最优解
+
+## AI 模块形式
+```
+模块：ConstraintPenalty
+输入：约束违反量 g(x) ∈ R^m（等式约束），h(x) ∈ R^p（不等式约束）
+
+方法1 - 自适应罚函数（最常用）：
+  L_penalty = Σ_i ρ_i/2 · g_i(x)²  +  Σ_j ρ_j/2 · max(0, h_j(x))²
+  // ρ 动态更新：每个 epoch ρ_i *= γ（γ=2~10）直到约束满足
+  // 不同约束可有不同的 ρ，按违反程度自适应
+
+方法2 - 增广拉格朗日（ALM）：
+  L_ALM = λ^T · g(x) + ρ/2 · ‖g(x)‖²
+  // λ 为可学习参数（nn.Parameter），通过梯度上升更新：
+  λ.data += ρ * g(x).detach()   // 对偶上升步
+  // 比纯罚函数收敛快，避免 ρ → ∞
+
+方法3 - Softmax 投影到单纯形（负载均衡特例）：
+  p = softmax(logits / τ)           // 投影到 Δ^{K-1}
+  L_balance = ‖p - 1/K‖²            // 均匀性惩罚
+  // 或 Switch Transformer 的辅助损失：
+  L_aux = K · Σ_k f_k · P_k         // f_k = 分配比例, P_k = 平均概率
+
+方法4 - 正交约束投影：
+  W_proj = W · (W^T W)^{-1/2}       // 通过矩阵平方根反投影到 Stiefel 流形
+  // 或用 Cayley 变换参数化：W = (I-A)(I+A)^{-1} · W_0, A 为反对称矩阵
+```
+
+## 可实现结构
+- **Loss Wrapper**：ConstraintLoss(base_loss, constraints, ρ_schedule)
+  forward 时计算 base_loss + Σ constraint.penalty()
+- **对偶变量管理**：nn.Parameter 存储 λ，optimizer 中为 λ 设置负学习率实现梯度上升
+- **warm-up 策略**：前 N 步只优化 base_loss，之后逐步激活约束惩罚
+- **约束监控**：每 step 记录 ‖g(x)‖ 用于可视化和自适应 ρ 调整
+
+## GPU 可行性
+- **张量化**：约束违反量为向量/矩阵运算，罚项为 element-wise 平方和
+- **GEMM 可映射**：负载均衡的 f_k, P_k 计算为 softmax + reduce_sum；正交约束为 matmul
+- **复杂度**：罚项计算 O(m) 或 O(m²)，远小于主网络前向，可忽略
+- **显存与 KV-Cache**：仅额外存储 λ (m 维) 和 ρ (m 维)，极小开销
+- **低精度稳定**：罚项为平方运算，fp16 安全；ALM 的 λ 更新建议 fp32 避免累积误差
+- **并行与通信**：各约束独立计算，可并行；多 GPU 时 λ 更新需 all-reduce g(x)
+- **稀疏结构**：max(0, h(x))² 在约束满足时梯度为零，天然稀疏激活
+- **算子融合**：constraint 计算 + 加权求和 + 与 base_loss 合并可融合为单 kernel
+
+## 论文表述方式
+"采用增广拉格朗日法将硬约束 g(x)=0 转化为可微惩罚项 λ^Tg(x) + ρ/2‖g(x)‖²，
+通过对偶变量 λ 的交替上升更新，在无需 ρ→∞ 的条件下保证约束满足度以 O(1/ρ) 收敛，
+相比纯罚函数法在实验中减少 3× 的约束违反。"
+
+## 风险
+- ρ 增大过快导致优化 landscape 病态（条件数恶化），梯度消失或爆炸
+- ALM 的 λ 更新频率和步长需调参，更新过快震荡、过慢收敛
+- 多个约束同时存在时 ρ 的相对比例影响优化路径
+- 投影操作（如矩阵平方根逆）计算代价高且数值不稳定

package/design-patterns/loss/contrastive-loss.en.md

@@ -0,0 +1,59 @@
+# Contrastive Loss
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the model needs to learn "what is similar to what and what is different from what." Typical scenarios: (1) Different augmented views of the same input should be pulled closer (positive pairs), while different inputs should be pushed apart (negative pairs); (2) Shared representations should capture cross-task commonalities, while Private representations should distinguish task-specific features; (3) In the expert embedding space, similar inputs should be routed to the same expert. Core objective: **learn relative relationships rather than absolute values**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (metric spaces and distance functions), lenses/probabilistic.md (mutual information maximization)
+- Knowledge: knowledge-base/probability/entropy.md (conditional distributions and likelihood), knowledge-base/differential-geometry/manifold.md (geodesics and curvature)
+
+## Required Mathematical Knowledge
+- **InfoNCE Loss**: $L = -\log[\exp(\text{sim}(q,k^+)/\tau) / \sum_j \exp(\text{sim}(q,k_j)/\tau)]$ -- essentially a lower-bound estimate of mutual information in contrastive learning; $\tau$ is the temperature parameter controlling distribution sharpness
+- **Margin-Based Metric Learning**: Triplet Loss $= \max(0, d(a,p) - d(a,n) + \text{margin})$ -- explicitly widens the distance gap between positive and negative pairs in metric space
+- **NT-Xent (Normalized Temperature-scaled Cross Entropy)**: Softmax contrastive loss on the unit sphere $S^{d-1}$, with normalization eliminating scale effects
+- **Debiased Contrastive Learning**: Corrects for false negatives in negative samples, using prior $\tau^+$ to estimate the true negative sample distribution
+
+## AI Module Form
+```
+Module: ContrastiveLoss
+Input: anchors z_a in R^{B x d}, positives z_p in R^{B x d}, negative pool z_n in R^{N x d}
+
+Core formula (InfoNCE + temperature scaling):
+  sim(q, k) = q^T k / (||q|| * ||k||)       // cosine similarity
+  logits_i = [sim(z_a_i, z_p_i)] (+) [sim(z_a_i, z_n_j)]_{j=1}^N  // concatenation
+  L_contrast = -1/B * Sum_i log( exp(logits_i[0]/tau) / Sum_j exp(logits_i[j]/tau) )
+
+Queue mechanism (MoCo style):
+  z_n = FIFO_queue.enqueue(z_p.detach())   // negative sample queue, capacity N >> B
+  // Queue stores encodings from historical batches, increasing negative count without additional memory
+
+Hard Negative Mining:
+  top-k indices = argsort(sim(z_a, z_n), descending=True)[:k]
+  z_n_hard = z_n[top-k indices]            // retain only the k hardest negatives
+```
+
+## Implementable Architectures
+- **Dual-Tower Encoder + Projection Head**: encoder -> projection_head (2-layer MLP) -> normalize -> loss
+- **Negative Sample Queue**: Maintain a FIFO queue of momentum encoder outputs, capacity N=65536
+- **Symmetric Loss**: $L = L(a \to p) + L(p \to a)$, swapping positive/negative roles for enhanced training stability
+- **Multi-Granularity Contrast**: Apply contrastive objectives simultaneously at token-level, sequence-level, and expert-level
+
+## GPU Feasibility
+- **Tensorization**: Similarity computation is $z_a @ z_n^T$ -- standard GEMM $(B \times d) @ (d \times N) = B \times N$
+- **GEMM-mappability**: Core computation is 1-2 matrix multiplications, perfectly mapped to cuBLAS
+- **Complexity**: $O(B \cdot N \cdot d)$ computation + $O(B \cdot N)$ storage for the logits matrix; approximately 64MB when B=256, N=65536
+- **Memory & KV-Cache**: Negative sample queue occupies $N \cdot d \cdot 4$ bytes, approximately 65536 * 256 * 4 = 64MB, fixed overhead
+- **Low Precision Stability**: Cosine similarity + softmax under fp16 requires caution for exp overflow; use log-sum-exp trick
+- **Parallelism & Communication**: On multiple GPUs, use all-gather to collect negatives from other GPUs to enlarge $N$ (MoCo v3 strategy)
+- **Sparse Structure**: After hard negative mining, only $k \ll N$ negatives are retained, effectively sparsifying the logits
+- **Operator Fusion**: L2-norm -> matmul -> scale -> log-softmax -> nll_loss can be fused
+
+## Paper Phrasing
+"We employ temperature-scaled InfoNCE contrastive loss with a momentum encoder maintaining a negative sample queue of N=65536, maximizing the lower bound of positive-pair mutual information on the unit sphere. Theoretical analysis shows this lower bound converges at rate $O(1/\sqrt{N})$."
+
+## Risks
+- Too small $\tau$ causes training instability (excessively large gradients); too large $\tau$ makes all samples indistinguishable (degenerates to uniform distribution)
+- Stale encodings in the negative sample queue introduce representation bias
+- False negative problem: unsupervised negative sampling may select samples that are semantically similar but differently labeled
+- When $B$ is too small, the positive/negative pair imbalance within the batch must be compensated by the queue