math-skill 2.1.0 → 3.0.0

package/design-patterns/compression/topology-preserving-compression.md

@@ -0,0 +1,64 @@
+# Topology-Preserving Compression（拓扑保持压缩）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当压缩表示时需要同构保持数据的本质拓扑结构（连通性、环、空腔）时使用：隐空间压缩（环形流形不能压成线段）、知识蒸馏（学生-教师同调等价）、3D 网格简化（genus 不变）、KV-Cache 语义保持（聚类结构不坍缩）。核心诉求：**压缩维度或数量，保证持续同调的 persistence diagram 变化可控**。
+
+## 数学思想来源
+- 透镜：lenses/topological.md（拓扑不变量——连通性、洞数在连续变形下不变）、lenses/spectral.md（Gauss-Bonnet 连接曲率与欧拉示性数）、lenses/variational.md（压缩率 vs. 拓扑保真度的约束优化）
+- 知识：knowledge-base/topology/persistent-homology.md（持续同调、Vietoris-Rips 滤流、Bottleneck 距离）、knowledge-base/topology/euler-characteristic.md（欧拉示性数快速拓扑诊断）、knowledge-base/matrix-analysis/matrix-perturbation.md（Davis-Kahan 子空间扰动界）
+
+## 需要的数学知识
+- **持续同调稳定性定理**：$d_B(D(X), D(Y)) \leq d_{GH}(X, Y)$，Hausdorff 距离界定 persistence diagram 变化
+- **欧拉示性数曲线**：$\chi(\epsilon) = \sum_k (-1)^k \beta_k(\epsilon)$，比单一 $\chi$ 信息更丰富，计算 $O(N^2)$
+- **映射柱同构**：若 $f: X \to Y$ 是 $\epsilon$-等距，则 $f_*$ 在持续区间 $> 2\epsilon$ 的特征上同构
+- **Landmark 近似**：用 witness complex 在 $m$ 个 landmark 上构建，$O(m^3)$ 替代 $O(N^3)$
+
+## AI 模块形式
+```
+模块：TopologyPreservingCompressor
+输入：X ∈ R^{N×d}    参数：拓扑权重 λ_topo，目标维度 r < d
+
+方法1 - Euler curve 匹配（最实用，可微）：
+  Z = encoder(X)                              // R^{N×r}
+  D_orig = cdist(X, X); D_comp = cdist(Z, Z)  // N×N 距离矩阵
+  chi_orig = euler_curve(D_orig, eps_grid)      // |eps_grid| 维向量
+  chi_comp = euler_curve(D_comp, eps_grid)
+  L_topo = ‖chi_orig - chi_comp‖_2²            // 拓扑匹配损失（可微）
+  L_total = L_recon + λ_topo · L_topo
+
+方法2 - 持续同调正则（精确但昂贵）：
+  D_orig = persistent_homology(X, max_dim=1)    // H_0 + H_1 barcode
+  D_comp = persistent_homology(Z, max_dim=1)
+  // 可微代理：persistence landscape/image
+  L_topo = ‖landscape(D_orig) - landscape(D_comp)‖_2²
+
+方法3 - 拓扑监控 + 自适应压缩率（推理时）：
+  Z = compress(X, ratio=ρ)
+  if count_components(Z, τ) < 0.8 * count_components(X, τ):
+    ρ *= 1.2; Z = compress(X, ratio=ρ)          // 拓扑坍缩 → 降低压缩比
+```
+
+## 可实现结构
+- **Euler curve 匹配层**：$\chi(\epsilon)$ 只需距离矩阵 + 阈值计数，$\epsilon$ 扫描可并行
+- **Landmark 采样器**：FPS（最远点采样）$O(Nm)$，保证覆盖
+- **拓扑感知蒸馏**：教师-学生表示的 persistence image 差异作为附加蒸馏目标
+- **拓扑诊断仪表板**：训练中实时绘制 $\beta_0(\epsilon), \beta_1(\epsilon)$ 曲线
+
+## GPU 可行性
+- 张量化/GEMM：距离矩阵 `cdist` 核心为 GEMM；$\chi(\epsilon)$ 为阈值计数 + 累积和
+- 复杂度：完整持续同调 $O(N^3)$ 不可行；Euler curve $O(N^2 |\epsilon|)$ 可行；Landmark $O(m^3)$
+- 显存：$N \times N$ 距离矩阵在 $N > 8K$ 时需分块或 Landmark 降维
+- 低精度：距离计算在 bf16 下稳定（正数加法）；$\chi$ 为整数运算无精度问题
+- 并行：$\epsilon$ 扫描的每个阈值独立并行；Landmark 选择可批并行
+- 算子融合：cdist + threshold + count 可融合避免物化大距离矩阵
+
+## 论文表述方式
+"以持续同调的 Bottleneck 稳定性定理为理论基础，通过 Euler characteristic curve 匹配实现 $O(N^2)$ 复杂度的拓扑保真正则化，保证压缩后 Betti 数在持续区间上的偏差受 Hausdorff 距离控制。"
+
+## 风险
+- **持续同调计算瓶颈**：精确边界矩阵约化高度串行（$O(N^3)$），必须依赖 Euler curve 或 Landmark 近似
+- **Euler curve 信息退化**：$\chi = \sum(-1)^k \beta_k$ 将多阶 Betti 数压成单值，不同拓扑可共享同一 $\chi$
+- **拓扑 ≠ 语义**：拓扑保持不等于语义保持——两个语义不同的空间可能拓扑同构
+- **尺度参数敏感**：滤流的 $\epsilon$ 范围需手动设定，不同数据集差异大
+- **Landmark 采样偏差**：FPS 在高维空间中可能不均匀，导致拓扑估计偏差

package/design-patterns/loss/constraint-penalty.en.md

@@ -0,0 +1,67 @@
+# Constraint Penalty
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the design involves hard constraints (e.g., probability simplex, orthogonality, capacity limits, load balancing) but end-to-end training is required. Typical scenarios: (1) MoE routing probabilities must lie on the $K$-simplex with load balancing; (2) Expert activation count is constrained (top-k); (3) Subspace projection matrices must satisfy orthogonality $W^T W = I$; (4) Feature norms are bounded $\|z\| \leq R$. Core objective: **transform mathematical constraints into differentiable penalty terms integrated into gradient-based variational**.
+
+## Mathematical Inspiration
+- Lenses: lenses/variational.md (constrained variational, Lagrangian duality, KKT conditions), lenses/geometric.md (manifold projection)
+- Knowledge: knowledge-base/optimization/lagrangian-duality.md (augmented Lagrangian method, penalty function method), knowledge-base/matrix-analysis/projection.md (projection operators, constraint sets)
+
+## Required Mathematical Knowledge
+- **Penalty Function Method**: $\min f(x)$ s.t. $g(x)=0 \to \min f(x) + \rho/2 \cdot \|g(x)\|^2$ -- $\rho$ is gradually increased (exterior point method), driving constraint violation $\|g(x)\| \to 0$
+- **Augmented Lagrangian Method**: $\min f(x) + \lambda^T g(x) + \rho/2 \cdot \|g(x)\|^2$ -- introduces dual variable $\lambda$, alternating between updating $\lambda \leftarrow \lambda + \rho \cdot g(x)$ and optimizing $x$; converges better than pure penalty methods
+- **Projected Gradient Method**: $x_{k+1} = \text{Proj}_C(x_k - \alpha \nabla f(x_k))$ -- for simple constraint sets $C$ (e.g., simplex, sphere), closed-form projection formulas exist
+- **Barrier Function Method**: $\min f(x) - \mu \sum \log(-g_i(x))$ for inequality constraints $g_i(x) \leq 0$ -- as $\mu \to 0$, approaches the constrained optimum
+
+## AI Module Form
+```
+Module: ConstraintPenalty
+Input: constraint violations g(x) in R^m (equality constraints), h(x) in R^p (inequality constraints)
+
+Method 1 - Adaptive Penalty Function (most commonly used):
+  L_penalty = Sum_i rho_i/2 * g_i(x)^2  +  Sum_j rho_j/2 * max(0, h_j(x))^2
+  // rho updated dynamically: each epoch rho_i *= gamma (gamma=2~10) until constraints are satisfied
+  // Different constraints can have different rho values, adapted to violation severity
+
+Method 2 - Augmented Lagrangian Method (ALM):
+  L_ALM = lambda^T * g(x) + rho/2 * ||g(x)||^2
+  // lambda is a learnable parameter (nn.Parameter), updated via gradient ascent:
+  lambda.data += rho * g(x).detach()   // dual ascent step
+  // Converges faster than pure penalty, avoids rho -> infinity
+
+Method 3 - Softmax Projection onto Simplex (load balancing special case):
+  p = softmax(logits / tau)           // project onto Delta^{K-1}
+  L_balance = ||p - 1/K||^2           // uniformity penalty
+  // Or Switch Transformer auxiliary loss:
+  L_aux = K * Sum_k f_k * P_k         // f_k = allocation fraction, P_k = average probability
+
+Method 4 - Orthogonal Constraint Projection:
+  W_proj = W * (W^T W)^{-1/2}       // project onto Stiefel manifold via matrix square root inverse
+  // Or parameterize via Cayley transform: W = (I-A)(I+A)^{-1} * W_0, A is skew-symmetric
+```
+
+## Implementable Architectures
+- **Loss Wrapper**: ConstraintLoss(base_loss, constraints, rho_schedule) -- forward computes base_loss + Sum constraint.penalty()
+- **Dual Variable Management**: nn.Parameter stores lambda; negative learning rate set in the optimizer to achieve gradient ascent
+- **Warm-up Strategy**: Optimize only base_loss for the first $N$ steps, then progressively activate constraint penalties
+- **Constraint Monitoring**: Record $\|g(x)\|$ at each step for visualization and adaptive rho adjustment
+
+## GPU Feasibility
+- **Tensorization**: Constraint violations are vector/matrix operations; penalty terms are element-wise squared sums
+- **GEMM-mappability**: Load balancing $f_k$, $P_k$ computation uses softmax + reduce_sum; orthogonal constraint uses matmul
+- **Complexity**: Penalty computation is $O(m)$ or $O(m^2)$, far smaller than the main network forward pass; negligible
+- **Memory & KV-Cache**: Only additional storage for lambda ($m$ dimensions) and rho ($m$ dimensions); minimal overhead
+- **Low Precision Stability**: Penalty terms are squaring operations, safe under fp16; ALM lambda updates are recommended in fp32 to avoid accumulated errors
+- **Parallelism & Communication**: Constraints are independently computable and parallelizable; on multiple GPUs, lambda updates require all-reduce of $g(x)$
+- **Sparse Structure**: $\max(0, h(x))^2$ has zero gradient when constraints are satisfied, naturally sparse activation
+- **Operator Fusion**: Constraint computation + weighted summation + merging with base_loss can be fused into a single kernel
+
+## Paper Phrasing
+"We employ the augmented Lagrangian method to convert hard constraints $g(x)=0$ into differentiable penalty terms $\lambda^T g(x) + \rho/2 \|g(x)\|^2$. Through alternating dual ascent updates of $\lambda$, constraint satisfaction is guaranteed to converge at rate $O(1/\rho)$ without requiring $\rho \to \infty$, reducing constraint violations by 3x compared to pure penalty methods in experiments."
+
+## Risks
+- Excessively fast rho growth leads to ill-conditioned variational landscapes (deteriorating condition numbers), causing gradient vanishing or explosion
+- ALM lambda update frequency and step size require tuning; too fast causes oscillation, too slow impedes convergence
+- When multiple constraints coexist, the relative ratios of their rho values influence the variational trajectory
+- Projection operations (e.g., matrix square root inverse) are computationally expensive and numerically unstable

package/design-patterns/loss/constraint-penalty.md

@@ -0,0 +1,78 @@
+# Constraint Penalty（约束惩罚损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当设计中有硬约束（如概率单纯形、正交性、容量限制、负载均衡）但需要端到端训练时使用。
+典型场景：(1) MoE 路由概率必须在 K-simplex 上且负载均衡；(2) 专家激活数量受限（top-k）；
+(3) 子空间投影矩阵必须满足正交性 W^TW = I；(4) 特征范数有界 ‖z‖ ≤ R。
+核心诉求：**将数学约束转化为可微惩罚项，融入梯度优化流程**。
+
+## 数学思想来源
+- 透镜：lenses/variational.md（约束优化、拉格朗日对偶、KKT 条件）、lenses/geometric.md（流形投影）
+- 知识：knowledge-base/optimization/lagrangian-duality.md（增广拉格朗日法、惩罚函数法）、
+  knowledge-base/matrix-analysis/projection.md（投影算子、约束集）
+
+## 需要的数学知识
+- **罚函数法**：min f(x) s.t. g(x)=0 → min f(x) + ρ/2 · ‖g(x)‖²
+  ρ 逐步增大（外点法），约束违反量 ‖g(x)‖ → 0
+- **增广拉格朗日法**：min f(x) + λ^T g(x) + ρ/2 · ‖g(x)‖²
+  引入对偶变量 λ，交替更新 λ ← λ + ρ·g(x) 和优化 x，比纯罚函数收敛更好
+- **投影梯度法**：x_{k+1} = Proj_C(x_k - α∇f(x_k))
+  对简单约束集 C（如 simplex、球面）有闭合投影公式
+- **屏障函数法**：min f(x) - μ Σ log(-g_i(x))  对不等式约束 g_i(x) ≤ 0
+  μ → 0 时逼近约束最优解
+
+## AI 模块形式
+```
+模块：ConstraintPenalty
+输入：约束违反量 g(x) ∈ R^m（等式约束），h(x) ∈ R^p（不等式约束）
+
+方法1 - 自适应罚函数（最常用）：
+  L_penalty = Σ_i ρ_i/2 · g_i(x)²  +  Σ_j ρ_j/2 · max(0, h_j(x))²
+  // ρ 动态更新：每个 epoch ρ_i *= γ（γ=2~10）直到约束满足
+  // 不同约束可有不同的 ρ，按违反程度自适应
+
+方法2 - 增广拉格朗日（ALM）：
+  L_ALM = λ^T · g(x) + ρ/2 · ‖g(x)‖²
+  // λ 为可学习参数（nn.Parameter），通过梯度上升更新：
+  λ.data += ρ * g(x).detach()   // 对偶上升步
+  // 比纯罚函数收敛快，避免 ρ → ∞
+
+方法3 - Softmax 投影到单纯形（负载均衡特例）：
+  p = softmax(logits / τ)           // 投影到 Δ^{K-1}
+  L_balance = ‖p - 1/K‖²            // 均匀性惩罚
+  // 或 Switch Transformer 的辅助损失：
+  L_aux = K · Σ_k f_k · P_k         // f_k = 分配比例, P_k = 平均概率
+
+方法4 - 正交约束投影：
+  W_proj = W · (W^T W)^{-1/2}       // 通过矩阵平方根反投影到 Stiefel 流形
+  // 或用 Cayley 变换参数化：W = (I-A)(I+A)^{-1} · W_0, A 为反对称矩阵
+```
+
+## 可实现结构
+- **Loss Wrapper**：ConstraintLoss(base_loss, constraints, ρ_schedule)
+  forward 时计算 base_loss + Σ constraint.penalty()
+- **对偶变量管理**：nn.Parameter 存储 λ，optimizer 中为 λ 设置负学习率实现梯度上升
+- **warm-up 策略**：前 N 步只优化 base_loss，之后逐步激活约束惩罚
+- **约束监控**：每 step 记录 ‖g(x)‖ 用于可视化和自适应 ρ 调整
+
+## GPU 可行性
+- **张量化**：约束违反量为向量/矩阵运算，罚项为 element-wise 平方和
+- **GEMM 可映射**：负载均衡的 f_k, P_k 计算为 softmax + reduce_sum；正交约束为 matmul
+- **复杂度**：罚项计算 O(m) 或 O(m²)，远小于主网络前向，可忽略
+- **显存与 KV-Cache**：仅额外存储 λ (m 维) 和 ρ (m 维)，极小开销
+- **低精度稳定**：罚项为平方运算，fp16 安全；ALM 的 λ 更新建议 fp32 避免累积误差
+- **并行与通信**：各约束独立计算，可并行；多 GPU 时 λ 更新需 all-reduce g(x)
+- **稀疏结构**：max(0, h(x))² 在约束满足时梯度为零，天然稀疏激活
+- **算子融合**：constraint 计算 + 加权求和 + 与 base_loss 合并可融合为单 kernel
+
+## 论文表述方式
+"采用增广拉格朗日法将硬约束 g(x)=0 转化为可微惩罚项 λ^Tg(x) + ρ/2‖g(x)‖²，
+通过对偶变量 λ 的交替上升更新，在无需 ρ→∞ 的条件下保证约束满足度以 O(1/ρ) 收敛，
+相比纯罚函数法在实验中减少 3× 的约束违反。"
+
+## 风险
+- ρ 增大过快导致优化 landscape 病态（条件数恶化），梯度消失或爆炸
+- ALM 的 λ 更新频率和步长需调参，更新过快震荡、过慢收敛
+- 多个约束同时存在时 ρ 的相对比例影响优化路径
+- 投影操作（如矩阵平方根逆）计算代价高且数值不稳定

package/design-patterns/loss/contrastive-loss.en.md

@@ -0,0 +1,59 @@
+# Contrastive Loss
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the model needs to learn "what is similar to what and what is different from what." Typical scenarios: (1) Different augmented views of the same input should be pulled closer (positive pairs), while different inputs should be pushed apart (negative pairs); (2) Shared representations should capture cross-task commonalities, while Private representations should distinguish task-specific features; (3) In the expert embedding space, similar inputs should be routed to the same expert. Core objective: **learn relative relationships rather than absolute values**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (metric spaces and distance functions), lenses/probabilistic.md (mutual information maximization)
+- Knowledge: knowledge-base/probability/entropy.md (conditional distributions and likelihood), knowledge-base/differential-geometry/manifold.md (geodesics and curvature)
+
+## Required Mathematical Knowledge
+- **InfoNCE Loss**: $L = -\log[\exp(\text{sim}(q,k^+)/\tau) / \sum_j \exp(\text{sim}(q,k_j)/\tau)]$ -- essentially a lower-bound estimate of mutual information in contrastive learning; $\tau$ is the temperature parameter controlling distribution sharpness
+- **Margin-Based Metric Learning**: Triplet Loss $= \max(0, d(a,p) - d(a,n) + \text{margin})$ -- explicitly widens the distance gap between positive and negative pairs in metric space
+- **NT-Xent (Normalized Temperature-scaled Cross Entropy)**: Softmax contrastive loss on the unit sphere $S^{d-1}$, with normalization eliminating scale effects
+- **Debiased Contrastive Learning**: Corrects for false negatives in negative samples, using prior $\tau^+$ to estimate the true negative sample distribution
+
+## AI Module Form
+```
+Module: ContrastiveLoss
+Input: anchors z_a in R^{B x d}, positives z_p in R^{B x d}, negative pool z_n in R^{N x d}
+
+Core formula (InfoNCE + temperature scaling):
+  sim(q, k) = q^T k / (||q|| * ||k||)       // cosine similarity
+  logits_i = [sim(z_a_i, z_p_i)] (+) [sim(z_a_i, z_n_j)]_{j=1}^N  // concatenation
+  L_contrast = -1/B * Sum_i log( exp(logits_i[0]/tau) / Sum_j exp(logits_i[j]/tau) )
+
+Queue mechanism (MoCo style):
+  z_n = FIFO_queue.enqueue(z_p.detach())   // negative sample queue, capacity N >> B
+  // Queue stores encodings from historical batches, increasing negative count without additional memory
+
+Hard Negative Mining:
+  top-k indices = argsort(sim(z_a, z_n), descending=True)[:k]
+  z_n_hard = z_n[top-k indices]            // retain only the k hardest negatives
+```
+
+## Implementable Architectures
+- **Dual-Tower Encoder + Projection Head**: encoder -> projection_head (2-layer MLP) -> normalize -> loss
+- **Negative Sample Queue**: Maintain a FIFO queue of momentum encoder outputs, capacity N=65536
+- **Symmetric Loss**: $L = L(a \to p) + L(p \to a)$, swapping positive/negative roles for enhanced training stability
+- **Multi-Granularity Contrast**: Apply contrastive objectives simultaneously at token-level, sequence-level, and expert-level
+
+## GPU Feasibility
+- **Tensorization**: Similarity computation is $z_a @ z_n^T$ -- standard GEMM $(B \times d) @ (d \times N) = B \times N$
+- **GEMM-mappability**: Core computation is 1-2 matrix multiplications, perfectly mapped to cuBLAS
+- **Complexity**: $O(B \cdot N \cdot d)$ computation + $O(B \cdot N)$ storage for the logits matrix; approximately 64MB when B=256, N=65536
+- **Memory & KV-Cache**: Negative sample queue occupies $N \cdot d \cdot 4$ bytes, approximately 65536 * 256 * 4 = 64MB, fixed overhead
+- **Low Precision Stability**: Cosine similarity + softmax under fp16 requires caution for exp overflow; use log-sum-exp trick
+- **Parallelism & Communication**: On multiple GPUs, use all-gather to collect negatives from other GPUs to enlarge $N$ (MoCo v3 strategy)
+- **Sparse Structure**: After hard negative mining, only $k \ll N$ negatives are retained, effectively sparsifying the logits
+- **Operator Fusion**: L2-norm -> matmul -> scale -> log-softmax -> nll_loss can be fused
+
+## Paper Phrasing
+"We employ temperature-scaled InfoNCE contrastive loss with a momentum encoder maintaining a negative sample queue of N=65536, maximizing the lower bound of positive-pair mutual information on the unit sphere. Theoretical analysis shows this lower bound converges at rate $O(1/\sqrt{N})$."
+
+## Risks
+- Too small $\tau$ causes training instability (excessively large gradients); too large $\tau$ makes all samples indistinguishable (degenerates to uniform distribution)
+- Stale encodings in the negative sample queue introduce representation bias
+- False negative problem: unsupervised negative sampling may select samples that are semantically similar but differently labeled
+- When $B$ is too small, the positive/negative pair imbalance within the batch must be compensated by the queue

package/design-patterns/loss/contrastive-loss.md

@@ -0,0 +1,68 @@
+# Contrastive Loss（对比损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要让模型学会"什么和什么相似、什么和什么不同"时使用。典型场景：
+(1) 同一输入的不同增强视图应拉近（正对），不同输入应推远（负对）；
+(2) Shared 表示应包含跨任务共性，Private 表示应区分任务特异性；
+(3) 专家嵌入空间中相似输入应路由到同一专家。核心诉求：**学习相对关系而非绝对值**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（度量空间与距离函数）、lenses/probabilistic.md（互信息最大化）
+- 知识：knowledge-base/probability/entropy.md（条件分布与似然）、
+  knowledge-base/differential-geometry/manifold.md（测地线与曲率）
+
+## 需要的数学知识
+- **InfoNCE 损失**：L = -log[exp(sim(q,k⁺)/τ) / Σ_j exp(sim(q,k_j)/τ)]
+  本质是对比学习中的互信息下界估计，τ 为温度参数控制分布锐度
+- **Margin-based 度量学习**：Triplet Loss = max(0, d(a,p) - d(a,n) + margin)
+  在度量空间中显式拉开正负对距离差
+- **NT-Xent (Normalized Temperature-scaled Cross Entropy)**：
+  在单位球面 S^{d-1} 上的 softmax 对比，归一化消除尺度影响
+- **去偏对比学习 (Debiased Contrastive)**：
+  修正负样本中的假阴性问题，使用先验 τ⁺ 估计真实负样本分布
+
+## AI 模块形式
+```
+模块：ContrastiveLoss
+输入：锚点 z_a ∈ R^{B×d}，正样本 z_p ∈ R^{B×d}，负样本库 z_n ∈ R^{N×d}
+
+核心公式 (InfoNCE + 温度缩放)：
+  sim(q, k) = q^T k / (‖q‖ · ‖k‖)       // cosine 相似度
+  logits_i = [sim(z_a_i, z_p_i)] ⊕ [sim(z_a_i, z_n_j)]_{j=1}^N  // 拼接
+  L_contrast = -1/B · Σ_i log( exp(logits_i[0]/τ) / Σ_j exp(logits_i[j]/τ) )
+
+Queue 机制（MoCo 风格）:
+  z_n = FIFO_queue.enqueue(z_p.detach())   // 负样本队列，容量 N >> B
+  // 队列中存储的是历史 batch 的编码，增大负样本数而不增加显存
+
+Hard Negative Mining:
+  top-k indices = argsort(sim(z_a, z_n), descending=True)[:k]
+  z_n_hard = z_n[top-k indices]            // 只保留最难的 k 个负样本
+```
+
+## 可实现结构
+- **双塔编码器 + 投影头**：encoder → projection_head(MLP 2层) → 归一化 → loss
+- **负样本队列**：维护 momentum encoder 输出的 FIFO queue，容量 N=65536
+- **对称损失**：L = L(a→p) + L(p→a)，正负角色互换，增强训练稳定性
+- **多粒度对比**：同时在 token-level、sequence-level、expert-level 施加对比
+
+## GPU 可行性
+- **张量化**：sim 计算为 z_a @ z_n^T → 标准 GEMM (B×d) @ (d×N) = B×N
+- **GEMM 可映射**：核心就是 1-2 次矩阵乘法，完美映射 cuBLAS
+- **复杂度**：O(B·N·d) 计算 + O(B·N) 存储 logits 矩阵，B=256,N=65536 时约 64MB
+- **显存与 KV-Cache**：负样本队列占 N·d·4 bytes ≈ 65536·256·4 = 64MB，固定开销
+- **低精度稳定**：cosine 相似度 + softmax 在 fp16 下需注意 exp 溢出，用 log-sum-exp trick
+- **并行与通信**：多 GPU 时用 all-gather 收集其他 GPU 的负样本扩大 N（MoCo v3 策略）
+- **稀疏结构**：hard negative mining 后只保留 k<<N 个负样本，有效稀疏化 logits
+- **算子融合**：L2-norm → matmul → scale → log-softmax → nll_loss 可融合
+
+## 论文表述方式
+"采用温度缩放的 InfoNCE 对比损失，通过动量编码器维护 N=65536 的负样本队列，
+在单位球面上最大化正对互信息的下界，理论分析表明该下界以 O(1/√N) 收敛。"
+
+## 风险
+- τ 过小导致训练不稳定（梯度过大），τ 过大导致所有样本不分难易（退化均匀分布）
+- 负样本队列中的过期编码引入 stale representation 偏差
+- 假阴性问题：无监督负采样可能采到语义相似但标注不同的样本
+- B 过小时 batch 内正负对不平衡，需依赖队列补偿

package/design-patterns/loss/information-bottleneck-loss.en.md

@@ -0,0 +1,69 @@
+# Information Bottleneck Loss
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When a representation $Z$ must achieve an optimal balance between "retaining task-relevant information" and "compressing input redundancy." Typical scenarios: (1) Shared representations should retain only cross-task common information, discarding task-specific noise; (2) Private representations should retain only single-task unique information; (3) Routing features should maximize expert-task matching information. Core objective: **optimal information compression -- nothing more, nothing less, retaining only what is useful**.
+
+## Mathematical Inspiration
+- Lenses: lenses/probabilistic.md (information bottleneck principle, mutual information variational), lenses/variational.md (Lagrangian duality)
+- Knowledge: knowledge-base/probability/kl-divergence.md (IB theory, rate-distortion function), knowledge-base/probability/entropy.md (mutual information and conditional entropy)
+
+## Required Mathematical Knowledge
+- **Information Bottleneck Objective**: $\min I(X;Z) - \beta \cdot I(Z;Y)$, compressing $X \to Z$ while preserving the predictive power of $Z$ for $Y$
+- **Variational Bounds on Mutual Information**:
+  $I(X;Z) \leq \mathbb{E}_{p(x,z)}[\log q(z|x)] - \mathbb{E}_{p(z)}[\log q(z)]$ (upper bound for compression term)
+  $I(Z;Y) \geq \mathbb{E}_{p(z,y)}[\log q(y|z)] + H(Y)$ (lower bound for prediction term)
+- **CPC (Contrastive Predictive Coding)**: InfoNCE lower bound on $I(Z_t; Z_{t+k})$
+- **MINE (Mutual Information Neural Estimation)**:
+  $I(X;Z) = \sup_\theta \{ \mathbb{E}[\log T_\theta(x,z)] - \log \mathbb{E}[T_\theta(x,z')] \}$
+
+## AI Module Form
+```
+Module: InformationBottleneckLoss
+Input: representation Z in R^{B x d}, input X (or its encoding), label Y
+
+Method 1 - VIB (Variational Information Bottleneck):
+  // Compression upper bound: variational approximation q(z) = N(0, I)
+  I_upper = KL(q(z|x) || p(z))  // standard VAE KL term
+  // Prediction lower bound: classifier/regressor q(y|z)
+  I_lower = CE(q(y|z), y)  // cross-entropy = estimate of -H(Y|Z)
+  L_IB = I_upper + beta * I_lower
+  // beta controls compression-prediction trade-off: larger beta = more aggressive compression
+
+Method 2 - Contrastive Mutual Information Estimation (no distributional assumptions):
+  // NWJ estimator instead of KL
+  I_nwj(x;z) = E[f(x,z)] - exp(E[f(x,z')] - 1)  // f is a discriminator network
+  L_IB_contrast = I_nwj(x;z) - beta * InfoNCE(z, y)  // both terms differentiable
+
+Method 3 - Shared/Private IB Decomposition:
+  Z_s = enc_shared(x), Z_p = enc_private(x)
+  L = I(Z_s; X) + I(Z_p; X)           // total compression
+    - beta_1 * I(Z_s; Y_common)        // Shared retains common information
+    - beta_2 * I(Z_p; Y_specific)      // Private retains specific information
+    + gamma * OrthLoss(Z_s, Z_p)       // orthogonality ensures decomposition
+```
+
+## Implementable Architectures
+- **Dual-Encoder Architecture**: enc_shared and enc_private share a base trunk, branching into separate heads
+- **Mutual Information Estimator**: Small MLP discriminator $T(x,z) \to$ scalar, alternating variational with the main network
+- **Beta Scheduling**: Set $\beta = 0$ at training start (no compression), gradually increase to target value during training
+- **Gradient Reversal**: Gradient of $I(X;Z)$ is reversed via z.flip_gradient(), implementing adversarial compression
+
+## GPU Feasibility
+- **Tensorization**: Mutual information estimator is a standard MLP -> GEMM chain; KL is element-wise
+- **GEMM-mappability**: VIB method requires only encoder GEMM + KL computation; contrastive method adds 1 GEMM for shuffled negatives
+- **Complexity**: One additional KL term $O(B \cdot d)$ or one discriminator forward pass $O(B \cdot d^2)$ beyond the standard network; acceptable
+- **Memory & KV-Cache**: Requires additional storage for discriminator parameters (small MLP) and intermediate activations, <10MB
+- **Low Precision Stability**: The exp operation in the MINE estimator requires clipping under fp16; VIB KL is recommended in fp32
+- **Parallelism & Communication**: Discriminator and main network forward passes can run in parallel; gradients are synchronized through the shared representation layer
+- **Sparse Structure**: Compressed $Z$ dimensions can be dynamically pruned (Automatic Relevance Determination, ARD)
+- **Operator Fusion**: Encoder forward + KL computation + discriminator forward can be partially fused
+
+## Paper Phrasing
+"Based on information bottleneck theory, we formalize Shared/Private decomposition as $\min I(X;Z_s) + I(X;Z_p) - \beta_1 I(Z_s;Y_c) - \beta_2 I(Z_p;Y_s)$, replacing mutual information terms with variational upper and lower bounds for end-to-end variational, theoretically guaranteeing beta-optimality on the compression-prediction Pareto frontier."
+
+## Risks
+- Mutual information estimators (MINE/NWJ) have high variance, causing training instability; large batches or moving averages are needed
+- Improper beta selection leads to over-compression (underfitting) or insufficient compression (overfitting)
+- VIB assumes Gaussian posteriors, which may be inadequate for complex posterior distributions
+- When optimizing multiple IB objectives jointly, the relative ratio of $\beta_1$ and $\beta_2$ is sensitive

package/design-patterns/loss/information-bottleneck-loss.md

@@ -0,0 +1,74 @@
+# Information Bottleneck Loss（信息瓶颈损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要让表示 Z 在"保留任务相关信息"与"压缩输入冗余信息"之间取得最优平衡时使用。
+典型场景：(1) Shared 表示应只保留跨任务共性信息，丢弃任务特异噪声；
+(2) Private 表示应只保留单任务独有信息；(3) 路由特征应最大化专家-任务匹配信息。
+核心诉求：**信息最优压缩——不多不少，只保留有用的**。
+
+## 数学思想来源
+- 透镜：lenses/probabilistic.md（信息瓶颈原理、互信息优化）、lenses/variational.md（拉格朗日对偶）
+- 知识：knowledge-base/probability/kl-divergence.md（IB 理论、率失真函数）、
+  knowledge-base/probability/entropy.md（互信息与条件熵）
+
+## 需要的数学知识
+- **信息瓶颈目标**：min I(X;Z) - β·I(Z;Y)，压缩 X→Z 同时保留 Z 对 Y 的预测力
+- **互信息变分下界/上界**：
+  I(X;Z) ≤ E_{p(x,z)}[log q(z|x)] - E_{p(z)}[log q(z)]  (用上界估计压缩项)
+  I(Z;Y) ≥ E_{p(z,y)}[log q(y|z)] + H(Y)  (用下界估计预测项)
+- **CPC (Contrastive Predictive Coding)**：I(Z_t; Z_{t+k}) 的 InfoNCE 下界
+- **MINE (Mutual Information Neural Estimation)**：
+  I(X;Z) = sup_θ { E[log T_θ(x,z)] - log E[T_θ(x,z')] }
+
+## AI 模块形式
+```
+模块：InformationBottleneckLoss
+输入：表示 Z ∈ R^{B×d}，输入 X（或其编码），标签 Y
+
+方法1 - VIB (Variational Information Bottleneck)：
+  // 压缩项上界：用变分近似 q(z) = N(0, I)
+  I_upper = KL(q(z|x) ‖ p(z))  // 标准 VAE 的 KL 项
+  // 预测项下界：用分类器/回归器 q(y|z)
+  I_lower = CE(q(y|z), y)  // 交叉熵 = -H(Y|Z) 的估计
+  L_IB = I_upper + β · I_lower
+  // β 控制压缩-预测权衡：β↑ 更激进压缩，β↓ 保留更多预测信息
+
+方法2 - 对比式互信息估计（无需分布假设）：
+  // 用 NWJ 估计器替代 KL
+  I_nwj(x;z) = E[f(x,z)] - exp(E[f(x,z')] - 1)  // f 为判别网络
+  L_IB_contrast = I_nwj(x;z) - β · InfoNCE(z, y)  // 两项均可微
+
+方法3 - Shared/Private IB 分解：
+  Z_s = enc_shared(x), Z_p = enc_private(x)
+  L = I(Z_s; X) + I(Z_p; X)           // 总压缩
+    - β₁ · I(Z_s; Y_common)            // Shared 保留公共信息
+    - β₂ · I(Z_p; Y_specific)          // Private 保留特异信息
+    + γ · OrthLoss(Z_s, Z_p)           // 正交性确保分解
+```
+
+## 可实现结构
+- **双编码器架构**：enc_shared 和 enc_private 共享底层 trunk，分叉出各自 head
+- **互信息估计器**：小型 MLP 判别器 T(x,z) → scalar，与主网络交替优化
+- **β 调度**：训练初期 β=0（不压缩），随训练进行逐步增大到目标值
+- **梯度反转**：I(X;Z) 的梯度通过 z.flip_gradient() 反转，实现对抗式压缩
+
+## GPU 可行性
+- **张量化**：互信息估计器为标准 MLP → GEMM 链；KL 为 element-wise
+- **GEMM 可映射**：VIB 方法仅需 encoder GEMM + KL 计算；对比方法额外 1 次 GEMM 做 shuffle 负样本
+- **复杂度**：比标准网络多 1 个 KL 项 O(B·d) 或 1 个判别器前向 O(B·d²)，可接受
+- **显存与 KV-Cache**：需额外存储判别器参数（小型 MLP）和中间激活，<10MB
+- **低精度稳定**：MINE 估计器的 exp 运算在 fp16 下需 clip；VIB 的 KL 建议 fp32
+- **并行与通信**：判别器与主网络可并行前向，梯度通过共享表示层同步
+- **稀疏结构**：压缩后的 Z 维度可动态裁剪（自动相关性确定 ARD）
+- **算子融合**：encoder 前向 + KL 计算 + 判别器前向可部分融合
+
+## 论文表述方式
+"基于信息瓶颈理论，将 Shared/Private 分解形式化为 min I(X;Z_s)+I(X;Z_p)-β₁I(Z_s;Y_c)-β₂I(Z_p;Y_s)，
+通过变分上下界替代互信息项实现端到端优化，理论上保证压缩-预测 Pareto 前沿的 β-最优性。"
+
+## 风险
+- 互信息估计（MINE/NWJ）方差大，训练不稳定，需要大 batch 或 moving average
+- β 选择不当导致过度压缩（欠拟合）或压缩不足（过拟合）
+- VIB 假设高斯后验，对复杂后验分布近似不足
+- 多 IB 联合优化时 β₁, β₂ 的相对比例敏感

package/design-patterns/loss/orthogonality-loss.en.md

@@ -0,0 +1,56 @@
+# Orthogonality Loss
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+In multi-expert / multi-task settings, representations learned by submodules are highly overlapping and redundant, leading to poor parameter utilization. This loss is used when the $d$-dimensional feature space needs to be partitioned into $K$ non-interfering subspaces -- such as Shared-Private separation, MoE expert differentiation, and multi-task head decorrelation. Core objective: **ensure different modules see different things**.
+
+## Mathematical Inspiration
+- Lenses: lenses/projection.md (orthogonal projection and subspace decomposition), lenses/variational.md (regularization and saddle points)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (spectral theorem, SVD, Schur decomposition), knowledge-base/probability/kl-divergence.md (redundancy and mutual information)
+
+## Required Mathematical Knowledge
+- **Frobenius Inner Product and Orthogonality**: $\langle A, B \rangle_F = \text{tr}(A^T B)$; when $\langle A, B \rangle_F = 0$, $A \perp B$
+- **Stiefel Manifold Constraint**: $W \in \text{St}(d, k)$, i.e., $W^T W = I_k$, projecting onto the set of orthogonal matrices
+- **DPP (Determinantal Point Process)**: $\det(W^T W)$ increases as column vectors become more spread out, serving as a diversity proxy
+- **Off-Diagonal Elements of the Cosine Similarity Matrix**: $C_{ij} = |\langle w_i, w_j \rangle| / (\|w_i\|\|w_j\|)$, minimizing $\sum_{i \neq j} C_{ij}^2$
+
+## AI Module Form
+```
+Module: OrthogonalDiversityLoss
+Input: K feature matrices {W_k in R^{d x r}}_{k=1}^K (weights or features of K submodules)
+
+Method 1 - Frobenius Orthogonal Regularization:
+  L_orth = Sum_{i<j} ||W_i^T W_j||_F^2
+  // Computation: O(K^2 * d * r^2), K typically <16 so cost is manageable
+
+Method 2 - Grassmann Distance (based on principal angles):
+  sigma_k = singular values of SVD(W_i^T W_j)
+  L_grass = Sum_{i<j} Sum_k sigma_k^2 * (1 - sigma_k^2)  // penalizes singular values that are neither 0 nor 1
+
+Method 3 - Efficient Cosine Decorrelation:
+  G = concat([W_1,...,W_K])^T * concat([W_1,...,W_K])  // single GEMM
+  L_corr = ||G * (1 - I)||_F^2   // mask out diagonal, penalize off-diagonal elements
+```
+
+## Implementable Architectures
+- **Embedded as nn.Module**: forward receives $K$ tensors and returns a scalar loss; supports direct .backward()
+- **Weighted Combination with Main Loss**: L_total = L_task + lambda * L_orth; lambda can use cosine annealing or warm-up
+- **Block Computation**: When $K$ is large, perform mini-batch sampling over $(i,j)$ pairs, computing only $B$ out of $\binom{K}{2}$ pairs per step
+
+## GPU Feasibility
+- **Tensorization**: Core operation is matmul ($W^T W$) -- standard GEMM, perfectly mapped to Tensor Cores
+- **GEMM-mappability**: Method 3 requires only 1 GEMM + 1 element-wise mask + Frobenius norm
+- **Complexity**: $O(K \cdot d \cdot r)$ storage + $O(d \cdot r^2 \cdot K)$ or $O(K^2 \cdot d \cdot r^2)$ computation; negligible when $K < 16$
+- **Memory & KV-Cache**: Intermediate matrices are on the order of $d \times r \cdot K$, adding no KV-Cache overhead
+- **Low Precision Stability**: Frobenius norm is a sum of squares, safe under fp16; Grassmann SVD is recommended in fp32
+- **Parallelism & Communication**: The $K$ pairs are embarrassingly parallel; can be distributed across GPUs with all-reduce
+- **Sparse Structure**: If $W_k$ is itself sparse (e.g., MoE gate), masking further increases sparsity
+- **Operator Fusion**: matmul -> mask -> square -> sum can be fused into a single CUDA kernel
+
+## Paper Phrasing
+"We introduce an orthogonality regularizer L_orth = Sum_{i<j} ||W_i^T W_j||_F^2 that constrains the feature spaces of submodules to the approximately orthogonal Grassmann sub-manifold, theoretically guaranteeing that redundancy across $K$ subspaces decays as $O(1/\sqrt{d})$."
+
+## Risks
+- Excessively large lambda causes variational difficulties (orthogonal constraint conflicts with task objective); requires careful tuning or adaptive lambda
+- The SVD in Method 2 produces unstable gradients during backpropagation; epsilon-regularization on singular values is needed
+- When $K \cdot r > d$, strict orthogonality is impossible; dimensionality reduction or acceptance of approximate orthogonality is required

package/design-patterns/loss/orthogonality-loss.md

@@ -0,0 +1,60 @@
+# Orthogonality Loss（正交性损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+多专家/多任务场景中，各子模块学习到的表示高度重叠、冗余，导致参数利用率低下。
+当需要将 d 维特征空间分割为 K 个互不干扰的子空间时使用——如 Shared-Private 分离、
+MoE 专家差异化、多任务 head 去相关。核心诉求：**让不同模块看到不同的东西**。
+
+## 数学思想来源
+- 透镜：lenses/projection.md（正交投影与子空间分解）、lenses/variational.md（正则化与鞍点）
+- 知识：knowledge-base/matrix-analysis/projection.md（谱定理、SVD、Schur 分解）、
+  knowledge-base/probability/kl-divergence.md（冗余度与互信息）
+
+## 需要的数学知识
+- **Frobenius 内积与正交性**：⟨A, B⟩_F = tr(A^T B)，当 ⟨A, B⟩_F = 0 时 A⊥B
+- **Stiefel 流形约束**：W ∈ St(d, k) 即 W^T W = I_k，投影到正交矩阵集合
+- **DPP 行列式点过程**：det(W^T W) 越大表示列向量越分散，可作为多样性代理
+- **cosine 相似度矩阵去对角外元素**：C_ij = |⟨w_i, w_j⟩| / (‖w_i‖‖w_j‖)，最小化 Σ_{i≠j} C_ij²
+
+## AI 模块形式
+```
+模块：OrthogonalDiversityLoss
+输入：K 个特征矩阵 {W_k ∈ R^{d×r}}_{k=1}^K（K 个子模块的权重或特征）
+
+方法1 - Frobenius 正交正则：
+  L_orth = Σ_{i<j} ‖W_i^T W_j‖_F²
+  // 计算量：O(K² · d · r²)，K 一般 <16 所以可控
+
+方法2 - Grassmann 距离（基于主角度）：
+  σ_k = SVD(W_i^T W_j) 的奇异值
+  L_grass = Σ_{i<j} Σ_k σ_k²(1 - σ_k²)  // 惩罚非 0 非 1 的奇异值
+
+方法3 - 高效 cosine 去相关：
+  G = concat([W_1,...,W_K])^T · concat([W_1,...,W_K])  // 一次 GEMM
+  L_corr = ‖G ⊙ (1 - I)‖_F²   // mask 掉对角线，惩罚非对角元素
+```
+
+## 可实现结构
+- **嵌入为 nn.Module**：forward 接收 K 个 tensor，返回标量 loss，可直接 .backward()
+- **与主 loss 加权组合**：L_total = L_task + λ · L_orth，λ 可用 cosine annealing 或 warm-up
+- **分块计算**：当 K 很大时，对 (i,j) 对做 mini-batch 采样，每步只算 C(K,2) 中的 B 对
+
+## GPU 可行性
+- **张量化**：核心操作为 matmul（W^T W）→ 标准 GEMM，完美映射 Tensor Core
+- **GEMM 可映射**：方法3 只需 1 次 GEMM + 1 次 element-wise mask + Frobenius 范数
+- **复杂度**：O(K·d·r) 存储 + O(d·r²·K) 或 O(K²·d·r²) 计算，K<16 时 negligible
+- **显存与 KV-Cache**：中间矩阵 d×r·K 量级，不增加 KV-Cache 负担
+- **低精度稳定**：Frobenius 范数为平方和，fp16 下 OK；Grassmann SVD 建议 fp32
+- **并行与通信**：K 对之间 embarrassingly parallel，可分 GPU 计算后 all-reduce
+- **稀疏结构**：若 W_k 本身稀疏（如 MoE gate），mask 后稀疏度进一步提升
+- **算子融合**：matmul → mask → square → sum 可融合为单个 CUDA kernel
+
+## 论文表述方式
+"我们引入正交性正则项 L_orth = Σ_{i<j}‖W_i^T W_j‖_F²，将各子模块的特征空间约束
+到近似正交的 Grassmann 子流形上，理论上保证 K 个子空间的冗余度以 O(1/√d) 衰减。"
+
+## 风险
+- λ 过大导致优化困难（正交约束与任务目标冲突），需 careful tuning 或 adaptive λ
+- 方法2 的 SVD 在反向传播时梯度不稳定，需添加 ε-正则化到奇异值
+- 当 K·r > d 时正交性不可能严格满足，需降维或接受近似正交