math-skill 2.0.1 → 3.0.0

package/design-patterns/loss/contrastive-loss.md

@@ -0,0 +1,68 @@
+# Contrastive Loss（对比损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要让模型学会"什么和什么相似、什么和什么不同"时使用。典型场景：
+(1) 同一输入的不同增强视图应拉近（正对），不同输入应推远（负对）；
+(2) Shared 表示应包含跨任务共性，Private 表示应区分任务特异性；
+(3) 专家嵌入空间中相似输入应路由到同一专家。核心诉求：**学习相对关系而非绝对值**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（度量空间与距离函数）、lenses/probabilistic.md（互信息最大化）
+- 知识：knowledge-base/probability/entropy.md（条件分布与似然）、
+  knowledge-base/differential-geometry/manifold.md（测地线与曲率）
+
+## 需要的数学知识
+- **InfoNCE 损失**：L = -log[exp(sim(q,k⁺)/τ) / Σ_j exp(sim(q,k_j)/τ)]
+  本质是对比学习中的互信息下界估计，τ 为温度参数控制分布锐度
+- **Margin-based 度量学习**：Triplet Loss = max(0, d(a,p) - d(a,n) + margin)
+  在度量空间中显式拉开正负对距离差
+- **NT-Xent (Normalized Temperature-scaled Cross Entropy)**：
+  在单位球面 S^{d-1} 上的 softmax 对比，归一化消除尺度影响
+- **去偏对比学习 (Debiased Contrastive)**：
+  修正负样本中的假阴性问题，使用先验 τ⁺ 估计真实负样本分布
+
+## AI 模块形式
+```
+模块：ContrastiveLoss
+输入：锚点 z_a ∈ R^{B×d}，正样本 z_p ∈ R^{B×d}，负样本库 z_n ∈ R^{N×d}
+
+核心公式 (InfoNCE + 温度缩放)：
+  sim(q, k) = q^T k / (‖q‖ · ‖k‖)       // cosine 相似度
+  logits_i = [sim(z_a_i, z_p_i)] ⊕ [sim(z_a_i, z_n_j)]_{j=1}^N  // 拼接
+  L_contrast = -1/B · Σ_i log( exp(logits_i[0]/τ) / Σ_j exp(logits_i[j]/τ) )
+
+Queue 机制（MoCo 风格）:
+  z_n = FIFO_queue.enqueue(z_p.detach())   // 负样本队列，容量 N >> B
+  // 队列中存储的是历史 batch 的编码，增大负样本数而不增加显存
+
+Hard Negative Mining:
+  top-k indices = argsort(sim(z_a, z_n), descending=True)[:k]
+  z_n_hard = z_n[top-k indices]            // 只保留最难的 k 个负样本
+```
+
+## 可实现结构
+- **双塔编码器 + 投影头**：encoder → projection_head(MLP 2层) → 归一化 → loss
+- **负样本队列**：维护 momentum encoder 输出的 FIFO queue，容量 N=65536
+- **对称损失**：L = L(a→p) + L(p→a)，正负角色互换，增强训练稳定性
+- **多粒度对比**：同时在 token-level、sequence-level、expert-level 施加对比
+
+## GPU 可行性
+- **张量化**：sim 计算为 z_a @ z_n^T → 标准 GEMM (B×d) @ (d×N) = B×N
+- **GEMM 可映射**：核心就是 1-2 次矩阵乘法，完美映射 cuBLAS
+- **复杂度**：O(B·N·d) 计算 + O(B·N) 存储 logits 矩阵，B=256,N=65536 时约 64MB
+- **显存与 KV-Cache**：负样本队列占 N·d·4 bytes ≈ 65536·256·4 = 64MB，固定开销
+- **低精度稳定**：cosine 相似度 + softmax 在 fp16 下需注意 exp 溢出，用 log-sum-exp trick
+- **并行与通信**：多 GPU 时用 all-gather 收集其他 GPU 的负样本扩大 N（MoCo v3 策略）
+- **稀疏结构**：hard negative mining 后只保留 k<<N 个负样本，有效稀疏化 logits
+- **算子融合**：L2-norm → matmul → scale → log-softmax → nll_loss 可融合
+
+## 论文表述方式
+"采用温度缩放的 InfoNCE 对比损失，通过动量编码器维护 N=65536 的负样本队列，
+在单位球面上最大化正对互信息的下界，理论分析表明该下界以 O(1/√N) 收敛。"
+
+## 风险
+- τ 过小导致训练不稳定（梯度过大），τ 过大导致所有样本不分难易（退化均匀分布）
+- 负样本队列中的过期编码引入 stale representation 偏差
+- 假阴性问题：无监督负采样可能采到语义相似但标注不同的样本
+- B 过小时 batch 内正负对不平衡，需依赖队列补偿

package/design-patterns/loss/information-bottleneck-loss.en.md

@@ -0,0 +1,69 @@
+# Information Bottleneck Loss
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When a representation $Z$ must achieve an optimal balance between "retaining task-relevant information" and "compressing input redundancy." Typical scenarios: (1) Shared representations should retain only cross-task common information, discarding task-specific noise; (2) Private representations should retain only single-task unique information; (3) Routing features should maximize expert-task matching information. Core objective: **optimal information compression -- nothing more, nothing less, retaining only what is useful**.
+
+## Mathematical Inspiration
+- Lenses: lenses/probabilistic.md (information bottleneck principle, mutual information variational), lenses/variational.md (Lagrangian duality)
+- Knowledge: knowledge-base/probability/kl-divergence.md (IB theory, rate-distortion function), knowledge-base/probability/entropy.md (mutual information and conditional entropy)
+
+## Required Mathematical Knowledge
+- **Information Bottleneck Objective**: $\min I(X;Z) - \beta \cdot I(Z;Y)$, compressing $X \to Z$ while preserving the predictive power of $Z$ for $Y$
+- **Variational Bounds on Mutual Information**:
+  $I(X;Z) \leq \mathbb{E}_{p(x,z)}[\log q(z|x)] - \mathbb{E}_{p(z)}[\log q(z)]$ (upper bound for compression term)
+  $I(Z;Y) \geq \mathbb{E}_{p(z,y)}[\log q(y|z)] + H(Y)$ (lower bound for prediction term)
+- **CPC (Contrastive Predictive Coding)**: InfoNCE lower bound on $I(Z_t; Z_{t+k})$
+- **MINE (Mutual Information Neural Estimation)**:
+  $I(X;Z) = \sup_\theta \{ \mathbb{E}[\log T_\theta(x,z)] - \log \mathbb{E}[T_\theta(x,z')] \}$
+
+## AI Module Form
+```
+Module: InformationBottleneckLoss
+Input: representation Z in R^{B x d}, input X (or its encoding), label Y
+
+Method 1 - VIB (Variational Information Bottleneck):
+  // Compression upper bound: variational approximation q(z) = N(0, I)
+  I_upper = KL(q(z|x) || p(z))  // standard VAE KL term
+  // Prediction lower bound: classifier/regressor q(y|z)
+  I_lower = CE(q(y|z), y)  // cross-entropy = estimate of -H(Y|Z)
+  L_IB = I_upper + beta * I_lower
+  // beta controls compression-prediction trade-off: larger beta = more aggressive compression
+
+Method 2 - Contrastive Mutual Information Estimation (no distributional assumptions):
+  // NWJ estimator instead of KL
+  I_nwj(x;z) = E[f(x,z)] - exp(E[f(x,z')] - 1)  // f is a discriminator network
+  L_IB_contrast = I_nwj(x;z) - beta * InfoNCE(z, y)  // both terms differentiable
+
+Method 3 - Shared/Private IB Decomposition:
+  Z_s = enc_shared(x), Z_p = enc_private(x)
+  L = I(Z_s; X) + I(Z_p; X)           // total compression
+    - beta_1 * I(Z_s; Y_common)        // Shared retains common information
+    - beta_2 * I(Z_p; Y_specific)      // Private retains specific information
+    + gamma * OrthLoss(Z_s, Z_p)       // orthogonality ensures decomposition
+```
+
+## Implementable Architectures
+- **Dual-Encoder Architecture**: enc_shared and enc_private share a base trunk, branching into separate heads
+- **Mutual Information Estimator**: Small MLP discriminator $T(x,z) \to$ scalar, alternating variational with the main network
+- **Beta Scheduling**: Set $\beta = 0$ at training start (no compression), gradually increase to target value during training
+- **Gradient Reversal**: Gradient of $I(X;Z)$ is reversed via z.flip_gradient(), implementing adversarial compression
+
+## GPU Feasibility
+- **Tensorization**: Mutual information estimator is a standard MLP -> GEMM chain; KL is element-wise
+- **GEMM-mappability**: VIB method requires only encoder GEMM + KL computation; contrastive method adds 1 GEMM for shuffled negatives
+- **Complexity**: One additional KL term $O(B \cdot d)$ or one discriminator forward pass $O(B \cdot d^2)$ beyond the standard network; acceptable
+- **Memory & KV-Cache**: Requires additional storage for discriminator parameters (small MLP) and intermediate activations, <10MB
+- **Low Precision Stability**: The exp operation in the MINE estimator requires clipping under fp16; VIB KL is recommended in fp32
+- **Parallelism & Communication**: Discriminator and main network forward passes can run in parallel; gradients are synchronized through the shared representation layer
+- **Sparse Structure**: Compressed $Z$ dimensions can be dynamically pruned (Automatic Relevance Determination, ARD)
+- **Operator Fusion**: Encoder forward + KL computation + discriminator forward can be partially fused
+
+## Paper Phrasing
+"Based on information bottleneck theory, we formalize Shared/Private decomposition as $\min I(X;Z_s) + I(X;Z_p) - \beta_1 I(Z_s;Y_c) - \beta_2 I(Z_p;Y_s)$, replacing mutual information terms with variational upper and lower bounds for end-to-end variational, theoretically guaranteeing beta-optimality on the compression-prediction Pareto frontier."
+
+## Risks
+- Mutual information estimators (MINE/NWJ) have high variance, causing training instability; large batches or moving averages are needed
+- Improper beta selection leads to over-compression (underfitting) or insufficient compression (overfitting)
+- VIB assumes Gaussian posteriors, which may be inadequate for complex posterior distributions
+- When optimizing multiple IB objectives jointly, the relative ratio of $\beta_1$ and $\beta_2$ is sensitive

package/design-patterns/loss/information-bottleneck-loss.md

@@ -0,0 +1,74 @@
+# Information Bottleneck Loss（信息瓶颈损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要让表示 Z 在"保留任务相关信息"与"压缩输入冗余信息"之间取得最优平衡时使用。
+典型场景：(1) Shared 表示应只保留跨任务共性信息，丢弃任务特异噪声；
+(2) Private 表示应只保留单任务独有信息；(3) 路由特征应最大化专家-任务匹配信息。
+核心诉求：**信息最优压缩——不多不少，只保留有用的**。
+
+## 数学思想来源
+- 透镜：lenses/probabilistic.md（信息瓶颈原理、互信息优化）、lenses/variational.md（拉格朗日对偶）
+- 知识：knowledge-base/probability/kl-divergence.md（IB 理论、率失真函数）、
+  knowledge-base/probability/entropy.md（互信息与条件熵）
+
+## 需要的数学知识
+- **信息瓶颈目标**：min I(X;Z) - β·I(Z;Y)，压缩 X→Z 同时保留 Z 对 Y 的预测力
+- **互信息变分下界/上界**：
+  I(X;Z) ≤ E_{p(x,z)}[log q(z|x)] - E_{p(z)}[log q(z)]  (用上界估计压缩项)
+  I(Z;Y) ≥ E_{p(z,y)}[log q(y|z)] + H(Y)  (用下界估计预测项)
+- **CPC (Contrastive Predictive Coding)**：I(Z_t; Z_{t+k}) 的 InfoNCE 下界
+- **MINE (Mutual Information Neural Estimation)**：
+  I(X;Z) = sup_θ { E[log T_θ(x,z)] - log E[T_θ(x,z')] }
+
+## AI 模块形式
+```
+模块：InformationBottleneckLoss
+输入：表示 Z ∈ R^{B×d}，输入 X（或其编码），标签 Y
+
+方法1 - VIB (Variational Information Bottleneck)：
+  // 压缩项上界：用变分近似 q(z) = N(0, I)
+  I_upper = KL(q(z|x) ‖ p(z))  // 标准 VAE 的 KL 项
+  // 预测项下界：用分类器/回归器 q(y|z)
+  I_lower = CE(q(y|z), y)  // 交叉熵 = -H(Y|Z) 的估计
+  L_IB = I_upper + β · I_lower
+  // β 控制压缩-预测权衡：β↑ 更激进压缩，β↓ 保留更多预测信息
+
+方法2 - 对比式互信息估计（无需分布假设）：
+  // 用 NWJ 估计器替代 KL
+  I_nwj(x;z) = E[f(x,z)] - exp(E[f(x,z')] - 1)  // f 为判别网络
+  L_IB_contrast = I_nwj(x;z) - β · InfoNCE(z, y)  // 两项均可微
+
+方法3 - Shared/Private IB 分解：
+  Z_s = enc_shared(x), Z_p = enc_private(x)
+  L = I(Z_s; X) + I(Z_p; X)           // 总压缩
+    - β₁ · I(Z_s; Y_common)            // Shared 保留公共信息
+    - β₂ · I(Z_p; Y_specific)          // Private 保留特异信息
+    + γ · OrthLoss(Z_s, Z_p)           // 正交性确保分解
+```
+
+## 可实现结构
+- **双编码器架构**：enc_shared 和 enc_private 共享底层 trunk，分叉出各自 head
+- **互信息估计器**：小型 MLP 判别器 T(x,z) → scalar，与主网络交替优化
+- **β 调度**：训练初期 β=0（不压缩），随训练进行逐步增大到目标值
+- **梯度反转**：I(X;Z) 的梯度通过 z.flip_gradient() 反转，实现对抗式压缩
+
+## GPU 可行性
+- **张量化**：互信息估计器为标准 MLP → GEMM 链；KL 为 element-wise
+- **GEMM 可映射**：VIB 方法仅需 encoder GEMM + KL 计算；对比方法额外 1 次 GEMM 做 shuffle 负样本
+- **复杂度**：比标准网络多 1 个 KL 项 O(B·d) 或 1 个判别器前向 O(B·d²)，可接受
+- **显存与 KV-Cache**：需额外存储判别器参数（小型 MLP）和中间激活，<10MB
+- **低精度稳定**：MINE 估计器的 exp 运算在 fp16 下需 clip；VIB 的 KL 建议 fp32
+- **并行与通信**：判别器与主网络可并行前向，梯度通过共享表示层同步
+- **稀疏结构**：压缩后的 Z 维度可动态裁剪（自动相关性确定 ARD）
+- **算子融合**：encoder 前向 + KL 计算 + 判别器前向可部分融合
+
+## 论文表述方式
+"基于信息瓶颈理论，将 Shared/Private 分解形式化为 min I(X;Z_s)+I(X;Z_p)-β₁I(Z_s;Y_c)-β₂I(Z_p;Y_s)，
+通过变分上下界替代互信息项实现端到端优化，理论上保证压缩-预测 Pareto 前沿的 β-最优性。"
+
+## 风险
+- 互信息估计（MINE/NWJ）方差大，训练不稳定，需要大 batch 或 moving average
+- β 选择不当导致过度压缩（欠拟合）或压缩不足（过拟合）
+- VIB 假设高斯后验，对复杂后验分布近似不足
+- 多 IB 联合优化时 β₁, β₂ 的相对比例敏感

package/design-patterns/loss/orthogonality-loss.en.md

@@ -0,0 +1,56 @@
+# Orthogonality Loss
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+In multi-expert / multi-task settings, representations learned by submodules are highly overlapping and redundant, leading to poor parameter utilization. This loss is used when the $d$-dimensional feature space needs to be partitioned into $K$ non-interfering subspaces -- such as Shared-Private separation, MoE expert differentiation, and multi-task head decorrelation. Core objective: **ensure different modules see different things**.
+
+## Mathematical Inspiration
+- Lenses: lenses/projection.md (orthogonal projection and subspace decomposition), lenses/variational.md (regularization and saddle points)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (spectral theorem, SVD, Schur decomposition), knowledge-base/probability/kl-divergence.md (redundancy and mutual information)
+
+## Required Mathematical Knowledge
+- **Frobenius Inner Product and Orthogonality**: $\langle A, B \rangle_F = \text{tr}(A^T B)$; when $\langle A, B \rangle_F = 0$, $A \perp B$
+- **Stiefel Manifold Constraint**: $W \in \text{St}(d, k)$, i.e., $W^T W = I_k$, projecting onto the set of orthogonal matrices
+- **DPP (Determinantal Point Process)**: $\det(W^T W)$ increases as column vectors become more spread out, serving as a diversity proxy
+- **Off-Diagonal Elements of the Cosine Similarity Matrix**: $C_{ij} = |\langle w_i, w_j \rangle| / (\|w_i\|\|w_j\|)$, minimizing $\sum_{i \neq j} C_{ij}^2$
+
+## AI Module Form
+```
+Module: OrthogonalDiversityLoss
+Input: K feature matrices {W_k in R^{d x r}}_{k=1}^K (weights or features of K submodules)
+
+Method 1 - Frobenius Orthogonal Regularization:
+  L_orth = Sum_{i<j} ||W_i^T W_j||_F^2
+  // Computation: O(K^2 * d * r^2), K typically <16 so cost is manageable
+
+Method 2 - Grassmann Distance (based on principal angles):
+  sigma_k = singular values of SVD(W_i^T W_j)
+  L_grass = Sum_{i<j} Sum_k sigma_k^2 * (1 - sigma_k^2)  // penalizes singular values that are neither 0 nor 1
+
+Method 3 - Efficient Cosine Decorrelation:
+  G = concat([W_1,...,W_K])^T * concat([W_1,...,W_K])  // single GEMM
+  L_corr = ||G * (1 - I)||_F^2   // mask out diagonal, penalize off-diagonal elements
+```
+
+## Implementable Architectures
+- **Embedded as nn.Module**: forward receives $K$ tensors and returns a scalar loss; supports direct .backward()
+- **Weighted Combination with Main Loss**: L_total = L_task + lambda * L_orth; lambda can use cosine annealing or warm-up
+- **Block Computation**: When $K$ is large, perform mini-batch sampling over $(i,j)$ pairs, computing only $B$ out of $\binom{K}{2}$ pairs per step
+
+## GPU Feasibility
+- **Tensorization**: Core operation is matmul ($W^T W$) -- standard GEMM, perfectly mapped to Tensor Cores
+- **GEMM-mappability**: Method 3 requires only 1 GEMM + 1 element-wise mask + Frobenius norm
+- **Complexity**: $O(K \cdot d \cdot r)$ storage + $O(d \cdot r^2 \cdot K)$ or $O(K^2 \cdot d \cdot r^2)$ computation; negligible when $K < 16$
+- **Memory & KV-Cache**: Intermediate matrices are on the order of $d \times r \cdot K$, adding no KV-Cache overhead
+- **Low Precision Stability**: Frobenius norm is a sum of squares, safe under fp16; Grassmann SVD is recommended in fp32
+- **Parallelism & Communication**: The $K$ pairs are embarrassingly parallel; can be distributed across GPUs with all-reduce
+- **Sparse Structure**: If $W_k$ is itself sparse (e.g., MoE gate), masking further increases sparsity
+- **Operator Fusion**: matmul -> mask -> square -> sum can be fused into a single CUDA kernel
+
+## Paper Phrasing
+"We introduce an orthogonality regularizer L_orth = Sum_{i<j} ||W_i^T W_j||_F^2 that constrains the feature spaces of submodules to the approximately orthogonal Grassmann sub-manifold, theoretically guaranteeing that redundancy across $K$ subspaces decays as $O(1/\sqrt{d})$."
+
+## Risks
+- Excessively large lambda causes variational difficulties (orthogonal constraint conflicts with task objective); requires careful tuning or adaptive lambda
+- The SVD in Method 2 produces unstable gradients during backpropagation; epsilon-regularization on singular values is needed
+- When $K \cdot r > d$, strict orthogonality is impossible; dimensionality reduction or acceptance of approximate orthogonality is required

package/design-patterns/loss/orthogonality-loss.md

@@ -0,0 +1,60 @@
+# Orthogonality Loss（正交性损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+多专家/多任务场景中，各子模块学习到的表示高度重叠、冗余，导致参数利用率低下。
+当需要将 d 维特征空间分割为 K 个互不干扰的子空间时使用——如 Shared-Private 分离、
+MoE 专家差异化、多任务 head 去相关。核心诉求：**让不同模块看到不同的东西**。
+
+## 数学思想来源
+- 透镜：lenses/projection.md（正交投影与子空间分解）、lenses/variational.md（正则化与鞍点）
+- 知识：knowledge-base/matrix-analysis/projection.md（谱定理、SVD、Schur 分解）、
+  knowledge-base/probability/kl-divergence.md（冗余度与互信息）
+
+## 需要的数学知识
+- **Frobenius 内积与正交性**：⟨A, B⟩_F = tr(A^T B)，当 ⟨A, B⟩_F = 0 时 A⊥B
+- **Stiefel 流形约束**：W ∈ St(d, k) 即 W^T W = I_k，投影到正交矩阵集合
+- **DPP 行列式点过程**：det(W^T W) 越大表示列向量越分散，可作为多样性代理
+- **cosine 相似度矩阵去对角外元素**：C_ij = |⟨w_i, w_j⟩| / (‖w_i‖‖w_j‖)，最小化 Σ_{i≠j} C_ij²
+
+## AI 模块形式
+```
+模块：OrthogonalDiversityLoss
+输入：K 个特征矩阵 {W_k ∈ R^{d×r}}_{k=1}^K（K 个子模块的权重或特征）
+
+方法1 - Frobenius 正交正则：
+  L_orth = Σ_{i<j} ‖W_i^T W_j‖_F²
+  // 计算量：O(K² · d · r²)，K 一般 <16 所以可控
+
+方法2 - Grassmann 距离（基于主角度）：
+  σ_k = SVD(W_i^T W_j) 的奇异值
+  L_grass = Σ_{i<j} Σ_k σ_k²(1 - σ_k²)  // 惩罚非 0 非 1 的奇异值
+
+方法3 - 高效 cosine 去相关：
+  G = concat([W_1,...,W_K])^T · concat([W_1,...,W_K])  // 一次 GEMM
+  L_corr = ‖G ⊙ (1 - I)‖_F²   // mask 掉对角线，惩罚非对角元素
+```
+
+## 可实现结构
+- **嵌入为 nn.Module**：forward 接收 K 个 tensor，返回标量 loss，可直接 .backward()
+- **与主 loss 加权组合**：L_total = L_task + λ · L_orth，λ 可用 cosine annealing 或 warm-up
+- **分块计算**：当 K 很大时，对 (i,j) 对做 mini-batch 采样，每步只算 C(K,2) 中的 B 对
+
+## GPU 可行性
+- **张量化**：核心操作为 matmul（W^T W）→ 标准 GEMM，完美映射 Tensor Core
+- **GEMM 可映射**：方法3 只需 1 次 GEMM + 1 次 element-wise mask + Frobenius 范数
+- **复杂度**：O(K·d·r) 存储 + O(d·r²·K) 或 O(K²·d·r²) 计算，K<16 时 negligible
+- **显存与 KV-Cache**：中间矩阵 d×r·K 量级，不增加 KV-Cache 负担
+- **低精度稳定**：Frobenius 范数为平方和，fp16 下 OK；Grassmann SVD 建议 fp32
+- **并行与通信**：K 对之间 embarrassingly parallel，可分 GPU 计算后 all-reduce
+- **稀疏结构**：若 W_k 本身稀疏（如 MoE gate），mask 后稀疏度进一步提升
+- **算子融合**：matmul → mask → square → sum 可融合为单个 CUDA kernel
+
+## 论文表述方式
+"我们引入正交性正则项 L_orth = Σ_{i<j}‖W_i^T W_j‖_F²，将各子模块的特征空间约束
+到近似正交的 Grassmann 子流形上，理论上保证 K 个子空间的冗余度以 O(1/√d) 衰减。"
+
+## 风险
+- λ 过大导致优化困难（正交约束与任务目标冲突），需 careful tuning 或 adaptive λ
+- 方法2 的 SVD 在反向传播时梯度不稳定，需添加 ε-正则化到奇异值
+- 当 K·r > d 时正交性不可能严格满足，需降维或接受近似正交

package/design-patterns/loss/variational-loss.en.md

@@ -0,0 +1,61 @@
+# Variational Loss
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When sampling from latent variable distributions is required to generate diverse outputs. Typical scenarios: (1) Expert selection introduces discrete latent variables $z$ that need end-to-end variational; (2) Representation spaces need to model uncertainty; (3) Generative routing requires sampling from posterior distributions $p(z|x)$; (4) Bayesian mixture of experts. Core objective: **model distributions rather than point estimates in latent space, enabling uncertainty awareness and diversity**.
+
+## Mathematical Inspiration
+- Lenses: lenses/probabilistic.md (variational inference and ELBO), lenses/probabilistic.md (posterior and prior)
+- Knowledge: knowledge-base/probability/entropy.md (KL divergence, variational families), knowledge-base/probability/kl-divergence.md (ELBO derivation)
+
+## Required Mathematical Knowledge
+- **ELBO (Evidence Lower Bound)**: $\log p(x) \geq \mathbb{E}_{q(z|x)}[\log p(x|z)] - \text{KL}(q(z|x) \| p(z))$ -- the first term is reconstruction likelihood, the second regularizes the posterior toward the prior
+- **Reparameterization Trick**: $z = \mu + \sigma \odot \varepsilon$, $\varepsilon \sim \mathcal{N}(0, I)$ -- enables gradient backpropagation through sampling operations
+- **Closed-Form KL Divergence**: When both $q$ and $p$ are Gaussian, $\text{KL}(\mathcal{N}(\mu,\sigma^2) \| \mathcal{N}(0,1)) = -\frac{1}{2} \sum(1 + \log \sigma^2 - \mu^2 - \sigma^2)$
+- **Gumbel-Softmax (Discrete Latent Variables)**: $z = \text{softmax}((\log \pi + g) / \tau)$, $g \sim \text{Gumbel}(0,1)$ -- continuous relaxation of discrete sampling
+
+## AI Module Form
+```
+Module: VariationalLoss
+Input: encoder outputs (mu, log_sigma^2) in R^{B x d_z}, reconstruction output x_hat in R^{B x d_x}, original input x
+
+Method 1 - Gaussian VAE Loss:
+  KL = -0.5 * sum(1 + log_sigma^2 - mu^2 - exp(log_sigma^2), dim=-1)   // closed form
+  recon = -log p(x|x_hat)  // MSE or BCE depending on data distribution
+  L_vae = recon + beta * KL    // beta-VAE controls disentanglement
+
+Method 2 - Gumbel-Softmax (discrete expert selection):
+  logits = encoder(x)  in R^{B x K}    // logits for K experts
+  g = -log(-log(uniform(B x K) + eps) + eps)   // Gumbel noise sampling
+  z_soft = softmax((logits + g) / tau)       // temperature tau annealed from 1.0 to 0.1
+  L_gumbel = CE(task_head(z_soft * features), y)  // straight-through estimator backprop
+
+Method 3 - Variational Information Bottleneck:
+  L_IB = -I(X; Z) + beta * I(Z; Y)  // maximize MI between Z and X, minimize redundancy between Z and Y
+  ~ E[-log q(x|z)] + beta * KL(q(z|x) || p(z))  // variational approximation
+```
+
+## Implementable Architectures
+- **Dual-Head Encoder Output**: Linear(d, 2 * d_z) -> split -> (mu, log_sigma^2), sharing base parameters
+- **Beta Annealing Strategy**: Linearly increase beta from 0 to the target value to prevent KL collapse (posterior collapse)
+- **Free Bits**: Set a KL lower bound lambda per dimension, penalizing only the excess: $\sum \max(\text{KL}_j, \lambda)$
+- **IWAE Multi-Particle**: Use log-mean-exp over $K$ samples instead of single-sample ELBO for a tighter lower bound
+
+## GPU Feasibility
+- **Tensorization**: Computation of mu and sigma^2 is a Linear layer (GEMM); KL involves element-wise operations
+- **GEMM-mappability**: Encoder 1 GEMM -> split -> reparameterization -> decoder 1 GEMM
+- **Complexity**: Same order as standard feedforward networks $O(B \cdot d^2)$; KL computation $O(B \cdot d_z)$ is negligible
+- **Memory & KV-Cache**: Additional storage of mu and sigma^2, two $B \times d_z$ matrices; minimal overhead
+- **Low Precision Stability**: log/exp operations in KL are recommended in fp32; Gumbel softmax log-log requires fp32
+- **Parallelism & Communication**: The $K$ samples in multi-particle IWAE can be sampled and computed in parallel
+- **Sparse Structure**: Discrete latent variables (Gumbel) degenerate to one-hot as $\tau \to 0$, naturally sparse
+- **Operator Fusion**: The Linear layers for mu and sigma^2 can share a single GEMM followed by split; KL exp/sub/add can be fused
+
+## Paper Phrasing
+"We adopt the variational inference framework, geometric expert selection as posterior inference over discrete latent variables $z$. End-to-end variational is achieved through Gumbel-Softmax relaxation, combined with a beta annealing strategy that effectively prevents posterior collapse. The ELBO lower bound converges at rate $O(\sqrt{d/n})$ with respect to latent variable dimensionality."
+
+## Risks
+- Posterior collapse: The KL term converges to 0 prematurely, causing latent variables to degenerate into prior samples and lose information
+- Gumbel-Softmax temperature $\tau$ annealing requires careful scheduling; too fast causes gradient vanishing, too slow loses discreteness
+- IWAE multi-particle log-mean-exp is numerically unstable in high dimensions; log-sum-exp trick is needed
+- Excessively large beta in beta-VAE degrades reconstruction quality; task-specific balancing is required

package/design-patterns/loss/variational-loss.md

@@ -0,0 +1,71 @@
+# Variational Loss（变分损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要从隐变量分布中采样并生成多样化输出时使用。典型场景：
+(1) 专家选择引入离散隐变量 z，需要端到端优化；(2) 表示空间需要建模不确定性；
+(3) 生成式路由中需要从后验分布 p(z|x) 采样；(4) 贝叶斯专家混合。
+核心诉求：**在潜空间中建模分布而非点估计，获得不确定性感知与多样性**。
+
+## 数学思想来源
+- 透镜：lenses/probabilistic.md（变分推断与 ELBO）、lenses/probabilistic.md（后验与先验）
+- 知识：knowledge-base/probability/entropy.md（KL 散度、变分族）、
+  knowledge-base/probability/kl-divergence.md（ELBO 推导）
+
+## 需要的数学知识
+- **ELBO（证据下界）**：log p(x) ≥ E_{q(z|x)}[log p(x|z)] - KL(q(z|x) ‖ p(z))
+  第一项为重构似然，第二项为正则项将后验推近先验
+- **重参数化技巧 (Reparameterization Trick)**：z = μ + σ ⊙ ε, ε ~ N(0, I)
+  使梯度可通过采样操作反向传播
+- **KL 散度闭合形式**：当 q 和 p 均为高斯时，
+  KL(N(μ,σ²) ‖ N(0,1)) = -½ Σ(1 + log σ² - μ² - σ²)
+- **Gumbel-Softmax（离散隐变量）**：
+  z = softmax((log π + g) / τ), g ~ Gumbel(0,1)，连续松弛离散采样
+
+## AI 模块形式
+```
+模块：VariationalLoss
+输入：编码器输出 (μ, log_σ²) ∈ R^{B×d_z}，重构输出 x̂ ∈ R^{B×d_x}，原始输入 x
+
+方法1 - 高斯 VAE 损失：
+  KL = -0.5 * sum(1 + log_σ² - μ² - exp(log_σ²), dim=-1)   // 闭合形式
+  recon = -log p(x|x̂)  // MSE 或 BCE 取决于数据分布
+  L_vae = recon + β · KL    // β-VAE 控制解耦程度
+
+方法2 - Gumbel-Softmax（离散专家选择）：
+  logits = encoder(x)  ∈ R^{B×K}    // K 个专家的 logits
+  g = -log(-log(uniform(B×K) + ε) + ε)   // Gumbel 噪声采样
+  z_soft = softmax((logits + g) / τ)       // 温度 τ 从 1.0 退火到 0.1
+  L_gumbel = CE(task_head(z_soft ⊙ features), y)  // 直通估计器反向传播
+
+方法3 - 信息瓶颈变分：
+  L_IB = -I(X; Z) + β · I(Z; Y)  // 最大化 Z 与 X 的互信息，最小化 Z 与 Y 的冗余
+  ≈ E[-log q(x|z)] + β · KL(q(z|x) ‖ p(z))  // 变分近似
+```
+
+## 可实现结构
+- **编码器双头输出**：Linear(d, 2·d_z) → split → (μ, log_σ²)，共享底层参数
+- **β 退火策略**：β 从 0 线性增长到目标值，防止 KL 坍塌（posterior collapse）
+- **Free Bits**：每个维度设定 KL 下界 λ，只惩罚超出部分：Σ max(KL_j, λ)
+- **IWAE 多粒子**：用 K 个样本的 log-mean-exp 替代单样本 ELBO，获得更紧下界
+
+## GPU 可行性
+- **张量化**：μ,σ² 的计算为 Linear 层（GEMM），KL 为 element-wise 运算
+- **GEMM 可映射**：编码器 1 次 GEMM → split → 重参数化 → 解码器 1 次 GEMM
+- **复杂度**：与标准前向网络同阶 O(B·d²)，KL 计算 O(B·d_z) 可忽略
+- **显存与 KV-Cache**：额外存储 μ, σ² 两个 B×d_z 矩阵，开销极小
+- **低精度稳定**：KL 的 log/exp 运算建议 fp32；Gumbel softmax 的 log-log 需 fp32
+- **并行与通信**：多粒子 IWAE 的 K 个样本可并行采样和计算
+- **稀疏结构**：离散隐变量 (Gumbel) 在 τ→0 时退化为 one-hot，天然稀疏
+- **算子融合**：μ 和 σ² 的 Linear 可共享一次 GEMM 后 split；KL 的 exp/sub/add 可融合
+
+## 论文表述方式
+"采用变分推断框架，将专家选择建模为离散隐变量 z 的后验推断，通过 Gumbel-Softmax
+松弛实现端到端优化，配合 β 退火策略有效避免 posterior collapse，
+ELBO 下界随隐变量维度以 O(√d/n) 收敛。"
+
+## 风险
+- Posterior collapse：KL 项过早收敛到 0，隐变量退化为先验采样，丧失信息
+- Gumbel-Softmax 的 τ 退火需要精心调度，过快导致梯度消失，过慢丧失离散性
+- IWAE 多粒子在高维下 log-mean-exp 数值不稳定，需 log-sum-exp trick
+- β-VAE 的 β 过大导致重构质量下降，需根据任务平衡

package/design-patterns/representation/equivariant-split.en.md

@@ -0,0 +1,87 @@
+# Equivariant Split
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when the input possesses symmetries (e.g., permutations, rotations, translations) and the representation should preserve or reflect those symmetries.
+Typical scenarios: (1) Token permutation equivariance -- representation should change accordingly when token order in a sentence changes (positional encoding);
+(2) Symmetry-based grouping of feature dimensions -- certain feature dimensions are invariant under specific transformations, while others are equivariant;
+(3) Multi-expert symmetry specialization -- different experts process different symmetry subspaces;
+(4) Geometric deep learning -- SE(3) equivariance in 3D molecular/protein structures.
+Core requirement: **encode symmetry priors into network architecture to reduce learning burden and improve generalization**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (group actions, invariant/equivariant maps), lenses/probabilistic.md (symmetry and information redundancy)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (group representation theory, irreducible representations),
+  knowledge-base/differential-geometry/manifold.md (Lie groups, homogeneous spaces)
+
+## Required Mathematical Background
+- **Group Action and Equivariance**: A map f is equivariant with respect to group G if and only if f(g * x) = g * f(x) for all g in G
+  Invariance is a special case of equivariance (g * f(x) = f(x), i.e., the trivial representation)
+- **Schur's Lemma and Irreducible Representation Decomposition**:
+  Any finite group representation decomposes as a direct sum of irreducible representations: V = direct_sum_i m_i * V_i
+  Equivariant linear maps are diagonal/block-diagonal between irreducible components
+- **Peter-Weyl Theorem**: Functions on a compact group decompose as a series of irreducible representation matrix elements
+  f(x) = sum_rho sum_{ij} c_{rho,ij} * rho_{ij}(g) (generalized Fourier expansion)
+- **Steerable Feature Spaces**: Features are organized according to irreducible representations of the group;
+  under the action of duality g, each component transforms via the corresponding representation matrix: f_i -> sum_j rho_{ij}(g) f_j
+
+## AI Module Form
+```
+Module: EquivariantSplit
+Input: X in R^{N x d}, symmetry group G (e.g., S_n permutation group, Z_n cyclic group, SO(3) rotation group)
+
+Method 1 - Split feature dimensions by irreducible representations:
+  // Decompose d-dimensional features by irreducible representations of the group
+  irreps = decompose(G, d)  // [(d_1, rho_1), (d_2, rho_2), ...] where sum d_i = d
+  X_split = split(X, [d_1, d_2, ...], dim=-1)  // split by irreducible components
+  // Process each component independently with equivariant layers:
+  for (X_i, rho_i) in zip(X_split, irreps):
+    Y_i = EquivariantLinear(X_i, rho_i)  // weights constrained by Schur's lemma
+  Y = concat(Y_i, dim=-1)                // reassemble
+
+Method 2 - Positional equivariant split (Token permutation group S_n):
+  // Transformer self-attention is naturally permutation equivariant (without positional encoding)
+  // Explicitly introduce controllable permutation equivariance:
+  X_content = X[:, :d_content]            // permutation-invariant content part
+  X_position = X[:, d_content:]           // position-dependent part
+  // Content part uses permutation-invariant pooling:
+  z_inv = mean(X_content, dim=1)          // global invariant features
+  // Position part uses equivariant operations:
+  z_equiv = Attention(X_position, X_position, X_position)  // permutation equivariant
+  output = z_equiv + MLP(z_inv).unsqueeze(1)  // broadcast invariant signal back
+
+Method 3 - Group convolution / group pooling:
+  // Features defined on group G: f: G -> R^c
+  // Group convolution: (f * psi)(g) = sum_{h in G} f(h) * psi(h^{-1} g)
+  // Group pooling: pool over orbits of subgroup H < G
+  // Implemented as matrix multiplication (group multiplication table -> sparse permutation matrices)
+  for g in generators(G):
+    X_g = permutation_matrix(g) @ X       // group generator action
+    features_g = Linear(X_g)              // equivariant processing with shared weights
+  output = aggregate(features_g)          // aggregate along group dimension
+```
+
+## Implementable Structures
+- **e3nn / lie_learn integration**: Use existing libraries for SO(3)/SE(3) irreducible representations and spherical harmonics
+- **Blocked feature storage**: Organize feature dimensions by irreducible representations; each block independently normalized and processed
+- **Precomputed group operations**: Group multiplication tables, Clebsch-Gordan coefficients, etc., computed once and cached
+- **Symmetry augmentation**: Apply random group elements g in G to inputs during training (data augmentation) to encourage equivariance
+
+## GPU Feasibility
+- **Tensorization**: Processing irreducible components is batched GEMM; group convolution is sparse GEMM or batched matmul
+- **GEMM-mappable**: Each block of EquivariantLinear is an independent GEMM (N x d_i) @ (d_i x d_i_out), batchable
+- **Complexity**: Same order as standard networks (Schur constraints actually reduce parameters); group convolution incurs an extra |G| factor
+- **Memory & KV-Cache**: Group convolution requires storing |G| copies of features; significant memory pressure when |G| is large
+- **Low-precision stability**: Spherical harmonics Y_l^m computations involve factorials and square roots; fp32 recommended
+- **Parallelism & Communication**: Irreducible components are independent, perfectly parallel; different g in group convolution can be parallelized
+- **Sparse structure**: Permutation matrices in group convolution are extremely sparse (exactly one nonzero per row/column); SpMM is efficient
+- **Operator fusion**: Split -> batched matmul -> concat can be fused; group pooling scatter + reduce can be fused
+
+## Paper-Worthy Formulation
+"Leveraging Schur's lemma from group representation theory, we decompose the d-dimensional feature space into a direct sum of irreducible representations of symmetry group G, with each component independently processed by equivariance-constrained linear layers. This achieves a |G| / sum(m_i^2) improvement in parameter efficiency while guaranteeing network equivariance under group actions, reducing generalization error by O(1/sqrt(|G|)) on symmetric data."
+
+## Risks
+- Improper group selection (too large constrains cause underfitting, too small fails to capture symmetries)
+- Irreducible representation decomposition requires domain knowledge; implementation is complex for non-standard groups
+- The |G|-fold feature storage in group convolution becomes infeasible for large groups (e.g., S_10 has 3.6M elements)
+- The trade-off between approximate equivariance (soft equivariance) and strict equivariance is difficult to control