math-skill 2.1.0 → 3.0.0

package/design-patterns/representation/shared-private-decomposition.md

@@ -0,0 +1,86 @@
+# Shared-Private Decomposition（共享-私有分解）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+多任务/多领域学习中，需要将表示分解为"跨任务共性部分"和"任务特异部分"。
+典型场景：(1) 多任务 MoE——Shared 专家处理语言共性，Private 专家处理任务特异逻辑；
+(2) 多领域适配——Shared 表示捕获通用语义，Private 表示捕获领域术语；
+(3) 持续学习——Shared 保留稳定知识，Private 容纳新知识而不干扰旧知识。
+核心诉求：**显式分离共性与个性，避免负迁移和灾难性遗忘**。
+
+## 数学思想来源
+- 透镜：lenses/projection.md（子空间分解、直和分解）、lenses/probabilistic.md（信息分解）
+- 知识：knowledge-base/matrix-analysis/projection.md（直和分解 V = U ⊕ W、投影算子）、
+  knowledge-base/probability/kl-divergence.md（信息分解：shared/synergy/unique）
+
+## 需要的数学知识
+- **直和分解**：R^d = S ⊕ P，其中 S ∩ P = {0}，每个 x = x_S + x_P 唯一
+  投影矩阵 P_S + P_P = I，P_S · P_P = 0
+- **信息分解 (Williams & Beer PID)**：
+  I(X;Y₁,Y₂) = Shared + Unique₁ + Unique₂ + Synergy
+  Shared = min(I(X;Y₁), I(X;Y₂)) 的冗余信息部分
+- **低秩+稀疏分解 (RPCA)**：M = L + S，L 低秩（共性）+ S 稀疏（特异）
+  通过核范数 + L1 范数凸松弛求解
+- **CCA (典型相关分析)**：max corr(W₁^T X, W₂^T Y)，提取两组变量的共享变异
+
+## AI 模块形式
+```
+模块：SharedPrivateDecomposer
+输入：X ∈ R^{N×d}，任务标识 t ∈ {1,...,T}
+
+方法1 - 加法分解（最常用）：
+  z_shared = E_shared(X)          // 共享编码器：MLP or Transformer block
+  z_private = E_private[t](X)     // 私有编码器：每个任务独立参数
+  z = z_shared + z_private        // 加法融合
+  // 训练目标：L_task(z, y) + λ₁·OrthLoss(z_shared, z_private)
+  // 正交性确保 shared 和 private 学到不同的东西
+
+方法2 - 门控分解（动态权重）：
+  z_shared = E_shared(X)
+  z_private = E_private[t](X)
+  gate = sigmoid(Linear(z_shared ⊕ z_private))  // 动态融合门
+  z = gate ⊙ z_shared + (1 - gate) ⊙ z_private
+  // 门控允许逐维度选择 shared/private 的贡献比例
+
+方法3 - 对抗分解（信息论保证）：
+  z_shared = E_shared(X)
+  z_private = E_private[t](X)
+  // Shared 应无法区分任务（对抗梯度）：
+  task_pred = classifier(z_shared.flip_gradient())
+  L_adv = -CE(task_pred, t)       // Shared 不含任务信息
+  // Private 应能区分任务：
+  L_private = CE(classifier(z_private), t)
+  L = L_task + λ_adv·L_adv + λ_priv·L_private
+
+维度分配原则：
+  d_shared = d · T/(T+1)          // T 个任务时 shared 占大部分
+  d_private = d · 1/(T+1)         // 每个 private 占较小部分
+  // 或用 PCA 变异解释率动态分配
+```
+
+## 可实现结构
+- **双编码器 + 融合层**：shared_encoder (大) + T 个 private_encoder (小) + fusion
+- **参数效率**：private 用 LoRA（低秩适配）而非完整编码器，参数 O(d·r) per task
+- **动态路由集成**：shared 专家 + private 专家通过 MoE 路由选择
+- **渐进扩展**：新任务时只增加 private 编码器，frozen shared 参数
+
+## GPU 可行性
+- **张量化**：两个编码器前向为独立 GEMM 链，可并行执行
+- **GEMM 可映射**：shared/private 编码器各为标准 Transformer FFN（2×GEMM）
+- **复杂度**：shared O(N·d²) + T 个 private O(N·d²/T)，总 ≈ 2× 单编码器
+- **显存与 KV-Cache**：T 个 private 编码器参数全存储，T 大时需 LoRA 压缩
+- **低精度稳定**：加法/门控融合在 fp16 安全；对抗训练的梯度反转需 fp32
+- **并行与通信**：shared 和 private 编码器可分配到不同 GPU；多任务 batch 混合训练
+- **稀疏结构**：private 编码器可稀疏化（仅当前任务激活），T 个中仅激活 1 个
+- **算子融合**：加法融合 trivial；门控融合的 sigmoid→multiply→add 可融合
+
+## 论文表述方式
+"将多任务表示空间 R^d 分解为直和 S ⊕ P，Shared 子空间通过对抗训练确保任务无关性
+（H(T|Z_s)→log T），Private 子空间通过正交正则保证与 Shared 的信息互补，
+理论分析表明负迁移随正交性以 O(‖P_S·P_P‖_F) 衰减。"
+
+## 风险
+- 对抗训练的 min-max 优化不稳定，梯度反转的 scale 和 λ_adv 需精心调节
+- Shared 过度压缩导致共性信息不足，Private 负担过重
+- T 增大时 Private 参数总量线性增长，需要 LoRA 或 adapter 控制
+- 任务相似度低时 Shared 可能学到空洞的"公共部分"

package/design-patterns/representation/subspace-alignment.en.md

@@ -0,0 +1,90 @@
+# Subspace Alignment
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when two or more representation spaces need to be aligned to a common subspace. Typical scenarios:
+(1) Multi-modal alignment -- aligning text and image representations to a shared semantic subspace;
+(2) Cross-layer alignment -- aligning shallow features to the subspace of deep features for residual connections or distillation;
+(3) Expert output alignment -- different experts produce outputs with different dimensions/distributions that must be aligned before fusion;
+(4) Domain adaptation -- source and target domain feature distributions differ, requiring subspace alignment.
+Core requirement: **find the optimal linear/nonlinear mapping between two spaces such that corresponding semantics are aligned**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (Grassmann manifold, principal angles), lenses/variational.md (Procrustes problem)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (SVD, orthogonal Procrustes),
+  knowledge-base/differential-geometry/manifold.md (Grassmann distance, geodesics)
+
+## Required Mathematical Background
+- **Orthogonal Procrustes Problem**: min_{W in O(d)} ||AW - B||_F^2
+  Closed-form solution: W* = UV^T, where USV^T = SVD(A^T B)
+- **CCA (Canonical Correlation Analysis)**: max_{W_1, W_2} tr(W_1^T Sigma_{XY} W_2) s.t. W_1^T Sigma_{XX} W_1 = I
+  Solution via generalized eigenvalue problem or SVD(Sigma_{XX}^{-1/2} Sigma_{XY} Sigma_{YY}^{-1/2})
+- **Distance on the Grassmann Manifold**: Gr(d, r) = {r-dimensional subspaces of R^d}
+  Distance between two points (subspaces) determined by principal angles theta_i: d_G(U, V) = sqrt(sum theta_i^2)
+  cos(theta_i) = sigma_i(U^T V) (singular values of U^T V)
+- **Subspace Tracking**: Online update of subspace basis
+  Oja's rule: W_{t+1} = W_t + eta * (x_t x_t^T W_t - W_t diag(W_t^T x_t x_t^T W_t))
+
+## AI Module Form
+```
+Module: SubspaceAligner
+Input: Source representation A in R^{N x d_a}, target representation B in R^{N x d_b} (paired data)
+
+Method 1 - Linear Procrustes Alignment (most common):
+  // Find optimal linear duality W such that A @ W is close to B
+  M = A^T @ B                         // d_a x d_b, single GEMM
+  U, S, V^T = SVD(M)                  // singular value decomposition
+  W* = U @ V^T                        // d_a x d_b optimal orthogonal duality
+  // Differentiable version: parameterize W as nn.Parameter, optimize with SGD
+  L_align = ||A @ W - B||_F^2 / N    // alignment loss
+
+Method 2 - Deep CCA (nonlinear subspace alignment):
+  f_A = MLP_A(A)                      // R^{d_a} -> R^r (nonlinear projection)
+  f_B = MLP_B(B)                      // R^{d_b} -> R^r
+  Sigma_AA = f_A^T @ f_A / N + lambda * I  // r x r auto-covariance
+  Sigma_BB = f_B^T @ f_B / N + lambda * I
+  Sigma_AB = f_A^T @ f_B / N         // r x r cross-covariance
+  T = Sigma_AA^{-1/2} @ Sigma_AB @ Sigma_BB^{-1/2}  // whitened cross-correlation
+  L_cca = -tr(SVD(T).S[:k])          // maximize sum of top k canonical correlations
+  // Equivalent to minimizing Grassmann distance
+
+Method 3 - Subspace Angle Regularization (multi-expert output alignment):
+  // Outputs of multiple experts should be aligned to the same subspace
+  U_k = SVD(expert_k_output)[0][:, :r]  // r-dim principal components of each expert output
+  for i, j in expert_pairs:
+    cos_angles = SVD(U_i^T @ U_j).S     // cosines of principal angles
+    L_subspace = -mean(cos_angles)       // maximize principal angle cosines -> minimize angles
+  // Or use Grassmann distance:
+  L_grass = sum(angles^2)                // angles = arccos(cos_angles)
+
+Online Subspace Tracking (adaptation at inference):
+  // New samples arrive at test time; incrementally update alignment matrix
+  W_new = W_old + eta * (x_new @ (y_new^T - x_new^T @ W_old))  // Oja-like
+  // No need to recompute SVD; O(d^2) online update
+```
+
+## Implementable Structures
+- **Alignment Layer**: nn.Linear(d_a, d_b, bias=False) initialized with the Procrustes solution
+- **Whitening layer**: Use running mean/variance for online whitening, avoiding per-step Sigma^{-1/2} computation
+- **CCA alternative**: Use Barlow Twins-style redundancy reduction loss instead of CCA (avoids matrix inversion)
+  L_BT = ||C - I||_F^2 where C = corr(f_A, f_B)
+- **Chunked SVD**: Use randomized SVD approximation at large scale; sufficient precision and faster
+
+## GPU Feasibility
+- **Tensorization**: A^T @ B is standard GEMM (d x N) @ (N x d); SVD has cuSOLVER implementations
+- **GEMM-mappable**: Procrustes core requires 1 GEMM + 1 SVD; CCA requires 2 GEMMs + 1 SVD
+- **Complexity**: GEMM O(N * d^2); SVD O(d^3) (d typically < 1024, acceptable); online tracking O(d^2)
+- **Memory & KV-Cache**: Storing covariance matrix d x d (~4 MB for d = 1024); no KV-Cache overhead
+- **Low-precision stability**: SVD strongly recommended in fp32; whitening matrix inversion requires fp32 + epsilon regularization
+- **Parallelism & Communication**: Subspace angle computations for multiple expert pairs are independent and parallel; CCA GEMMs are highly parallel
+- **Sparse structure**: When source/target representations are sparse, covariance matrix Sigma is sparse, enabling sparse SVD
+- **Operator fusion**: Whitening (mean-sub -> cov -> inv_sqrt -> transform) can be partially fused
+
+## Paper-Worthy Formulation
+"Based on orthogonal Procrustes theory, we obtain the optimal isometric mapping W* = UV^T (where USV^T = SVD(A^T B)) between source and target representations. Extending this to Deep CCA for nonlinear subspace alignment, principal angle analysis on the Grassmann manifold shows that post-alignment subspace distance converges at rate O(r/sqrt(N)), while the Barlow Twins redundancy reduction objective ensures feature dimension decorrelation."
+
+## Risks
+- SVD during backpropagation produces undefined gradients when singular values coincide; epsilon regularization is needed
+- CCA whitening requires matrix inversion; numerically unstable when the covariance matrix is near-singular
+- Nonlinear CCA (Deep CCA) may overfit, particularly on small datasets
+- Online subspace tracking learning rate eta requires a decay schedule to prevent continuous drift

package/design-patterns/representation/subspace-alignment.md

@@ -0,0 +1,92 @@
+# Subspace Alignment（子空间对齐）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当两个或多个表示空间需要对齐到共同的子空间时使用。典型场景：
+(1) 多模态对齐——将文本和图像的表示对齐到共享语义子空间；
+(2) 跨层对齐——将浅层特征对齐到深层特征的子空间，便于残差连接或蒸馏；
+(3) 专家输出对齐——不同专家的输出维度/分布不同，需要对齐后融合；
+(4) 领域适配——源域和目标域的特征分布不同，需要对齐子空间。
+核心诉求：**找到两个空间之间的最优线性/非线性映射，使对应语义对齐**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（Grassmann 流形、主角度）、lenses/variational.md（Procrustes 问题）
+- 知识：knowledge-base/matrix-analysis/projection.md（SVD、正交 Procrustes）、
+  knowledge-base/differential-geometry/manifold.md（Grassmann 距离、测地线）
+
+## 需要的数学知识
+- **正交 Procrustes 问题**：min_{W∈O(d)} ‖AW - B‖_F²
+  闭合解：W* = UV^T，其中 USV^T = SVD(A^T B)
+- **CCA (典型相关分析)**：max_W₁,W₂ tr(W₁^T Σ_XY W₂) s.t. W₁^T Σ_XX W₁ = I
+  解为广义特征值问题或 SVD(Σ_XX^{-1/2} Σ_XY Σ_YY^{-1/2})
+- **Grassmann 流形上的距离**：Gr(d, r) = {r 维子空间 ⊂ R^d}
+  两点（子空间）的距离由主角度 θ_i 决定：d_G(U,V) = √(Σ θ_i²)
+  cos(θ_i) = σ_i(U^T V)（U^T V 的奇异值）
+- **子空间追踪 (Subspace Tracking)**：在线更新子空间基
+  Oja 规则：W_{t+1} = W_t + η(x_t x_t^T W_t - W_t diag(W_t^T x_t x_t^T W_t))
+
+## AI 模块形式
+```
+模块：SubspaceAligner
+输入：源表示 A ∈ R^{N×d_a}，目标表示 B ∈ R^{N×d_b}（配对数据）
+
+方法1 - 线性 Procrustes 对齐（最常用）：
+  // 找最优线性变换 W 使 A@W ≈ B
+  M = A^T @ B                       // d_a × d_b，一次 GEMM
+  U, S, V^T = SVD(M)                // 奇异值分解
+  W* = U @ V^T                      // d_a × d_b 最优正交变换
+  // 可微版本：将 W 参数化为 nn.Parameter，用 SGD 优化
+  L_align = ‖A @ W - B‖_F² / N     // 对齐损失
+
+方法2 - 深度 CCA（非线性子空间对齐）：
+  f_A = MLP_A(A)                    // R^{d_a} → R^r（非线性投影）
+  f_B = MLP_B(B)                    // R^{d_b} → R^r
+  Σ_AA = f_A^T @ f_A / N + λI       // r×r 自协方差
+  Σ_BB = f_B^T @ f_B / N + λI
+  Σ_AB = f_A^T @ f_B / N            // r×r 互协方差
+  T = Σ_AA^{-1/2} @ Σ_AB @ Σ_BB^{-1/2}  // 白化互相关
+  L_cca = -tr(SVD(T).S[:k])         // 最大化前 k 个典型相关的和
+  // 等效于最小化 Grassmann 距离
+
+方法3 - 子空间角度正则（多专家输出对齐）：
+  // 多个专家的输出应对齐到同一子空间
+  U_k = SVD(expert_k_output)[0][:, :r]  // 各专家输出的 r 维主成分
+  for i, j in expert_pairs:
+    cos_angles = SVD(U_i^T @ U_j).S    // 主角度的余弦
+    L_subspace = -mean(cos_angles)       // 最大化主角度余弦 → 最小化角度
+  // 或使用 Grassmann 距离：
+  L_grass = sum(angles²)                 // angles = arccos(cos_angles)
+
+在线子空间追踪（推理时适配）：
+  // 测试时新样本到来，增量更新对齐矩阵
+  W_new = W_old + η · (x_new @ (y_new^T - x_new^T @ W_old))  // Oja-like
+  // 无需重新 SVD，O(d²) 在线更新
+```
+
+## 可实现结构
+- **对齐层 (AlignmentLayer)**：nn.Linear(d_a, d_b, bias=False) 初始化为 Procrustes 解
+- **白化层**：用 running mean/variance 做在线白化，避免每步计算 Σ^{-1/2}
+- **CCA 替代方案**：用 Barlow Twins 式的冗余减少损失替代 CCA（避免矩阵逆）
+  L_BT = ‖C - I‖_F² 其中 C = corr(f_A, f_B)
+- **分块 SVD**：大规模时用 randomized SVD 近似，精度足够且更快
+
+## GPU 可行性
+- **张量化**：A^T@B 为标准 GEMM (d×N)@(N×d)；SVD 有 cuSOLVER 实现
+- **GEMM 可映射**：Procrustes 核心 1 次 GEMM + 1 次 SVD；CCA 2 次 GEMM + 1 次 SVD
+- **复杂度**：GEMM O(N·d²)；SVD O(d³)（d 通常 <1024，可接受）；在线追踪 O(d²)
+- **显存与 KV-Cache**：存储协方差矩阵 d×d（~4MB for d=1024），不增加 KV-Cache
+- **低精度稳定**：SVD 强烈建议 fp32；白化的矩阵逆需 fp32 + ε 正则化
+- **并行与通信**：多专家对的子空间角度计算独立并行；CCA 的 GEMM 高度并行
+- **稀疏结构**：当源/目标表示稀疏时，协方差矩阵 Σ 稀疏，可用稀疏 SVD
+- **算子融合**：白化 (mean-sub → cov → inv_sqrt → transform) 可部分融合
+
+## 论文表述方式
+"基于正交 Procrustes 理论求得源-目标表示间的最优等距映射 W*=UV^T（USV^T=SVD(A^TB)），
+将其扩展为深度 CCA 实现非线性子空间对齐，Grassmann 流形上的主角度分析表明
+对齐后子空间距离以 O(r/√N) 收敛，同时 Barlow Twins 冗余减少目标保证特征维度解耦。"
+
+## 风险
+- SVD 在反向传播时奇异值重合导致梯度未定义，需添加 ε 正则化
+- CCA 的白化步骤需要矩阵逆，协方差矩阵近奇异时数值不稳定
+- 非线性 CCA（深度 CCA）可能过拟合，特别是在小数据集上
+- 在线子空间追踪的学习率 η 需要衰减调度，否则持续漂移

package/design-patterns/routing/graph-routing.en.md

@@ -0,0 +1,79 @@
+# Graph Routing
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when there exists a known or learnable topological structure among modules/experts. Typical scenarios:
+(1) Hierarchical MoE -- experts organized in a tree structure, routing proceeds along tree edges;
+(2) Pipeline/serial routing -- inputs pass through multiple processing stages in DAG order;
+(3) Spatial/temporal correlation routing -- tokens at adjacent positions tend to be routed to similar experts (spatial continuity);
+(4) Knowledge-graph-guided expert selection -- experts organized according to a concept graph.
+Core requirement: **leverage structural priors to constrain routing decisions and reduce the search space**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (graph Laplacian, spectral graph theory), lenses/probabilistic.md (message passing, information flow)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (adjacency matrix, spectral decomposition),
+  knowledge-base/optimization/lagrangian-duality.md (variational on graphs, diffusion processes)
+
+## Required Mathematical Background
+- **Graph Laplacian**: L = D - A (combinatorial) or L_sym = D^{-1/2} L D^{-1/2} (normalized)
+  Eigendecomposition L = U Lambda U^T provides the graph frequency-domain basis; low-frequency components correspond to smooth signals
+- **Graph Diffusion / Random Walk**: P = D^{-1} A is the transition matrix, P^t describes the distribution after t steps
+  PageRank: pi = alpha * P^T * pi + (1 - alpha) * v, balancing graph structure and prior preferences
+- **Graph Neural Network Message Passing**:
+  h_i^{(l+1)} = sigma(sum_{j in N(i)} W^{(l)} h_j^{(l)} / sqrt(d_i * d_j))
+  Equivalent to a single sparse matrix multiplication L_sym * H * W
+- **Min-Cut Spectral Clustering**: min cut(A, B) s.t. vol(A) = vol(B) => approximate solution given by the Fiedler vector of L
+
+## AI Module Form
+```
+Module: GraphRouter
+Input: token representations X in R^{N x d}, expert graph G = (V, E) with |V| = K
+
+Method 1 - Graph Diffusion Routing (precomputed):
+  A in R^{K x K}   // expert adjacency matrix (predefined or learnable)
+  P = softmax(A / tau, dim=-1)   // normalized transition probabilities
+  P_t = matrix_power(P, t)       // t-step diffusion, t = 2 ~ 5
+  // routing score = initial score * diffusion matrix
+  score_init = X @ W_gate        // N x K, standard gate
+  score_final = score_init @ P_t // N x K, graph-diffusion smoothed
+  // A single GEMM (N x K) @ (K x K) incorporates graph structure
+
+Method 2 - GNN Routing (learnable graph structure):
+  H_0 = expert_embeddings        // K x d
+  H_1 = ReLU(L_norm @ H_0 @ W_1) // 1-layer GCN
+  H_2 = L_norm @ H_1 @ W_2      // 2-layer GCN
+  score = X @ H_2^T              // N x K routing scores
+  // Graph structure updated end-to-end via learnable parameterization of A
+
+Method 3 - Hierarchical Tree Routing (O(log K) complexity):
+  // Experts organized as a binary tree; each internal node is a binary classifier
+  for level in range(depth):      // depth = log2(K)
+    direction = sigmoid(X @ w_level + b_level)  // left/right subtree selection
+    path_prob *= direction         // accumulate path probability
+  // Total computation: O(N * d * log K) vs. O(N * d * K) for standard MoE
+```
+
+## Implementable Structures
+- **Sparse adjacency matrix**: Use torch.sparse to store A; sparse matmul replaces dense operations
+- **Precomputed diffusion kernel**: P_t fixed during early training, periodically recomputed (once per epoch)
+- **Graph structure learning**: A = softmax(MLP(E_i + E_j)) parameterizes edge weights for end-to-end learning
+- **Hierarchical tree implementation**: Represented as a complete binary tree array with level-wise vectorization
+
+## GPU Feasibility
+- **Tensorization**: GCN layers perform sparse matrix times dense matrix (SpMM), supported by both PyTorch and cuSPARSE
+- **GEMM-mappable**: score_init @ P_t in Method 1 is standard GEMM (N x K) @ (K x K)
+- **Complexity**: Method 1 O(N * K^2) diffusion + O(N * K * d) gate; Method 3 O(N * d * log K) significantly better than O(N * d * K)
+- **Memory & KV-Cache**: A matrix stored sparsely at K x K; hierarchical tree parameters d x log K are negligible
+- **Low-precision stability**: Probability matrix P and softmax in fp16 require attention to normalization precision
+- **Parallelism & Communication**: GNN message passing can be batched in parallel; hierarchical tree nodes at the same level can be evaluated independently in parallel
+- **Sparse structure**: Graph adjacency matrices are naturally sparse (degree << K); SpMM reduces from O(K^2) to O(K * avg_deg)
+- **Operator fusion**: L_norm @ H @ W in GCN can be fused into a single sparse GEMM
+
+## Paper-Worthy Formulation
+"Leveraging hierarchical tree/graph topology among experts, we compress routing decisions from a flat O(N * K) search to O(N * log K) tree traversal or O(N * K * avg_deg) graph diffusion. Fiedler spectral analysis reveals that the graph's algebraic connectivity lambda_2 directly governs the diversity-coherence trade-off in routing."
+
+## Risks
+- Incorrect graph structure priors can misguide routing toward suboptimal experts
+- Over-smoothing in graph diffusion (excessively large t) causes all tokens to route to the same expert
+- Learnable graph structure increases parameter count and overfitting risk
+- Binary classification errors in hierarchical trees accumulate layer by layer, degrading performance at greater depths

package/design-patterns/routing/graph-routing.md

@@ -0,0 +1,81 @@
+# Graph Routing（图路由）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当模块/专家之间存在已知的或可学习的拓扑结构时使用。典型场景：
+(1) 层次化 MoE——专家按树形结构组织，路由沿树的边进行；
+(2) Pipeline/串行路由——输入按 DAG 顺序经过多个处理阶段；
+(3) 空间/时间相关路由——相邻位置的 token 倾向路由到相近专家（空间连续性）；
+(4) 知识图谱引导的专家选择——专家按概念图谱组织。
+核心诉求：**利用结构化先验约束路由决策，减少搜索空间**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（图拉普拉斯、谱图论）、lenses/probabilistic.md（消息传递、信息流）
+- 知识：knowledge-base/matrix-analysis/projection.md（邻接矩阵、谱分解）、
+  knowledge-base/optimization/lagrangian-duality.md（图上优化、扩散过程）
+
+## 需要的数学知识
+- **图拉普拉斯**：L = D - A（组合）或 L_sym = D^{-1/2} L D^{-1/2}（归一化）
+  特征分解 L = U Λ U^T 给出图的频域基，低频分量对应平滑信号
+- **图扩散/Random Walk**：P = D^{-1}A 为转移矩阵，P^t 描述 t 步后的分布
+  PageRank: π = α·P^T·π + (1-α)·v，平衡图结构与先验偏好
+- **Graph Neural Network 消息传递**：
+  h_i^{(l+1)} = σ(Σ_{j∈N(i)} W^{(l)} h_j^{(l)} / √(d_i·d_j))
+  等价于一次稀疏矩阵乘法 L_sym · H · W
+- **Min-Cut 谱聚类**：min cut(A,B) s.t. vol(A)=vol(B) → 近似解为 L 的 Fiedler 向量
+
+## AI 模块形式
+```
+模块：GraphRouter
+输入：token 表示 X ∈ R^{N×d}，专家图 G = (V, E) 其中 |V|=K
+
+方法1 - 图扩散路由（预计算）：
+  A ∈ R^{K×K}   // 专家邻接矩阵（预定义或可学习）
+  P = softmax(A / τ, dim=-1)   // 归一化转移概率
+  P_t = matrix_power(P, t)     // t 步扩散，t=2~5
+  // 路由分数 = 初始分数 × 扩散矩阵
+  score_init = X @ W_gate      // N×K，标准 gate
+  score_final = score_init @ P_t  // N×K，图扩散平滑
+  // 一次 GEMM (N×K)@(K×K) 即可融入图结构
+
+方法2 - GNN 路由（可学习图结构）：
+  H_0 = expert_embeddings  // K×d
+  H_1 = ReLU(L_norm @ H_0 @ W_1)  // 1层 GCN
+  H_2 = L_norm @ H_1 @ W_2         // 2层 GCN
+  score = X @ H_2^T                // N×K 路由分数
+  // 图结构通过 A 的可学习参数化端到端更新
+
+方法3 - 层次树路由（O(log K) 复杂度）：
+  // 专家组织为二叉树，每个内部节点为二分类器
+  for level in range(depth):       // depth = log₂(K)
+    direction = sigmoid(X @ w_level + b_level)  // 左/右子树选择
+    path_prob *= direction          // 路径概率累积
+  // 总计算量：O(N·d·log K) vs 标准 MoE 的 O(N·d·K)
+```
+
+## 可实现结构
+- **稀疏邻接矩阵**：用 torch.sparse 存储 A，稀疏 matmul 替代 dense
+- **预计算扩散核**：P_t 在训练初期固定，周期性重计算（每 epoch 一次）
+- **图结构学习**：A = softmax(MLP(E_i ⊕ E_j)) 参数化边权重，端到端学习
+- **层次树实现**：用完全二叉树的数组表示，level-wise 向量化
+
+## GPU 可行性
+- **张量化**：GCN 层为稀疏矩阵×稠密矩阵 (SpMM)，PyTorch 和 cuSPARSE 均支持
+- **GEMM 可映射**：方法1 的 score_init@P_t 为标准 GEMM (N×K)@(K×K)
+- **复杂度**：方法1 O(N·K²) 扩散 + O(N·K·d) gate；方法3 O(N·d·log K) 显著优于 O(N·d·K)
+- **显存与 KV-Cache**：A 矩阵 K×K 稀疏存储；层次树参数 d×log K 极小
+- **低精度稳定**：概率矩阵 P 和 softmax 在 fp16 下需注意归一化精度
+- **并行与通信**：GNN 消息传递可批量并行；层次树同层节点独立可并行判断
+- **稀疏结构**：图邻接矩阵天然稀疏（度 << K），SpMM 加速比 O(K²) 到 O(K·avg_deg)
+- **算子融合**：GCN 的 L_norm@H@W 可融合为单次稀疏 GEMM
+
+## 论文表述方式
+"利用专家间的层次树/图拓扑结构，将路由决策从 O(N·K) 的平铺搜索压缩为 O(N·log K) 的
+树遍历或 O(N·K·avg_deg) 的图扩散，Fiedler 谱分析表明图的代数连通度 λ₂ 直接控制
+路由的多样性-一致性权衡。"
+
+## 风险
+- 图结构先验不正确时，路由被误导到次优专家
+- 图扩散过度平滑（t 过大）导致所有 token 路由到同一专家（oversmoothing）
+- 可学习图结构增加参数量和过拟合风险
+- 层次树的二分类误差逐层累积，depth 过深时性能下降

package/design-patterns/routing/moe-routing.en.md

@@ -0,0 +1,85 @@
+# MoE Routing (Mixture-of-Experts Routing)
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use in large-scale models where a small number of experts must be dynamically selected to process each token, achieving parameter scaling while keeping inference cost manageable.
+Typical scenarios: (1) Sparse MoE layers -- each token selects top-k experts (k << K);
+(2) Shared + Private expert mixing -- shared experts handle general features, private experts handle specialized features;
+(3) Multi-granularity MoE -- different layers employ different granularities of expert specialization.
+Core requirement: **sparse activation, load balancing, end-to-end trainability**.
+
+## Mathematical Inspiration
+- Lenses: lenses/variational.md (discrete variational relaxation, Gumbel-Softmax), lenses/probabilistic.md (information-theoretic routing)
+- Knowledge: knowledge-base/optimization/lagrangian-duality.md (combinatorial variational, integer programming relaxation),
+  knowledge-base/probability/entropy.md (entropy regularization, information bottleneck)
+
+## Required Mathematical Background
+- **Mixture Model EM**: p(y|x) = sum_k pi_k(x) * p(y|x, theta_k)
+  E-step estimates responsibilities gamma_{nk} = pi_k * p(y_n|x_n, theta_k) / sum_j pi_j * p(y_n|x_n, theta_j)
+  M-step updates expert parameters theta_k and mixture weights pi_k
+- **Top-k Sparse Gate**: G(x) = Softmax(TopK(x * W_g))
+  TopK is non-differentiable; during training, use noisy top-k or straight-through estimator
+- **Load-Balancing Auxiliary Loss**: L_aux = alpha * K * sum_k f_k * P_k
+  f_k = fraction of tokens assigned to expert k, P_k = average gating probability for expert k
+- **Expert Choice Routing**: Experts actively select tokens, rather than tokens selecting experts
+  score_{ki} = sim(e_k, x_i), each expert selects top-C tokens
+
+## AI Module Form
+```
+Module: MoERouter
+Input: X in R^{N x d}, K experts {E_k}_{k=1}^K, k experts activated per token
+
+Method 1 - Standard Top-K Gate (Switch/ST-MoE):
+  logits = X @ W_gate                  // N x K, standard GEMM
+  noise = randn(N, K) * softplus(X @ W_noise)  // learnable noise to encourage exploration
+  logits_noisy = logits + noise
+  topk_vals, topk_idx = topk(logits_noisy, k, dim=-1)  // select top-k
+  gate_weights = softmax(topk_vals, dim=-1)    // weights for k experts
+  // output = sum_{j in top-k} gate_weights_j * E_j(X)
+
+Method 2 - Shared + Private Dual-Path Routing:
+  // Shared expert always activated; Private experts selected via top-k
+  shared_out = E_shared(X)             // all tokens pass through shared expert
+  private_logits = X @ W_private_gate  // N x K_private
+  private_topk = topk(private_logits, k_p)
+  private_out = sum_j gate_j * E_private_j(X)
+  output = shared_out + private_out    // or concat + linear
+
+Method 3 - Expert Choice (Google 2022):
+  // Reverse perspective: each expert selects top-C tokens
+  affinity = E_embeddings @ X^T        // K x N, expert-token affinity
+  for k in range(K):
+    chosen_tokens = topk(affinity[k], C)  // each expert selects C = N/K tokens
+    expert_k.process(chosen_tokens)
+  // Natural load balancing: each expert processes exactly C tokens
+
+Auxiliary loss:
+  f = onehot(topk_idx).float().mean(dim=0)   // K-dim, per-expert load
+  P = softmax(logits, dim=-1).mean(dim=0)    // K-dim, per-expert average probability
+  L_aux = K * dot(f, P)                      // equals 1 when uniform, > 1 when non-uniform
+```
+
+## Implementable Structures
+- **Gate network**: Single-layer Linear(d, K) + optional noise network
+- **Expert parallelism**: Each expert on a separate GPU, all-to-all communication to exchange tokens
+- **Capacity factor**: cap = C_factor * N/K; tokens exceeding capacity go through residual (not dropped)
+- **Router Z-loss**: L_z = alpha * mean(logsumexp(logits)^2) stabilizes logit magnitudes
+
+## GPU Feasibility
+- **Tensorization**: Gate logits = X @ W_gate is standard GEMM (N x d) @ (d x K); expert computation is batched GEMM
+- **GEMM-mappable**: Gate requires 1 GEMM; each expert internally is a standard FFN (2 GEMMs + activation)
+- **Complexity**: Gate O(N * d * K); per expert O(N * d * d_ff / k); total FLOPs approximately standard FFN x k
+- **Memory & KV-Cache**: All K expert parameters stored but only k activated; activation memory approximately standard FFN x k
+- **Low-precision stability**: Gate softmax + top-k is safe in fp16; Router Z-loss requires fp32 logsumexp
+- **Parallelism & Communication**: Expert parallelism requires all-to-all communication (each GPU sends/receives N/K tokens); bandwidth-sensitive
+- **Sparse structure**: Top-k routing is naturally sparse (activates k/K fraction of parameters); sparsity = 1 - k/K
+- **Operator fusion**: Gate -> top-k -> softmax -> weighted-sum can be fused; intra-expert FFN can be fused
+
+## Paper-Worthy Formulation
+"We employ noisy top-k gating for sparse mixture-of-experts routing, activating a k/K fraction of parameters to achieve O(K) parameter capacity at O(k) inference cost. Combined with the load-balancing auxiliary loss L_aux = K * <f, P> and Router Z-loss, expert utilization exceeds 95%, yielding X% performance improvement over dense models under the same FLOPs budget."
+
+## Risks
+- Load imbalance: A few experts are over-selected (Matthew effect), while remaining experts receive insufficient training
+- Top-k operation is non-differentiable; straight-through estimation introduces gradient bias
+- All-to-all communication becomes a bottleneck in multi-GPU settings, especially when k > 1 as communication volume doubles
+- Noise injection promotes exploration but increases training variance, requiring careful annealing

package/design-patterns/routing/moe-routing.md

@@ -0,0 +1,87 @@
+# MoE Routing（混合专家路由）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+大规模模型中需要动态选择少量专家处理每个 token，以实现参数扩展而推理代价可控。
+典型场景：(1) Sparse MoE 层——每个 token 选择 top-k 专家（k<<K）；
+(2) Shared+Private 专家混合——共享专家处理通用特征，私有专家处理特异特征；
+(3) 多粒度 MoE——不同层使用不同粒度的专家分工。
+核心诉求：**稀疏激活、负载均衡、端到端可训练**。
+
+## 数学思想来源
+- 透镜：lenses/variational.md（离散优化松弛、Gumbel-Softmax）、lenses/probabilistic.md（信息论路由）
+- 知识：knowledge-base/optimization/lagrangian-duality.md（对偶分解、拉格朗日松弛）、
+  knowledge-base/probability/entropy.md（熵正则化、信息瓶颈）
+
+## 需要的数学知识
+- **混合模型 EM**：p(y|x) = Σ_k π_k(x) · p(y|x,θ_k)
+  E 步估计责任 γ_{nk} = π_k·p(y_n|x_n,θ_k) / Σ_j π_j·p(y_n|x_n,θ_j)
+  M 步更新专家参数 θ_k 和混合权重 π_k
+- **Top-k 稀疏 Gate**：G(x) = Softmax(TopK(x·W_g))
+  TopK 操作不可微，训练时用 noisy top-k 或 straight-through estimator
+- **负载均衡辅助损失**：L_aux = α · K · Σ_k f_k · P_k
+  f_k = 分配到专家 k 的 token 比例，P_k = 专家 k 的平均门控概率
+- **Expert Choice Routing**：专家主动选择 token，而非 token 选择专家
+  score_{ki} = sim(e_k, x_i)，每个专家选 top-C 个 token
+
+## AI 模块形式
+```
+模块：MoERouter
+输入：X ∈ R^{N×d}，K 个专家 {E_k}_{k=1}^K，每 token 激活 k 个专家
+
+方法1 - Standard Top-K Gate (Switch/ST-MoE)：
+  logits = X @ W_gate               // N×K，标准 GEMM
+  noise = randn(N, K) * softplus(X @ W_noise)  // 可学习噪声，促进探索
+  logits_noisy = logits + noise
+  topk_vals, topk_idx = topk(logits_noisy, k, dim=-1)  // 选 top-k
+  gate_weights = softmax(topk_vals, dim=-1)    // k 个专家的权重
+  // 输出 = Σ_{j∈top-k} gate_weights_j · E_j(X)
+
+方法2 - Shared + Private 双路路由：
+  // Shared 专家始终激活，Private 专家 top-k 选择
+  shared_out = E_shared(X)           // 所有 token 都过 shared 专家
+  private_logits = X @ W_private_gate  // N×K_private
+  private_topk = topk(private_logits, k_p)
+  private_out = Σ_j gate_j · E_private_j(X)
+  output = shared_out + private_out   // 或 concat + linear
+
+方法3 - Expert Choice (Google 2022)：
+  // 反转视角：每个专家选择 top-C 个 token
+  affinity = E_embeddings @ X^T      // K×N，专家与 token 的亲和力
+  for k in range(K):
+    chosen_tokens = topk(affinity[k], C)  // 每个专家选 C=N/K 个 token
+    expert_k.process(chosen_tokens)
+  // 天然负载均衡：每个专家处理恰好 C 个 token
+
+辅助损失：
+  f = onehot(topk_idx).float().mean(dim=0)   // K 维，各专家负载
+  P = softmax(logits, dim=-1).mean(dim=0)    // K 维，各专家平均概率
+  L_aux = K * dot(f, P)                      // 均匀时为 1，不均匀时 >1
+```
+
+## 可实现结构
+- **Gate 网络**：单层 Linear(d, K) + optional noise network
+- **专家并行**：each expert on separate GPU，all-to-all 通信交换 token
+- **容量因子**：cap = C_factor · N/K，超出容量的 token 走 residual（不被丢弃）
+- **Router Z-loss**：L_z = α·mean(logsumexp(logits)²) 稳定 logits 幅度
+
+## GPU 可行性
+- **张量化**：gate logits = X@W_gate 为标准 GEMM (N×d)@(d×K)；专家计算为 batched GEMM
+- **GEMM 可映射**：gate 1 次 GEMM；每个专家内部为标准 FFN（2 次 GEMM + activation）
+- **复杂度**：gate O(N·d·K)；每专家 O(N·d·d_ff/k)；总 FLOPs ≈ 标准 FFN × k
+- **显存与 KV-Cache**：K 个专家参数全存储但仅 k 个激活，激活显存 ≈ 标准 FFN × k
+- **低精度稳定**：gate 的 softmax + top-k 在 fp16 安全；Router Z-loss 需 fp32 logsumexp
+- **并行与通信**：专家并行需 all-to-all 通信（每个 GPU 收发 N/K 个 token），带宽敏感
+- **稀疏结构**：top-k 路由天然稀疏（激活 k/K 的参数），稀疏度 = 1 - k/K
+- **算子融合**：gate → top-k → softmax → weighted-sum 可融合；专家内 FFN 可 fuse
+
+## 论文表述方式
+"采用 noisy top-k 门控实现稀疏混合专家路由，激活 k/K 比例的参数以 O(k) 推理代价
+获得 O(K) 参数容量，配合负载均衡辅助损失 L_aux = K·⟨f,P⟩ 和 Router Z-loss 保证
+专家利用率 >95%，在同等 FLOPs 预算下相比 dense 模型提升 X% 性能。"
+
+## 风险
+- 负载不均衡：少数专家被过度选择（马太效应），其余专家得不到训练
+- Top-k 操作不可微，straight-through 估计引入梯度偏差
+- All-to-all 通信在多 GPU 下成为瓶颈，尤其 k>1 时通信量翻倍
+- 噪声注入虽促进探索但增加训练方差，需 careful annealing