math-skill 2.0.1 → 3.0.0

package/design-patterns/routing/graph-routing.en.md

@@ -0,0 +1,79 @@
+# Graph Routing
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when there exists a known or learnable topological structure among modules/experts. Typical scenarios:
+(1) Hierarchical MoE -- experts organized in a tree structure, routing proceeds along tree edges;
+(2) Pipeline/serial routing -- inputs pass through multiple processing stages in DAG order;
+(3) Spatial/temporal correlation routing -- tokens at adjacent positions tend to be routed to similar experts (spatial continuity);
+(4) Knowledge-graph-guided expert selection -- experts organized according to a concept graph.
+Core requirement: **leverage structural priors to constrain routing decisions and reduce the search space**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (graph Laplacian, spectral graph theory), lenses/probabilistic.md (message passing, information flow)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (adjacency matrix, spectral decomposition),
+  knowledge-base/optimization/lagrangian-duality.md (variational on graphs, diffusion processes)
+
+## Required Mathematical Background
+- **Graph Laplacian**: L = D - A (combinatorial) or L_sym = D^{-1/2} L D^{-1/2} (normalized)
+  Eigendecomposition L = U Lambda U^T provides the graph frequency-domain basis; low-frequency components correspond to smooth signals
+- **Graph Diffusion / Random Walk**: P = D^{-1} A is the transition matrix, P^t describes the distribution after t steps
+  PageRank: pi = alpha * P^T * pi + (1 - alpha) * v, balancing graph structure and prior preferences
+- **Graph Neural Network Message Passing**:
+  h_i^{(l+1)} = sigma(sum_{j in N(i)} W^{(l)} h_j^{(l)} / sqrt(d_i * d_j))
+  Equivalent to a single sparse matrix multiplication L_sym * H * W
+- **Min-Cut Spectral Clustering**: min cut(A, B) s.t. vol(A) = vol(B) => approximate solution given by the Fiedler vector of L
+
+## AI Module Form
+```
+Module: GraphRouter
+Input: token representations X in R^{N x d}, expert graph G = (V, E) with |V| = K
+
+Method 1 - Graph Diffusion Routing (precomputed):
+  A in R^{K x K}   // expert adjacency matrix (predefined or learnable)
+  P = softmax(A / tau, dim=-1)   // normalized transition probabilities
+  P_t = matrix_power(P, t)       // t-step diffusion, t = 2 ~ 5
+  // routing score = initial score * diffusion matrix
+  score_init = X @ W_gate        // N x K, standard gate
+  score_final = score_init @ P_t // N x K, graph-diffusion smoothed
+  // A single GEMM (N x K) @ (K x K) incorporates graph structure
+
+Method 2 - GNN Routing (learnable graph structure):
+  H_0 = expert_embeddings        // K x d
+  H_1 = ReLU(L_norm @ H_0 @ W_1) // 1-layer GCN
+  H_2 = L_norm @ H_1 @ W_2      // 2-layer GCN
+  score = X @ H_2^T              // N x K routing scores
+  // Graph structure updated end-to-end via learnable parameterization of A
+
+Method 3 - Hierarchical Tree Routing (O(log K) complexity):
+  // Experts organized as a binary tree; each internal node is a binary classifier
+  for level in range(depth):      // depth = log2(K)
+    direction = sigmoid(X @ w_level + b_level)  // left/right subtree selection
+    path_prob *= direction         // accumulate path probability
+  // Total computation: O(N * d * log K) vs. O(N * d * K) for standard MoE
+```
+
+## Implementable Structures
+- **Sparse adjacency matrix**: Use torch.sparse to store A; sparse matmul replaces dense operations
+- **Precomputed diffusion kernel**: P_t fixed during early training, periodically recomputed (once per epoch)
+- **Graph structure learning**: A = softmax(MLP(E_i + E_j)) parameterizes edge weights for end-to-end learning
+- **Hierarchical tree implementation**: Represented as a complete binary tree array with level-wise vectorization
+
+## GPU Feasibility
+- **Tensorization**: GCN layers perform sparse matrix times dense matrix (SpMM), supported by both PyTorch and cuSPARSE
+- **GEMM-mappable**: score_init @ P_t in Method 1 is standard GEMM (N x K) @ (K x K)
+- **Complexity**: Method 1 O(N * K^2) diffusion + O(N * K * d) gate; Method 3 O(N * d * log K) significantly better than O(N * d * K)
+- **Memory & KV-Cache**: A matrix stored sparsely at K x K; hierarchical tree parameters d x log K are negligible
+- **Low-precision stability**: Probability matrix P and softmax in fp16 require attention to normalization precision
+- **Parallelism & Communication**: GNN message passing can be batched in parallel; hierarchical tree nodes at the same level can be evaluated independently in parallel
+- **Sparse structure**: Graph adjacency matrices are naturally sparse (degree << K); SpMM reduces from O(K^2) to O(K * avg_deg)
+- **Operator fusion**: L_norm @ H @ W in GCN can be fused into a single sparse GEMM
+
+## Paper-Worthy Formulation
+"Leveraging hierarchical tree/graph topology among experts, we compress routing decisions from a flat O(N * K) search to O(N * log K) tree traversal or O(N * K * avg_deg) graph diffusion. Fiedler spectral analysis reveals that the graph's algebraic connectivity lambda_2 directly governs the diversity-coherence trade-off in routing."
+
+## Risks
+- Incorrect graph structure priors can misguide routing toward suboptimal experts
+- Over-smoothing in graph diffusion (excessively large t) causes all tokens to route to the same expert
+- Learnable graph structure increases parameter count and overfitting risk
+- Binary classification errors in hierarchical trees accumulate layer by layer, degrading performance at greater depths

package/design-patterns/routing/graph-routing.md

@@ -0,0 +1,81 @@
+# Graph Routing（图路由）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当模块/专家之间存在已知的或可学习的拓扑结构时使用。典型场景：
+(1) 层次化 MoE——专家按树形结构组织，路由沿树的边进行；
+(2) Pipeline/串行路由——输入按 DAG 顺序经过多个处理阶段；
+(3) 空间/时间相关路由——相邻位置的 token 倾向路由到相近专家（空间连续性）；
+(4) 知识图谱引导的专家选择——专家按概念图谱组织。
+核心诉求：**利用结构化先验约束路由决策，减少搜索空间**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（图拉普拉斯、谱图论）、lenses/probabilistic.md（消息传递、信息流）
+- 知识：knowledge-base/matrix-analysis/projection.md（邻接矩阵、谱分解）、
+  knowledge-base/optimization/lagrangian-duality.md（图上优化、扩散过程）
+
+## 需要的数学知识
+- **图拉普拉斯**：L = D - A（组合）或 L_sym = D^{-1/2} L D^{-1/2}（归一化）
+  特征分解 L = U Λ U^T 给出图的频域基，低频分量对应平滑信号
+- **图扩散/Random Walk**：P = D^{-1}A 为转移矩阵，P^t 描述 t 步后的分布
+  PageRank: π = α·P^T·π + (1-α)·v，平衡图结构与先验偏好
+- **Graph Neural Network 消息传递**：
+  h_i^{(l+1)} = σ(Σ_{j∈N(i)} W^{(l)} h_j^{(l)} / √(d_i·d_j))
+  等价于一次稀疏矩阵乘法 L_sym · H · W
+- **Min-Cut 谱聚类**：min cut(A,B) s.t. vol(A)=vol(B) → 近似解为 L 的 Fiedler 向量
+
+## AI 模块形式
+```
+模块：GraphRouter
+输入：token 表示 X ∈ R^{N×d}，专家图 G = (V, E) 其中 |V|=K
+
+方法1 - 图扩散路由（预计算）：
+  A ∈ R^{K×K}   // 专家邻接矩阵（预定义或可学习）
+  P = softmax(A / τ, dim=-1)   // 归一化转移概率
+  P_t = matrix_power(P, t)     // t 步扩散，t=2~5
+  // 路由分数 = 初始分数 × 扩散矩阵
+  score_init = X @ W_gate      // N×K，标准 gate
+  score_final = score_init @ P_t  // N×K，图扩散平滑
+  // 一次 GEMM (N×K)@(K×K) 即可融入图结构
+
+方法2 - GNN 路由（可学习图结构）：
+  H_0 = expert_embeddings  // K×d
+  H_1 = ReLU(L_norm @ H_0 @ W_1)  // 1层 GCN
+  H_2 = L_norm @ H_1 @ W_2         // 2层 GCN
+  score = X @ H_2^T                // N×K 路由分数
+  // 图结构通过 A 的可学习参数化端到端更新
+
+方法3 - 层次树路由（O(log K) 复杂度）：
+  // 专家组织为二叉树，每个内部节点为二分类器
+  for level in range(depth):       // depth = log₂(K)
+    direction = sigmoid(X @ w_level + b_level)  // 左/右子树选择
+    path_prob *= direction          // 路径概率累积
+  // 总计算量：O(N·d·log K) vs 标准 MoE 的 O(N·d·K)
+```
+
+## 可实现结构
+- **稀疏邻接矩阵**：用 torch.sparse 存储 A，稀疏 matmul 替代 dense
+- **预计算扩散核**：P_t 在训练初期固定，周期性重计算（每 epoch 一次）
+- **图结构学习**：A = softmax(MLP(E_i ⊕ E_j)) 参数化边权重，端到端学习
+- **层次树实现**：用完全二叉树的数组表示，level-wise 向量化
+
+## GPU 可行性
+- **张量化**：GCN 层为稀疏矩阵×稠密矩阵 (SpMM)，PyTorch 和 cuSPARSE 均支持
+- **GEMM 可映射**：方法1 的 score_init@P_t 为标准 GEMM (N×K)@(K×K)
+- **复杂度**：方法1 O(N·K²) 扩散 + O(N·K·d) gate；方法3 O(N·d·log K) 显著优于 O(N·d·K)
+- **显存与 KV-Cache**：A 矩阵 K×K 稀疏存储；层次树参数 d×log K 极小
+- **低精度稳定**：概率矩阵 P 和 softmax 在 fp16 下需注意归一化精度
+- **并行与通信**：GNN 消息传递可批量并行；层次树同层节点独立可并行判断
+- **稀疏结构**：图邻接矩阵天然稀疏（度 << K），SpMM 加速比 O(K²) 到 O(K·avg_deg)
+- **算子融合**：GCN 的 L_norm@H@W 可融合为单次稀疏 GEMM
+
+## 论文表述方式
+"利用专家间的层次树/图拓扑结构，将路由决策从 O(N·K) 的平铺搜索压缩为 O(N·log K) 的
+树遍历或 O(N·K·avg_deg) 的图扩散，Fiedler 谱分析表明图的代数连通度 λ₂ 直接控制
+路由的多样性-一致性权衡。"
+
+## 风险
+- 图结构先验不正确时，路由被误导到次优专家
+- 图扩散过度平滑（t 过大）导致所有 token 路由到同一专家（oversmoothing）
+- 可学习图结构增加参数量和过拟合风险
+- 层次树的二分类误差逐层累积，depth 过深时性能下降

package/design-patterns/routing/moe-routing.en.md

@@ -0,0 +1,85 @@
+# MoE Routing (Mixture-of-Experts Routing)
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use in large-scale models where a small number of experts must be dynamically selected to process each token, achieving parameter scaling while keeping inference cost manageable.
+Typical scenarios: (1) Sparse MoE layers -- each token selects top-k experts (k << K);
+(2) Shared + Private expert mixing -- shared experts handle general features, private experts handle specialized features;
+(3) Multi-granularity MoE -- different layers employ different granularities of expert specialization.
+Core requirement: **sparse activation, load balancing, end-to-end trainability**.
+
+## Mathematical Inspiration
+- Lenses: lenses/variational.md (discrete variational relaxation, Gumbel-Softmax), lenses/probabilistic.md (information-theoretic routing)
+- Knowledge: knowledge-base/optimization/lagrangian-duality.md (combinatorial variational, integer programming relaxation),
+  knowledge-base/probability/entropy.md (entropy regularization, information bottleneck)
+
+## Required Mathematical Background
+- **Mixture Model EM**: p(y|x) = sum_k pi_k(x) * p(y|x, theta_k)
+  E-step estimates responsibilities gamma_{nk} = pi_k * p(y_n|x_n, theta_k) / sum_j pi_j * p(y_n|x_n, theta_j)
+  M-step updates expert parameters theta_k and mixture weights pi_k
+- **Top-k Sparse Gate**: G(x) = Softmax(TopK(x * W_g))
+  TopK is non-differentiable; during training, use noisy top-k or straight-through estimator
+- **Load-Balancing Auxiliary Loss**: L_aux = alpha * K * sum_k f_k * P_k
+  f_k = fraction of tokens assigned to expert k, P_k = average gating probability for expert k
+- **Expert Choice Routing**: Experts actively select tokens, rather than tokens selecting experts
+  score_{ki} = sim(e_k, x_i), each expert selects top-C tokens
+
+## AI Module Form
+```
+Module: MoERouter
+Input: X in R^{N x d}, K experts {E_k}_{k=1}^K, k experts activated per token
+
+Method 1 - Standard Top-K Gate (Switch/ST-MoE):
+  logits = X @ W_gate                  // N x K, standard GEMM
+  noise = randn(N, K) * softplus(X @ W_noise)  // learnable noise to encourage exploration
+  logits_noisy = logits + noise
+  topk_vals, topk_idx = topk(logits_noisy, k, dim=-1)  // select top-k
+  gate_weights = softmax(topk_vals, dim=-1)    // weights for k experts
+  // output = sum_{j in top-k} gate_weights_j * E_j(X)
+
+Method 2 - Shared + Private Dual-Path Routing:
+  // Shared expert always activated; Private experts selected via top-k
+  shared_out = E_shared(X)             // all tokens pass through shared expert
+  private_logits = X @ W_private_gate  // N x K_private
+  private_topk = topk(private_logits, k_p)
+  private_out = sum_j gate_j * E_private_j(X)
+  output = shared_out + private_out    // or concat + linear
+
+Method 3 - Expert Choice (Google 2022):
+  // Reverse perspective: each expert selects top-C tokens
+  affinity = E_embeddings @ X^T        // K x N, expert-token affinity
+  for k in range(K):
+    chosen_tokens = topk(affinity[k], C)  // each expert selects C = N/K tokens
+    expert_k.process(chosen_tokens)
+  // Natural load balancing: each expert processes exactly C tokens
+
+Auxiliary loss:
+  f = onehot(topk_idx).float().mean(dim=0)   // K-dim, per-expert load
+  P = softmax(logits, dim=-1).mean(dim=0)    // K-dim, per-expert average probability
+  L_aux = K * dot(f, P)                      // equals 1 when uniform, > 1 when non-uniform
+```
+
+## Implementable Structures
+- **Gate network**: Single-layer Linear(d, K) + optional noise network
+- **Expert parallelism**: Each expert on a separate GPU, all-to-all communication to exchange tokens
+- **Capacity factor**: cap = C_factor * N/K; tokens exceeding capacity go through residual (not dropped)
+- **Router Z-loss**: L_z = alpha * mean(logsumexp(logits)^2) stabilizes logit magnitudes
+
+## GPU Feasibility
+- **Tensorization**: Gate logits = X @ W_gate is standard GEMM (N x d) @ (d x K); expert computation is batched GEMM
+- **GEMM-mappable**: Gate requires 1 GEMM; each expert internally is a standard FFN (2 GEMMs + activation)
+- **Complexity**: Gate O(N * d * K); per expert O(N * d * d_ff / k); total FLOPs approximately standard FFN x k
+- **Memory & KV-Cache**: All K expert parameters stored but only k activated; activation memory approximately standard FFN x k
+- **Low-precision stability**: Gate softmax + top-k is safe in fp16; Router Z-loss requires fp32 logsumexp
+- **Parallelism & Communication**: Expert parallelism requires all-to-all communication (each GPU sends/receives N/K tokens); bandwidth-sensitive
+- **Sparse structure**: Top-k routing is naturally sparse (activates k/K fraction of parameters); sparsity = 1 - k/K
+- **Operator fusion**: Gate -> top-k -> softmax -> weighted-sum can be fused; intra-expert FFN can be fused
+
+## Paper-Worthy Formulation
+"We employ noisy top-k gating for sparse mixture-of-experts routing, activating a k/K fraction of parameters to achieve O(K) parameter capacity at O(k) inference cost. Combined with the load-balancing auxiliary loss L_aux = K * <f, P> and Router Z-loss, expert utilization exceeds 95%, yielding X% performance improvement over dense models under the same FLOPs budget."
+
+## Risks
+- Load imbalance: A few experts are over-selected (Matthew effect), while remaining experts receive insufficient training
+- Top-k operation is non-differentiable; straight-through estimation introduces gradient bias
+- All-to-all communication becomes a bottleneck in multi-GPU settings, especially when k > 1 as communication volume doubles
+- Noise injection promotes exploration but increases training variance, requiring careful annealing

package/design-patterns/routing/moe-routing.md

@@ -0,0 +1,87 @@
+# MoE Routing（混合专家路由）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+大规模模型中需要动态选择少量专家处理每个 token，以实现参数扩展而推理代价可控。
+典型场景：(1) Sparse MoE 层——每个 token 选择 top-k 专家（k<<K）；
+(2) Shared+Private 专家混合——共享专家处理通用特征，私有专家处理特异特征；
+(3) 多粒度 MoE——不同层使用不同粒度的专家分工。
+核心诉求：**稀疏激活、负载均衡、端到端可训练**。
+
+## 数学思想来源
+- 透镜：lenses/variational.md（离散优化松弛、Gumbel-Softmax）、lenses/probabilistic.md（信息论路由）
+- 知识：knowledge-base/optimization/lagrangian-duality.md（对偶分解、拉格朗日松弛）、
+  knowledge-base/probability/entropy.md（熵正则化、信息瓶颈）
+
+## 需要的数学知识
+- **混合模型 EM**：p(y|x) = Σ_k π_k(x) · p(y|x,θ_k)
+  E 步估计责任 γ_{nk} = π_k·p(y_n|x_n,θ_k) / Σ_j π_j·p(y_n|x_n,θ_j)
+  M 步更新专家参数 θ_k 和混合权重 π_k
+- **Top-k 稀疏 Gate**：G(x) = Softmax(TopK(x·W_g))
+  TopK 操作不可微，训练时用 noisy top-k 或 straight-through estimator
+- **负载均衡辅助损失**：L_aux = α · K · Σ_k f_k · P_k
+  f_k = 分配到专家 k 的 token 比例，P_k = 专家 k 的平均门控概率
+- **Expert Choice Routing**：专家主动选择 token，而非 token 选择专家
+  score_{ki} = sim(e_k, x_i)，每个专家选 top-C 个 token
+
+## AI 模块形式
+```
+模块：MoERouter
+输入：X ∈ R^{N×d}，K 个专家 {E_k}_{k=1}^K，每 token 激活 k 个专家
+
+方法1 - Standard Top-K Gate (Switch/ST-MoE)：
+  logits = X @ W_gate               // N×K，标准 GEMM
+  noise = randn(N, K) * softplus(X @ W_noise)  // 可学习噪声，促进探索
+  logits_noisy = logits + noise
+  topk_vals, topk_idx = topk(logits_noisy, k, dim=-1)  // 选 top-k
+  gate_weights = softmax(topk_vals, dim=-1)    // k 个专家的权重
+  // 输出 = Σ_{j∈top-k} gate_weights_j · E_j(X)
+
+方法2 - Shared + Private 双路路由：
+  // Shared 专家始终激活，Private 专家 top-k 选择
+  shared_out = E_shared(X)           // 所有 token 都过 shared 专家
+  private_logits = X @ W_private_gate  // N×K_private
+  private_topk = topk(private_logits, k_p)
+  private_out = Σ_j gate_j · E_private_j(X)
+  output = shared_out + private_out   // 或 concat + linear
+
+方法3 - Expert Choice (Google 2022)：
+  // 反转视角：每个专家选择 top-C 个 token
+  affinity = E_embeddings @ X^T      // K×N，专家与 token 的亲和力
+  for k in range(K):
+    chosen_tokens = topk(affinity[k], C)  // 每个专家选 C=N/K 个 token
+    expert_k.process(chosen_tokens)
+  // 天然负载均衡：每个专家处理恰好 C 个 token
+
+辅助损失：
+  f = onehot(topk_idx).float().mean(dim=0)   // K 维，各专家负载
+  P = softmax(logits, dim=-1).mean(dim=0)    // K 维，各专家平均概率
+  L_aux = K * dot(f, P)                      // 均匀时为 1，不均匀时 >1
+```
+
+## 可实现结构
+- **Gate 网络**：单层 Linear(d, K) + optional noise network
+- **专家并行**：each expert on separate GPU，all-to-all 通信交换 token
+- **容量因子**：cap = C_factor · N/K，超出容量的 token 走 residual（不被丢弃）
+- **Router Z-loss**：L_z = α·mean(logsumexp(logits)²) 稳定 logits 幅度
+
+## GPU 可行性
+- **张量化**：gate logits = X@W_gate 为标准 GEMM (N×d)@(d×K)；专家计算为 batched GEMM
+- **GEMM 可映射**：gate 1 次 GEMM；每个专家内部为标准 FFN（2 次 GEMM + activation）
+- **复杂度**：gate O(N·d·K)；每专家 O(N·d·d_ff/k)；总 FLOPs ≈ 标准 FFN × k
+- **显存与 KV-Cache**：K 个专家参数全存储但仅 k 个激活，激活显存 ≈ 标准 FFN × k
+- **低精度稳定**：gate 的 softmax + top-k 在 fp16 安全；Router Z-loss 需 fp32 logsumexp
+- **并行与通信**：专家并行需 all-to-all 通信（每个 GPU 收发 N/K 个 token），带宽敏感
+- **稀疏结构**：top-k 路由天然稀疏（激活 k/K 的参数），稀疏度 = 1 - k/K
+- **算子融合**：gate → top-k → softmax → weighted-sum 可融合；专家内 FFN 可 fuse
+
+## 论文表述方式
+"采用 noisy top-k 门控实现稀疏混合专家路由，激活 k/K 比例的参数以 O(k) 推理代价
+获得 O(K) 参数容量，配合负载均衡辅助损失 L_aux = K·⟨f,P⟩ 和 Router Z-loss 保证
+专家利用率 >95%，在同等 FLOPs 预算下相比 dense 模型提升 X% 性能。"
+
+## 风险
+- 负载不均衡：少数专家被过度选择（马太效应），其余专家得不到训练
+- Top-k 操作不可微，straight-through 估计引入梯度偏差
+- All-to-all 通信在多 GPU 下成为瓶颈，尤其 k>1 时通信量翻倍
+- 噪声注入虽促进探索但增加训练方差，需 careful annealing

package/design-patterns/routing/optimal-transport-routing.en.md

@@ -0,0 +1,68 @@
+# Optimal Transport Routing
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when a set of input tokens/samples must be assigned to a set of experts/sub-modules while pursuing globally optimal matching cost.
+Typical scenarios: (1) Load-balanced MoE routing -- assigning N tokens to K experts with a cost matrix defined as negative similarity; (2) Cross-layer feature alignment -- transporting layer-l features to the optimal subset of layer-(l+1); (3) Multi-task sample assignment -- assigning samples in a batch to the most appropriate task head.
+Core requirement: **globally optimal assignment, rather than greedy per-point decisions**.
+
+## Mathematical Inspiration
+- Lenses: lenses/variational.md (convex variational, duality theory), lenses/geometric.md (Wasserstein distance)
+- Knowledge: knowledge-base/optimization/lagrangian-duality.md (duality theory, constrained optimization),
+  knowledge-base/probability/entropy.md (entropy regularization, marginal constraints)
+
+## Required Mathematical Background
+- **Discrete Optimal Transport**: min_{P in Pi(mu,nu)} <C, P> = sum_{ij} C_{ij} P_{ij}
+  where Pi(mu,nu) = {P >= 0 : P * 1 = mu, P^T * 1 = nu} is the set of couplings with marginal constraints
+- **Entropy-Regularized Sinkhorn**: min <C,P> - eps * H(P) => P* = diag(u) * exp(-C/eps) * diag(v)
+  Solved by alternating row/column scaling (Sinkhorn-Knopp), convergence rate O(1/eps^2)
+- **Wasserstein-1 Distance**: W_1(mu,nu) = min_{pi in Pi} E_pi[||x-y||] = sup_{||f||_L <= 1} E_mu[f] - E_nu[f]
+  Kantorovich-Rubinstein duality, used for continuous distribution matching
+- **Gromov-Wasserstein**: When source/target spaces have different dimensions, minimizes the structure-preserving transport cost
+
+## AI Module Form
+```
+Module: OptimalTransportRouter
+Input: token representations X in R^{N x d}, expert embeddings E in R^{K x d}, capacity constraint cap in R^K
+
+Cost matrix: C_{ik} = -sim(X_i, E_k)  or  ||X_i - E_k||^2  (N x K)
+
+Sinkhorn routing (entropy-regularized):
+  K_mat = exp(-C / eps)              // Gibbs kernel, eps = 0.05 ~ 0.1
+  for t = 1..T:                       // T = 5 ~ 20 iterations
+    u = a / (K_mat @ v)              // row scaling, a = 1/N
+    v = b / (K_mat^T @ u)            // column scaling, b = cap / sum(cap)
+  P = diag(u) @ K_mat @ diag(v)     // optimal transport plan, doubly stochastic matrix
+  assignment = argmax(P, dim=1)      // hard assignment (at inference)
+  // At training: weighted_features = P @ E  (soft assignment, differentiable)
+
+Capacity constraint (b vector):
+  b_k = total_tokens / K             // uniform allocation
+  b_k = alpha * uniform + (1 - alpha) * learned  // learned non-uniform allocation
+```
+
+## Implementable Structures
+- **Sinkhorn layer**: Custom autograd Function; forward pass performs Sinkhorn iterations, backward pass uses the implicit function theorem for gradients
+- **Fixed iteration count**: T = 10 fixed iterations => can be unrolled into a computation graph (unrolled variational)
+- **Log-domain stabilization**: Convert Sinkhorn to log domain to avoid exp overflow:
+  log_u = log_a - logsumexp(log_K + log_v, dim=1)
+- **Batch OT**: Solve independently per micro-batch, parallelize Sinkhorn iterations
+
+## GPU Feasibility
+- **Tensorization**: Sinkhorn core is matrix-vector multiplication K @ v of shape (N x K) * (K x 1), standard GEMV
+- **GEMM-mappable**: Computation of C via X @ E^T is GEMM (N x d) @ (d x K); Sinkhorn iterations are GEMV
+- **Complexity**: O(N * K * T) where T = 10 ~ 20; for N = 2048, K = 64 approximately 2.6M FLOPs, negligible
+- **Memory & KV-Cache**: Storing C (N x K) and P (N x K); for N = 2048, K = 64 approximately 1 MB
+- **Low-precision stability**: Sinkhorn in fp16 may cause exp(-C/eps) overflow; log-domain + fp32 recommended
+- **Parallelism & Communication**: Batch dimension is independent; Sinkhorn iterations have sequential dependencies, but each iteration's matvec is highly parallel
+- **Sparse structure**: As eps -> 0, P approaches sparsity (permutation matrix); top-k approximation can be used for acceleration
+- **Operator fusion**: The exp -> matvec -> division pipeline in a single Sinkhorn step can be fused into a CUDA kernel
+
+## Paper-Worthy Formulation
+"We formulate token-to-expert routing as an entropy-regularized optimal transport problem, obtaining an epsilon-approximate doubly stochastic assignment matrix via the Sinkhorn-Knopp algorithm within T = 10 iterations. The transport cost is guaranteed to converge to the global optimum within O(eps * log N), while marginal constraints b precisely control the load upper bound per expert."
+
+## Risks
+- Excessively small eps causes numerical instability in Sinkhorn (exp overflow); requires log-domain or increasing eps
+- Fixed unrolling of T iterations limits solution precision; too many iterations increase latency
+- The N x K cost matrix creates significant memory pressure when both N and K are large (N = 32K, K = 256 => 32 MB)
+- Training-inference discrepancy: performance gap between soft assignment and hard assignment

package/design-patterns/routing/optimal-transport-routing.md

@@ -0,0 +1,72 @@
+# Optimal Transport Routing（最优传输路由）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要将一组输入 token/样本 分配到一组专家/子模块，且追求全局最优的匹配代价时使用。
+典型场景：(1) 负载均衡的 MoE 路由——将 N 个 token 分配到 K 个专家，代价矩阵为负相似度；
+(2) 跨层特征对齐——将第 l 层特征传输到第 l+1 层的最优子集；
+(3) 多任务样本分配——将 batch 中的样本分配到最合适的任务头。
+核心诉求：**全局最优分配，而非贪心逐点决策**。
+
+## 数学思想来源
+- 透镜：lenses/variational.md（凸优化、对偶理论）、lenses/geometric.md（Wasserstein 距离）
+- 知识：knowledge-base/optimization/lagrangian-duality.md（对偶理论、约束优化）、
+  knowledge-base/probability/entropy.md（熵正则化、边际约束）
+
+## 需要的数学知识
+- **离散最优传输**：min_{P∈Π(μ,ν)} ⟨C, P⟩ = Σ_{ij} C_{ij} P_{ij}
+  其中 Π(μ,ν) = {P ≥ 0 : P·1 = μ, P^T·1 = ν} 为边际约束耦合集
+- **熵正则化 Sinkhorn**：min ⟨C,P⟩ - ε·H(P) → P* = diag(u)·exp(-C/ε)·diag(v)
+  通过交替缩放行/列（Sinkhorn-Knopp）求解，收敛速度 O(1/ε²)
+- **Wasserstein-1 距离**：W_1(μ,ν) = min_{π∈Π} E_π[‖x-y‖] = sup_{‖f‖_L≤1} E_μ[f] - E_ν[f]
+  Kantorovich-Rubinstein 对偶，用于连续分布匹配
+- **Gromov-Wasserstein**：当源/目标空间维度不同时，min 结构保持的传输代价
+
+## AI 模块形式
+```
+模块：OptimalTransportRouter
+输入：token 表示 X ∈ R^{N×d}，专家嵌入 E ∈ R^{K×d}，容量约束 cap ∈ R^K
+
+代价矩阵：C_{ik} = -sim(X_i, E_k)  或  ‖X_i - E_k‖²  (N×K)
+
+Sinkhorn 路由（熵正则化）：
+  K_mat = exp(-C / ε)              // Gibbs kernel, ε=0.05~0.1
+  for t = 1..T:                     // T=5~20 次迭代
+    u = a / (K_mat @ v)            // 行缩放，a = 1/N
+    v = b / (K_mat^T @ u)          // 列缩放，b = cap/sum(cap)
+  P = diag(u) @ K_mat @ diag(v)   // 最优传输计划，双随机矩阵
+  assignment = argmax(P, dim=1)    // 硬分配（推理时）
+  // 训练时：weighted_features = P @ E  (软分配，可微)
+
+容量约束（b 向量）：
+  b_k = total_tokens / K           // 均匀分配
+  b_k = α·uniform + (1-α)·learned  // 学习非均匀分配
+```
+
+## 可实现结构
+- **Sinkhorn 层**：自定义 autograd Function，前向做 Sinkhorn 迭代，反向用隐函数定理求梯度
+- **迭代次数固定**：T=10 次固定迭代 → 可展开为计算图（unrolled variational）
+- **log-domain 稳定化**：将 Sinkhorn 转换到 log 域避免 exp 溢出：
+  log_u = log_a - logsumexp(log_K + log_v, dim=1)
+- **Batch OT**：每个 micro-batch 独立求解，并行化 Sinkhorn 迭代
+
+## GPU 可行性
+- **张量化**：Sinkhorn 核心为矩阵向量乘法 K@v (N×K)·(K×1)，标准 GEMV
+- **GEMM 可映射**：C 的计算 X@E^T 为 GEMM (N×d)@(d×K)；Sinkhorn 迭代为 GEMV
+- **复杂度**：O(N·K·T) 其中 T=10~20，当 N=2048, K=64 时约 2.6M FLOPs，极小
+- **显存与 KV-Cache**：存储 C(N×K) 和 P(N×K)，N=2048,K=64 时约 1MB
+- **低精度稳定**：Sinkhorn 在 fp16 下 exp(-C/ε) 可能溢出，建议 log-domain + fp32
+- **并行与通信**：batch 维度独立；Sinkhorn 迭代为顺序依赖，但每次迭代的 matvec 高度并行
+- **稀疏结构**：ε→0 时 P 趋于稀疏（permutation matrix），可用 top-k 近似加速
+- **算子融合**：exp → matvec → division 的 Sinkhorn 单步可融合为 CUDA kernel
+
+## 论文表述方式
+"将 token-to-expert 路由建模为熵正则化最优传输问题，通过 Sinkhorn-Knopp 算法在 T=10 次
+迭代内求得 ε-近似最优的双随机分配矩阵，理论保证传输代价在 O(ε·log N) 内收敛到全局最优，
+同时通过边际约束 b 精确控制各专家的负载上限。"
+
+## 风险
+- ε 过小导致 Sinkhorn 数值不稳定（exp 溢出），需用 log-domain 或增大 ε
+- T 次迭代的固定展开限制了解精度，过多 T 增加延迟
+- N×K 代价矩阵在 N, K 都很大时内存压力显著（N=32K, K=256 → 32MB）
+- 训练-推理不一致：软分配 vs 硬分配的性能 gap

package/design-patterns/routing/spectral-clustering-routing.en.md

@@ -0,0 +1,88 @@
+# Spectral Clustering Routing
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when routing needs to be based on the intrinsic similarity structure of tokens/samples. Typical scenarios:
+(1) Unsupervised expert assignment -- when no routing labels are available, spectral clustering automatically discovers natural token clusters;
+(2) Adaptive expert initialization -- use spectral clustering results to initialize expert parameters at the beginning of training;
+(3) Input-aware dynamic clustering -- different batches have different token distributions, requiring adaptive routing;
+(4) Multi-granularity clustering -- different layers use spectral clustering at different granularities (coarse to fine).
+Core requirement: **discover the intrinsic cluster structure of data for routing or expert initialization**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (spectral graph theory, Laplacian eigenmaps), lenses/variational.md (relaxation and approximation)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (eigendecomposition, Rayleigh quotient),
+  knowledge-base/differential-geometry/manifold.md (manifold learning, graph cuts)
+
+## Required Mathematical Background
+- **Spectral Clustering (Ng-Jordan-Weiss)**:
+  1. Construct similarity graph W_{ij} = exp(-||x_i - x_j||^2 / (2 sigma^2))
+  2. Compute normalized Laplacian L_sym = I - D^{-1/2} W D^{-1/2}
+  3. Extract the k smallest eigenvectors U_k in R^{N x k}
+  4. Apply k-means to the rows of U_k to obtain k clusters
+- **Nystrom Approximation**: When N is too large for the full W matrix, sample m << N points
+  W approximately C * W_m^{-1} * C^T, reducing eigendecomposition to m x m
+- **Spectral Relaxation Continuation**: Discrete cluster assignment => continuous eigenvectors => differentiable routing
+  Use softmax(U_k * W_proj) instead of hard k-means assignment
+- **Power Iteration Acceleration**: Full eigendecomposition is unnecessary; only the top k eigenvectors are needed
+  Use Lanczos/Arnoldi iteration O(N^2 * k * iter) or randomized SVD O(N * k * log k)
+
+## AI Module Form
+```
+Module: SpectralClusterRouter
+Input: X in R^{N x d}, number of clusters K
+
+Method 1 - Online Spectral Clustering Routing (periodic updates during training):
+  // Update cluster centers every M steps; use nearest neighbor at inference
+  W = exp(-cdist(X_sample, X_sample) / (2 sigma^2))  // m x m sampled similarity
+  L = I - D^{-1/2} W D^{-1/2}                         // normalized Laplacian
+  U_k = eigsh(L, k=K, which='SM')                     // K smallest eigenvectors
+  centers = kmeans(U_k, K)                             // K cluster centers
+  // Routing: project new tokens into spectral space and assign
+  proj = X @ W_proj                                    // N -> K dimensional projection (learnable)
+  assignment = argmin(cdist(proj, centers))             // nearest center assignment
+
+Method 2 - Differentiable Spectral Routing (end-to-end):
+  // Use softmax relaxation instead of hard assignment
+  sim_matrix = X @ X^T                                 // N x N (or sampled m x m)
+  A = exp(sim_matrix / tau)                            // similarity graph (learnable tau)
+  D_inv_sqrt = diag(1 / sqrt(sum(A, dim=1) + eps))
+  L_norm = I - D_inv_sqrt @ A @ D_inv_sqrt             // normalized Laplacian
+  // Approximate top K eigenvectors (power iteration + orthogonalization)
+  U_k = power_iteration_approx(L_norm, K, steps=5)    // N x K
+  // Soft assignment
+  cluster_logits = U_k @ W_cluster                     // N x K (learnable projection)
+  route_probs = softmax(cluster_logits / tau_route)     // soft routing probabilities
+
+Method 3 - Anchor Spectral Clustering (large-scale):
+  anchors = kmeans_pp(X, m)                             // m anchor points, m << N
+  Z = exp(-cdist(X, anchors) / (2 sigma^2))             // N x m affinity matrix
+  L_anchor = I - D_z^{-1/2} Z^T Z D_z^{-1/2}            // m x m Laplacian
+  U_k = eigsh(L_anchor, K)                             // m x K eigenvectors
+  route = Z @ U_k @ W_proj                              // N x K routing scores
+```
+
+## Implementable Structures
+- **Periodic offline clustering**: Every N_step steps, collect token representations => offline spectral clustering => update routing table
+- **Nystrom sampling**: Randomly sample m = 1024 representative points, reducing the N x N problem to m x m
+- **Power iteration implementation**: 5-10 power iteration steps + Gram-Schmidt orthogonalization, GPU-friendly
+- **Progressive training**: Early stage uses k-means coarse routing => mid-stage spectral clustering refinement => late-stage fine-tuning of routing network
+
+## GPU Feasibility
+- **Tensorization**: Similarity matrix X @ X^T is GEMM; Laplacian construction is element-wise + diagonal matrix operations
+- **GEMM-mappable**: Z^T @ Z in Method 3 is GEMM (m x N) @ (N x m); Z @ U_k is GEMM (N x m) @ (m x K)
+- **Complexity**: Full spectral clustering O(N^2 * K) does not scale; Nystrom O(N * m * K + m^3); power iteration O(N^2 * K * T)
+- **Memory & KV-Cache**: N x N similarity matrix exceeds 64 MB when N > 4096; sampling-based dimensionality reduction is essential
+- **Low-precision stability**: Eigendecomposition recommended in fp32; exp(-dist/sigma^2) in fp16 requires distance clipping
+- **Parallelism & Communication**: Power iteration matvec is highly parallel; k-means assign + update steps can be batch-parallelized
+- **Sparse structure**: k-NN graph replaces fully connected graph; W sparsity > 95%, enabling SpMM acceleration
+- **Operator fusion**: Diagonal scaling in D^{-1/2} @ A @ D^{-1/2} can be fused; cdist + exp + normalize can be fused
+
+## Paper-Worthy Formulation
+"We achieve differentiable routing via continuous relaxation of spectral clustering: constructing the normalized Laplacian of the token similarity graph, reducing the O(N^2) eigendecomposition to O(Nm + K^3) through Nystrom approximation, and enabling GPU-friendly online spectral clustering via power iteration. Clustering quality, measured by Normalized Cut, guarantees an O(sqrt(log(N/K))) approximation ratio."
+
+## Risks
+- The memory and computation cost of the N x N similarity matrix does not scale for long sequences; sampling or k-NN sparsification is mandatory
+- Eigendecomposition is non-differentiable (gradients undefined when eigenvalues coincide); end-to-end training requires relaxation or stop-gradient
+- The number of clusters K must be specified a priori, and re-clustering is needed when K changes
+- The bandwidth parameter sigma is sensitive to clustering quality: too small causes isolated points, too large causes cluster merging