math-skill 2.1.0 → 3.0.0

package/design-patterns/routing/optimal-transport-routing.en.md

@@ -0,0 +1,68 @@
+# Optimal Transport Routing
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when a set of input tokens/samples must be assigned to a set of experts/sub-modules while pursuing globally optimal matching cost.
+Typical scenarios: (1) Load-balanced MoE routing -- assigning N tokens to K experts with a cost matrix defined as negative similarity; (2) Cross-layer feature alignment -- transporting layer-l features to the optimal subset of layer-(l+1); (3) Multi-task sample assignment -- assigning samples in a batch to the most appropriate task head.
+Core requirement: **globally optimal assignment, rather than greedy per-point decisions**.
+
+## Mathematical Inspiration
+- Lenses: lenses/variational.md (convex variational, duality theory), lenses/geometric.md (Wasserstein distance)
+- Knowledge: knowledge-base/optimization/lagrangian-duality.md (duality theory, constrained optimization),
+  knowledge-base/probability/entropy.md (entropy regularization, marginal constraints)
+
+## Required Mathematical Background
+- **Discrete Optimal Transport**: min_{P in Pi(mu,nu)} <C, P> = sum_{ij} C_{ij} P_{ij}
+  where Pi(mu,nu) = {P >= 0 : P * 1 = mu, P^T * 1 = nu} is the set of couplings with marginal constraints
+- **Entropy-Regularized Sinkhorn**: min <C,P> - eps * H(P) => P* = diag(u) * exp(-C/eps) * diag(v)
+  Solved by alternating row/column scaling (Sinkhorn-Knopp), convergence rate O(1/eps^2)
+- **Wasserstein-1 Distance**: W_1(mu,nu) = min_{pi in Pi} E_pi[||x-y||] = sup_{||f||_L <= 1} E_mu[f] - E_nu[f]
+  Kantorovich-Rubinstein duality, used for continuous distribution matching
+- **Gromov-Wasserstein**: When source/target spaces have different dimensions, minimizes the structure-preserving transport cost
+
+## AI Module Form
+```
+Module: OptimalTransportRouter
+Input: token representations X in R^{N x d}, expert embeddings E in R^{K x d}, capacity constraint cap in R^K
+
+Cost matrix: C_{ik} = -sim(X_i, E_k)  or  ||X_i - E_k||^2  (N x K)
+
+Sinkhorn routing (entropy-regularized):
+  K_mat = exp(-C / eps)              // Gibbs kernel, eps = 0.05 ~ 0.1
+  for t = 1..T:                       // T = 5 ~ 20 iterations
+    u = a / (K_mat @ v)              // row scaling, a = 1/N
+    v = b / (K_mat^T @ u)            // column scaling, b = cap / sum(cap)
+  P = diag(u) @ K_mat @ diag(v)     // optimal transport plan, doubly stochastic matrix
+  assignment = argmax(P, dim=1)      // hard assignment (at inference)
+  // At training: weighted_features = P @ E  (soft assignment, differentiable)
+
+Capacity constraint (b vector):
+  b_k = total_tokens / K             // uniform allocation
+  b_k = alpha * uniform + (1 - alpha) * learned  // learned non-uniform allocation
+```
+
+## Implementable Structures
+- **Sinkhorn layer**: Custom autograd Function; forward pass performs Sinkhorn iterations, backward pass uses the implicit function theorem for gradients
+- **Fixed iteration count**: T = 10 fixed iterations => can be unrolled into a computation graph (unrolled variational)
+- **Log-domain stabilization**: Convert Sinkhorn to log domain to avoid exp overflow:
+  log_u = log_a - logsumexp(log_K + log_v, dim=1)
+- **Batch OT**: Solve independently per micro-batch, parallelize Sinkhorn iterations
+
+## GPU Feasibility
+- **Tensorization**: Sinkhorn core is matrix-vector multiplication K @ v of shape (N x K) * (K x 1), standard GEMV
+- **GEMM-mappable**: Computation of C via X @ E^T is GEMM (N x d) @ (d x K); Sinkhorn iterations are GEMV
+- **Complexity**: O(N * K * T) where T = 10 ~ 20; for N = 2048, K = 64 approximately 2.6M FLOPs, negligible
+- **Memory & KV-Cache**: Storing C (N x K) and P (N x K); for N = 2048, K = 64 approximately 1 MB
+- **Low-precision stability**: Sinkhorn in fp16 may cause exp(-C/eps) overflow; log-domain + fp32 recommended
+- **Parallelism & Communication**: Batch dimension is independent; Sinkhorn iterations have sequential dependencies, but each iteration's matvec is highly parallel
+- **Sparse structure**: As eps -> 0, P approaches sparsity (permutation matrix); top-k approximation can be used for acceleration
+- **Operator fusion**: The exp -> matvec -> division pipeline in a single Sinkhorn step can be fused into a CUDA kernel
+
+## Paper-Worthy Formulation
+"We formulate token-to-expert routing as an entropy-regularized optimal transport problem, obtaining an epsilon-approximate doubly stochastic assignment matrix via the Sinkhorn-Knopp algorithm within T = 10 iterations. The transport cost is guaranteed to converge to the global optimum within O(eps * log N), while marginal constraints b precisely control the load upper bound per expert."
+
+## Risks
+- Excessively small eps causes numerical instability in Sinkhorn (exp overflow); requires log-domain or increasing eps
+- Fixed unrolling of T iterations limits solution precision; too many iterations increase latency
+- The N x K cost matrix creates significant memory pressure when both N and K are large (N = 32K, K = 256 => 32 MB)
+- Training-inference discrepancy: performance gap between soft assignment and hard assignment

package/design-patterns/routing/optimal-transport-routing.md

@@ -0,0 +1,72 @@
+# Optimal Transport Routing（最优传输路由）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要将一组输入 token/样本 分配到一组专家/子模块，且追求全局最优的匹配代价时使用。
+典型场景：(1) 负载均衡的 MoE 路由——将 N 个 token 分配到 K 个专家，代价矩阵为负相似度；
+(2) 跨层特征对齐——将第 l 层特征传输到第 l+1 层的最优子集；
+(3) 多任务样本分配——将 batch 中的样本分配到最合适的任务头。
+核心诉求：**全局最优分配，而非贪心逐点决策**。
+
+## 数学思想来源
+- 透镜：lenses/variational.md（凸优化、对偶理论）、lenses/geometric.md（Wasserstein 距离）
+- 知识：knowledge-base/optimization/lagrangian-duality.md（对偶理论、约束优化）、
+  knowledge-base/probability/entropy.md（熵正则化、边际约束）
+
+## 需要的数学知识
+- **离散最优传输**：min_{P∈Π(μ,ν)} ⟨C, P⟩ = Σ_{ij} C_{ij} P_{ij}
+  其中 Π(μ,ν) = {P ≥ 0 : P·1 = μ, P^T·1 = ν} 为边际约束耦合集
+- **熵正则化 Sinkhorn**：min ⟨C,P⟩ - ε·H(P) → P* = diag(u)·exp(-C/ε)·diag(v)
+  通过交替缩放行/列（Sinkhorn-Knopp）求解，收敛速度 O(1/ε²)
+- **Wasserstein-1 距离**：W_1(μ,ν) = min_{π∈Π} E_π[‖x-y‖] = sup_{‖f‖_L≤1} E_μ[f] - E_ν[f]
+  Kantorovich-Rubinstein 对偶，用于连续分布匹配
+- **Gromov-Wasserstein**：当源/目标空间维度不同时，min 结构保持的传输代价
+
+## AI 模块形式
+```
+模块：OptimalTransportRouter
+输入：token 表示 X ∈ R^{N×d}，专家嵌入 E ∈ R^{K×d}，容量约束 cap ∈ R^K
+
+代价矩阵：C_{ik} = -sim(X_i, E_k)  或  ‖X_i - E_k‖²  (N×K)
+
+Sinkhorn 路由（熵正则化）：
+  K_mat = exp(-C / ε)              // Gibbs kernel, ε=0.05~0.1
+  for t = 1..T:                     // T=5~20 次迭代
+    u = a / (K_mat @ v)            // 行缩放，a = 1/N
+    v = b / (K_mat^T @ u)          // 列缩放，b = cap/sum(cap)
+  P = diag(u) @ K_mat @ diag(v)   // 最优传输计划，双随机矩阵
+  assignment = argmax(P, dim=1)    // 硬分配（推理时）
+  // 训练时：weighted_features = P @ E  (软分配，可微)
+
+容量约束（b 向量）：
+  b_k = total_tokens / K           // 均匀分配
+  b_k = α·uniform + (1-α)·learned  // 学习非均匀分配
+```
+
+## 可实现结构
+- **Sinkhorn 层**：自定义 autograd Function，前向做 Sinkhorn 迭代，反向用隐函数定理求梯度
+- **迭代次数固定**：T=10 次固定迭代 → 可展开为计算图（unrolled variational）
+- **log-domain 稳定化**：将 Sinkhorn 转换到 log 域避免 exp 溢出：
+  log_u = log_a - logsumexp(log_K + log_v, dim=1)
+- **Batch OT**：每个 micro-batch 独立求解，并行化 Sinkhorn 迭代
+
+## GPU 可行性
+- **张量化**：Sinkhorn 核心为矩阵向量乘法 K@v (N×K)·(K×1)，标准 GEMV
+- **GEMM 可映射**：C 的计算 X@E^T 为 GEMM (N×d)@(d×K)；Sinkhorn 迭代为 GEMV
+- **复杂度**：O(N·K·T) 其中 T=10~20，当 N=2048, K=64 时约 2.6M FLOPs，极小
+- **显存与 KV-Cache**：存储 C(N×K) 和 P(N×K)，N=2048,K=64 时约 1MB
+- **低精度稳定**：Sinkhorn 在 fp16 下 exp(-C/ε) 可能溢出，建议 log-domain + fp32
+- **并行与通信**：batch 维度独立；Sinkhorn 迭代为顺序依赖，但每次迭代的 matvec 高度并行
+- **稀疏结构**：ε→0 时 P 趋于稀疏（permutation matrix），可用 top-k 近似加速
+- **算子融合**：exp → matvec → division 的 Sinkhorn 单步可融合为 CUDA kernel
+
+## 论文表述方式
+"将 token-to-expert 路由建模为熵正则化最优传输问题，通过 Sinkhorn-Knopp 算法在 T=10 次
+迭代内求得 ε-近似最优的双随机分配矩阵，理论保证传输代价在 O(ε·log N) 内收敛到全局最优，
+同时通过边际约束 b 精确控制各专家的负载上限。"
+
+## 风险
+- ε 过小导致 Sinkhorn 数值不稳定（exp 溢出），需用 log-domain 或增大 ε
+- T 次迭代的固定展开限制了解精度，过多 T 增加延迟
+- N×K 代价矩阵在 N, K 都很大时内存压力显著（N=32K, K=256 → 32MB）
+- 训练-推理不一致：软分配 vs 硬分配的性能 gap

package/design-patterns/routing/spectral-clustering-routing.en.md

@@ -0,0 +1,88 @@
+# Spectral Clustering Routing
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when routing needs to be based on the intrinsic similarity structure of tokens/samples. Typical scenarios:
+(1) Unsupervised expert assignment -- when no routing labels are available, spectral clustering automatically discovers natural token clusters;
+(2) Adaptive expert initialization -- use spectral clustering results to initialize expert parameters at the beginning of training;
+(3) Input-aware dynamic clustering -- different batches have different token distributions, requiring adaptive routing;
+(4) Multi-granularity clustering -- different layers use spectral clustering at different granularities (coarse to fine).
+Core requirement: **discover the intrinsic cluster structure of data for routing or expert initialization**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (spectral graph theory, Laplacian eigenmaps), lenses/variational.md (relaxation and approximation)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (eigendecomposition, Rayleigh quotient),
+  knowledge-base/differential-geometry/manifold.md (manifold learning, graph cuts)
+
+## Required Mathematical Background
+- **Spectral Clustering (Ng-Jordan-Weiss)**:
+  1. Construct similarity graph W_{ij} = exp(-||x_i - x_j||^2 / (2 sigma^2))
+  2. Compute normalized Laplacian L_sym = I - D^{-1/2} W D^{-1/2}
+  3. Extract the k smallest eigenvectors U_k in R^{N x k}
+  4. Apply k-means to the rows of U_k to obtain k clusters
+- **Nystrom Approximation**: When N is too large for the full W matrix, sample m << N points
+  W approximately C * W_m^{-1} * C^T, reducing eigendecomposition to m x m
+- **Spectral Relaxation Continuation**: Discrete cluster assignment => continuous eigenvectors => differentiable routing
+  Use softmax(U_k * W_proj) instead of hard k-means assignment
+- **Power Iteration Acceleration**: Full eigendecomposition is unnecessary; only the top k eigenvectors are needed
+  Use Lanczos/Arnoldi iteration O(N^2 * k * iter) or randomized SVD O(N * k * log k)
+
+## AI Module Form
+```
+Module: SpectralClusterRouter
+Input: X in R^{N x d}, number of clusters K
+
+Method 1 - Online Spectral Clustering Routing (periodic updates during training):
+  // Update cluster centers every M steps; use nearest neighbor at inference
+  W = exp(-cdist(X_sample, X_sample) / (2 sigma^2))  // m x m sampled similarity
+  L = I - D^{-1/2} W D^{-1/2}                         // normalized Laplacian
+  U_k = eigsh(L, k=K, which='SM')                     // K smallest eigenvectors
+  centers = kmeans(U_k, K)                             // K cluster centers
+  // Routing: project new tokens into spectral space and assign
+  proj = X @ W_proj                                    // N -> K dimensional projection (learnable)
+  assignment = argmin(cdist(proj, centers))             // nearest center assignment
+
+Method 2 - Differentiable Spectral Routing (end-to-end):
+  // Use softmax relaxation instead of hard assignment
+  sim_matrix = X @ X^T                                 // N x N (or sampled m x m)
+  A = exp(sim_matrix / tau)                            // similarity graph (learnable tau)
+  D_inv_sqrt = diag(1 / sqrt(sum(A, dim=1) + eps))
+  L_norm = I - D_inv_sqrt @ A @ D_inv_sqrt             // normalized Laplacian
+  // Approximate top K eigenvectors (power iteration + orthogonalization)
+  U_k = power_iteration_approx(L_norm, K, steps=5)    // N x K
+  // Soft assignment
+  cluster_logits = U_k @ W_cluster                     // N x K (learnable projection)
+  route_probs = softmax(cluster_logits / tau_route)     // soft routing probabilities
+
+Method 3 - Anchor Spectral Clustering (large-scale):
+  anchors = kmeans_pp(X, m)                             // m anchor points, m << N
+  Z = exp(-cdist(X, anchors) / (2 sigma^2))             // N x m affinity matrix
+  L_anchor = I - D_z^{-1/2} Z^T Z D_z^{-1/2}            // m x m Laplacian
+  U_k = eigsh(L_anchor, K)                             // m x K eigenvectors
+  route = Z @ U_k @ W_proj                              // N x K routing scores
+```
+
+## Implementable Structures
+- **Periodic offline clustering**: Every N_step steps, collect token representations => offline spectral clustering => update routing table
+- **Nystrom sampling**: Randomly sample m = 1024 representative points, reducing the N x N problem to m x m
+- **Power iteration implementation**: 5-10 power iteration steps + Gram-Schmidt orthogonalization, GPU-friendly
+- **Progressive training**: Early stage uses k-means coarse routing => mid-stage spectral clustering refinement => late-stage fine-tuning of routing network
+
+## GPU Feasibility
+- **Tensorization**: Similarity matrix X @ X^T is GEMM; Laplacian construction is element-wise + diagonal matrix operations
+- **GEMM-mappable**: Z^T @ Z in Method 3 is GEMM (m x N) @ (N x m); Z @ U_k is GEMM (N x m) @ (m x K)
+- **Complexity**: Full spectral clustering O(N^2 * K) does not scale; Nystrom O(N * m * K + m^3); power iteration O(N^2 * K * T)
+- **Memory & KV-Cache**: N x N similarity matrix exceeds 64 MB when N > 4096; sampling-based dimensionality reduction is essential
+- **Low-precision stability**: Eigendecomposition recommended in fp32; exp(-dist/sigma^2) in fp16 requires distance clipping
+- **Parallelism & Communication**: Power iteration matvec is highly parallel; k-means assign + update steps can be batch-parallelized
+- **Sparse structure**: k-NN graph replaces fully connected graph; W sparsity > 95%, enabling SpMM acceleration
+- **Operator fusion**: Diagonal scaling in D^{-1/2} @ A @ D^{-1/2} can be fused; cdist + exp + normalize can be fused
+
+## Paper-Worthy Formulation
+"We achieve differentiable routing via continuous relaxation of spectral clustering: constructing the normalized Laplacian of the token similarity graph, reducing the O(N^2) eigendecomposition to O(Nm + K^3) through Nystrom approximation, and enabling GPU-friendly online spectral clustering via power iteration. Clustering quality, measured by Normalized Cut, guarantees an O(sqrt(log(N/K))) approximation ratio."
+
+## Risks
+- The memory and computation cost of the N x N similarity matrix does not scale for long sequences; sampling or k-NN sparsification is mandatory
+- Eigendecomposition is non-differentiable (gradients undefined when eigenvalues coincide); end-to-end training requires relaxation or stop-gradient
+- The number of clusters K must be specified a priori, and re-clustering is needed when K changes
+- The bandwidth parameter sigma is sensitive to clustering quality: too small causes isolated points, too large causes cluster merging

package/design-patterns/routing/spectral-clustering-routing.md

@@ -0,0 +1,90 @@
+# Spectral Clustering Routing（谱聚类路由）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要基于 token/样本的内在相似性结构进行分组路由时使用。典型场景：
+(1) 无监督专家分配——没有路由标签时，用谱聚类自动发现 token 的自然分簇；
+(2) 自适应专家初始化——训练初期用谱聚类结果初始化专家参数；
+(3) 输入感知的动态聚类——不同 batch 的 token 分布不同，路由应自适应；
+(4) 多粒度聚类——不同层使用不同粒度的谱聚类（粗→细）。
+核心诉求：**发现数据内在的簇结构，用于路由或专家初始化**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（谱图论、拉普拉斯特征映射）、lenses/variational.md（松弛与近似）
+- 知识：knowledge-base/matrix-analysis/projection.md（特征值分解、Rayleigh 商）、
+  knowledge-base/differential-geometry/manifold.md（流形学习、图割）
+
+## 需要的数学知识
+- **谱聚类 (Ng-Jordan-Weiss)**：
+  1. 构造相似度图 W_{ij} = exp(-‖x_i - x_j‖² / 2σ²)
+  2. 计算归一化拉普拉斯 L_sym = I - D^{-1/2} W D^{-1/2}
+  3. 取前 k 个最小特征向量 U_k ∈ R^{N×k}
+  4. 对 U_k 的行做 k-means 得到 k 个簇
+- **Nyström 近似**：当 N 太大无法计算完整 W 时，采样 m<<N 个点
+  W ≈ C · W_m^{-1} · C^T，将特征分解降维到 m×m
+- **谱松弛连续化**：离散聚类分配 → 连续特征向量 → 可微路由
+  用 softmax(U_k · W_proj) 替代硬 k-means 分配
+- **幂迭代加速**：不需完整特征分解，只需前 k 个特征向量
+  用 Lanczos/Arnoldi 迭代 O(N²·k·iter) 或 randomized SVD O(N·k·log k)
+
+## AI 模块形式
+```
+模块：SpectralClusterRouter
+输入：X ∈ R^{N×d}，簇数 K
+
+方法1 - 在线谱聚类路由（训练时周期性更新）：
+  // 每 M 步更新一次聚类中心，推理时用最近邻
+  W = exp(-cdist(X_sample, X_sample) / (2σ²))  // m×m 采样相似度
+  L = I - D^{-1/2} W D^{-1/2}                    // 归一化拉普拉斯
+  U_k = eigsh(L, k=K, which='SM')                // 前 K 个最小特征向量
+  centers = kmeans(U_k, K)                        // K 个聚类中心
+  // 路由：将新 token 投影到谱空间后分配
+  proj = X @ W_proj                               // N→K 维投影（可学习）
+  assignment = argmin(cdist(proj, centers))        // 最近中心分配
+
+方法2 - 可微谱路由（端到端）：
+  // 用 softmax 松弛替代硬分配
+  sim_matrix = X @ X^T                             // N×N（或采样 m×m）
+  A = exp(sim_matrix / τ)                          // 相似度图（可学习 τ）
+  D_inv_sqrt = diag(1 / sqrt(sum(A, dim=1) + ε))
+  L_norm = I - D_inv_sqrt @ A @ D_inv_sqrt         // 归一化拉普拉斯
+  // 近似前 K 个特征向量（幂迭代 + 正交化）
+  U_k = power_iteration_approx(L_norm, K, steps=5)  // N×K
+  // 软分配
+  cluster_logits = U_k @ W_cluster                  // N×K（可学习投影）
+  route_probs = softmax(cluster_logits / τ_route)    // 软路由概率
+
+方法3 - 锚点谱聚类（大规模）：
+  anchors = kmeans_pp(X, m)                         // m 个锚点，m << N
+  Z = exp(-cdist(X, anchors) / (2σ²))               // N×m 亲和矩阵
+  L_anchor = I - D_z^{-1/2} Z^T Z D_z^{-1/2}        // m×m 拉普拉斯
+  U_k = eigsh(L_anchor, K)                          // m×K 特征向量
+  route = Z @ U_k @ W_proj                           // N×K 路由分数
+```
+
+## 可实现结构
+- **周期性离线聚类**：每 N_step 步收集 token 表示 → 离线谱聚类 → 更新路由表
+- **Nyström 采样**：随机采样 m=1024 个代表点，将 N×N 问题降为 m×m
+- **幂迭代实现**：5~10 步幂迭代 + Gram-Schmidt 正交化，GPU 友好
+- **渐进式训练**：初期用 k-means 粗路由 → 中期谱聚类精化 → 后期可微调路由网络
+
+## GPU 可行性
+- **张量化**：相似度矩阵 X@X^T 为 GEMM；拉普拉斯构造为 element-wise + 对角矩阵
+- **GEMM 可映射**：方法3 的 Z^T@Z 为 GEMM (m×N)@(N×m)；Z@U_k 为 GEMM (N×m)@(m×K)
+- **复杂度**：完整谱聚类 O(N²·K) 不可扩展；Nyström O(N·m·K+m³)；幂迭代 O(N²·K·T)
+- **显存与 KV-Cache**：N×N 相似度矩阵在 N>4096 时 >64MB，必须采样降维
+- **低精度稳定**：特征分解建议 fp32；exp(-dist/σ²) 在 fp16 下需 clip distance
+- **并行与通信**：幂迭代的 matvec 高度并行；k-means 的 assign+update 可批并行
+- **稀疏结构**：k-NN 图替代全连接图，W 稀疏度 >95%，SpMM 加速
+- **算子融合**：D^{-1/2}@A@D^{-1/2} 的对角缩放可融合；cdist+exp+normalize 可融合
+
+## 论文表述方式
+"利用谱聚类的连续松弛实现可微路由：构造 token 相似度图的归一化拉普拉斯，
+通过 Nyström 近似将 O(N²) 特征分解降为 O(Nm+K³)，配合幂迭代实现 GPU 友好的
+在线谱聚类，聚类质量以 Normalized Cut 衡量保证 O(√(log N/K)) 的近似比。"
+
+## 风险
+- N×N 相似度矩阵的显存和计算在长序列下不可扩展，必须采样或 k-NN 稀疏化
+- 特征分解不可微（特征值重合时梯度未定义），端到端训练需松弛或 stop-gradient
+- 簇数 K 需先验指定，且 K 变化时需重新聚类
+- σ（带宽参数）对聚类质量敏感，过小导致孤立点，过大导致合并

package/knowledge-base/differential-geometry/connection.en.md

@@ -0,0 +1,51 @@
+# Connection
+
+## Minimal Definition
+
+A connection $\nabla$ is a rule specifying "how to transport a vector along a curve from one point to another without extraneous rotation." Formally, $\nabla: \mathfrak{X}(M) \times \mathfrak{X}(M) \to \mathfrak{X}(M)$ satisfies the Leibniz rule. The Levi-Civita connection is the unique connection that is metric-compatible ($\nabla g = 0$) and torsion-free ($T = 0$).
+
+## Core Formulas
+
+- Covariant derivative: $\nabla_X Y = \left(X^i \partial_i Y^k + X^i Y^j \Gamma^k_{ij}\right) \partial_k$
+- Christoffel symbols (Levi-Civita): $\Gamma^k_{ij} = \frac{1}{2} g^{kl}(\partial_i g_{jl} + \partial_j g_{il} - \partial_l g_{ij})$
+- Parallel transport equation: $\frac{D V^k}{dt} = \dot V^k + \Gamma^k_{ij} \dot\gamma^i V^j = 0$
+- Connection form (on a principal bundle): $\omega \in \Omega^1(P, \mathfrak{g})$; the gauge field $A_\mu$ is a local connection form
+- Curvature = non-commutativity of the connection: $R(X,Y) = [\nabla_X, \nabla_Y] - \nabla_{[X,Y]}$
+
+## Applicable Problems
+
+- Cross-point vector comparison: tangent spaces at different points cannot be directly summed; a connection specifies the "transport rule"
+- Momentum/state transport in optimization: historical gradients in Riemannian Adam must be carried across steps via parallel transport
+- Gauge-equivariant networks: the freedom in choosing local coordinate frames (gauges) is aligned by the connection
+- Physically constrained systems: the electromagnetic field equals the curvature of a U(1) connection; Yang-Mills equals the curvature of a non-Abelian connection
+
+## AI Design Translation
+
+- **Vector transport module**: In Riemannian optimizers, transport the momentum $m_k \in T_{x_k}M$ to $T_{x_{k+1}}M$; closed-form transport (e.g., projection on the Stiefel manifold) can be expressed as GEMM
+- **Gauge-equivariant CNN**: On manifolds/meshes, each edge carries a $G$-connection element that aligns the local frames of adjacent points, making convolution kernels invariant to local coordinate choices
+- **Parallel transport regularization**: Penalize non-parallelism of the feature field under the connection $\|\nabla_X f\|^2$, enforcing smooth feature variation along the manifold
+- **Connection learning parameterization**: Parameterize Christoffel symbols as neural network outputs, learning the "optimal transport rule" on the data manifold
+
+## Engineering Feasibility
+
+GPU friendliness: the core challenge of connections is "serial ODE integration."
+- **Closed-form parallel transport** (specific manifolds such as SO(3), Stiefel): single-step matrix operations, batchable, GPU-friendly
+- **Parallel transport for general connections**: integrating $\dot{V} + \Gamma \dot\gamma V = 0$ along a curve is a serial ODE with poor parallelism
+- **Christoffel symbol computation**: involves partial derivatives of the metric $g$ and $g^{-1}$; if $g$ has a closed form, $O(n^3)$; otherwise more expensive
+- **Connections in gauge-equivariant CNNs**: one $G$-element action per edge (matrix-times-feature-vector), expressible as sparse matmul or batched small GEMM
+- Key adaptation: use a single-step retraction/closed-form transport instead of step-by-step ODE integration
+
+## Risks and Failure Conditions
+
+- **Step-by-step ODE integration for parallel transport**: Serial recurrence kills parallelism; closed-form transport or single-step approximations must be used
+- **Inconsistent left/right connection conventions**: Mixing left-invariant and right-invariant connections leads to misaligned gradients
+- **Improper handling of gauge freedom**: If the connection parameterization in gauge-equivariant networks is incomplete, equivariance silently breaks
+- **Numerical derivatives of Christoffel symbols**: Finite-difference estimation of $\partial_i g_{jk}$ is noisy; analytical formulas or autodiff are preferable
+- **Connection does not imply metric compatibility**: A connection need not admit a compatible metric (non-metric connections); incorrectly assuming compatibility leads to inconsistencies
+
+## Further References
+
+- Distillation notes: references/books/differential-geometry.md (Ch 12 Connections and Covariant Derivatives, Section 12.2 Connection Forms, Section 12.4 Ehresmann, Section 12.12 G-Connections)
+- Distillation notes: references/books/differential-geometry.md (Section 6.8 Principal Bundles, Section 9.8 Electromagnetism)
+- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 12 (Section 12.1--Section 12.12, complete connection theory)
+- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Section 13.1 Levi-Civita Connection

package/knowledge-base/differential-geometry/connection.md

@@ -0,0 +1,51 @@
+# 联络 (Connection)
+
+## 最小定义
+
+联络 $\nabla$ 是流形上"如何把一个向量沿曲线搬到另一点而不额外旋转"的规则。形式化地，$\nabla: \mathfrak{X}(M) \times \mathfrak{X}(M) \to \mathfrak{X}(M)$ 满足 Leibniz 法则。Levi-Civita 联络是唯一与度量相容（$\nabla g = 0$）且无挠（$T = 0$）的联络。
+
+## 核心公式
+
+- 协变导数：$\nabla_X Y = \left(X^i \partial_i Y^k + X^i Y^j \Gamma^k_{ij}\right) \partial_k$
+- Christoffel 符号（Levi-Civita）：$\Gamma^k_{ij} = \frac{1}{2} g^{kl}(\partial_i g_{jl} + \partial_j g_{il} - \partial_l g_{ij})$
+- 平行移动方程：$\frac{D V^k}{dt} = \dot V^k + \Gamma^k_{ij} \dot\gamma^i V^j = 0$
+- 联络形式（主丛）：$\omega \in \Omega^1(P, \mathfrak{g})$，规范场 $A_\mu$ 即局部联络形式
+- 曲率 = 联络的不交换性：$R(X,Y) = [\nabla_X, \nabla_Y] - \nabla_{[X,Y]}$
+
+## 适用问题
+
+- 跨点向量比较：不同点的切空间无法直接相加，需要联络规定的"搬运规则"
+- 优化中的动量/状态搬运：Riemannian Adam 中历史梯度需要通过 parallel transport 跨步搬运
+- 规范等变网络：局部坐标系（gauge）的选择自由由联络对齐
+- 物理约束系统：电磁场 = U(1) 联络曲率，Yang-Mills = 非阿贝尔联络曲率
+
+## AI 设计翻译
+
+- **Vector transport 模块**：Riemannian 优化器中，将动量 $m_k \in T_{x_k}M$ 搬到 $T_{x_{k+1}}M$；闭式 transport（如 Stiefel 上的投影）可 GEMM 化
+- **Gauge-equivariant CNN**：在流形/网格上，每条边携带一个 $G$-联络元素对齐相邻点的局部 frame，使卷积核对局部坐标选择不变
+- **Parallel transport 正则化**：惩罚特征场在联络下的非平行性 $\|\nabla_X f\|^2$，强制特征沿流形平滑变化
+- **联络学习的参数化**：将 Christoffel 符号参数化为神经网络输出，学习数据流形上的"最优搬运规则"
+
+## 工程可行性
+
+GPU 友好度：联络的核心挑战是"串行 ODE 积分"。
+- **闭式 parallel transport**（特定流形如 SO(3)、Stiefel）：单步矩阵运算，可 batched，GPU 友好
+- **一般联络的平行移动**：沿曲线积分 $\dot{V} + \Gamma \dot\gamma V = 0$ 是串行 ODE，并行性差
+- **Christoffel 符号计算**：涉及度量 $g$ 的偏导和 $g^{-1}$，若 $g$ 有闭式则 $O(n^3)$，否则更贵
+- **规范等变 CNN 中的联络**：每边一个 $G$-元素作用（矩阵乘特征向量），可表达为 sparse matmul 或 batched small GEMM
+- 关键改造：用一步 retraction/closed-form transport 替代逐步 ODE 积分
+
+## 风险与失效条件
+
+- **逐步 ODE 积分做平行移动**：串行递推杀死并行度，必须用闭式 transport 或一步近似
+- **左右联络约定不统一**：左不变 vs 右不变联络的选择不一致导致梯度错位
+- **规范自由度未正确处理**：gauge-equivariant 网络中若联络参数化不完备，等变性会悄悄破缺
+- **Christoffel 符号的数值导数**：用有限差分估计 $\partial_i g_{jk}$ 时噪声大，最好有解析式或 autodiff
+- **联络 ≠ 度量**：有联络不一定有相容度量（非度量联络），错误假设相容性会导致不一致
+
+## 深入参考
+
+- 蒸馏稿：references/books/differential-geometry.md（Ch 12 Connections and Covariant Derivatives, §12.2 联络形式, §12.4 Ehresmann, §12.12 G-联络）
+- 蒸馏稿：references/books/differential-geometry.md（§6.8 Principal Bundles, §9.8 Electromagnetism）
+- 原书：Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 12（§12.1-§12.12 完整联络理论）
+- 原书：Jeffrey M. Lee, *Manifolds and Differential Geometry*, §13.1 Levi-Civita Connection

package/knowledge-base/differential-geometry/curvature.en.md

@@ -0,0 +1,49 @@
+# Curvature
+
+## Minimal Definition
+
+Curvature measures whether a manifold "bends" and whether "paths are path-dependent." The Riemann curvature tensor $R(X,Y)Z = \nabla_X\nabla_Y Z - \nabla_Y\nabla_X Z - \nabla_{[X,Y]}Z$ describes the deflection of a vector after parallel transport around an infinitesimal parallelogram. The sectional curvature $K(\sigma)$ is its simplest scalar extraction.
+
+## Core Formulas
+
+- Riemann curvature tensor: $R^l_{ijk} = \partial_i \Gamma^l_{jk} - \partial_j \Gamma^l_{ik} + \Gamma^l_{im}\Gamma^m_{jk} - \Gamma^l_{jm}\Gamma^m_{ik}$
+- Ricci curvature (contraction): $R_{ij} = \sum_k R^k_{ikj}$
+- Scalar curvature: $S = \sum_{ij} g^{ij} R_{ij}$
+- Sectional curvature: $K(X,Y) = \frac{\langle R(X,Y)Y, X\rangle}{\|X\|^2\|Y\|^2 - \langle X,Y\rangle^2}$
+- Jacobi equation: $\frac{D^2 J}{dt^2} + R(J, \dot\gamma)\dot\gamma = 0$ (describes divergence/convergence of geodesics)
+- Hessian-vector product: $Hv = \nabla(\nabla L \cdot v)$, $O(N)$ estimation of curvature information
+
+## Applicable Problems
+
+- Loss landscape analysis: curvature determines conditioning and sharpness, distinguishing sharp minima from flat minima
+- Optimization trajectory stability: Jacobi fields describe the divergence/convergence of neighboring optimization trajectories
+- Generalization diagnostics: flat minima (low curvature) tend to generalize better
+- Manifold learning: the curvature of the data manifold guides latent space dimension and metric selection
+
+## AI Design Translation
+
+- **Curvature regularization (geometric perspective on SAM)**: Use Hessian-vector products to estimate $\max_v v^T H v$, penalizing sharp minima and preferring flat minima
+- **HVP-based diagnostic**: $\|Hv\|/\|v\|$ as a cheap proxy for loss landscape curvature, used for learning rate adaptation and early stopping
+- **Jacobi field trajectory monitoring**: Track the distance evolution between two neighboring optimization trajectories, the discrete analog of $J''(t) + R(J,\dot\gamma)\dot\gamma = 0$, to detect divergence/convergence
+- **Ricci-flow-inspired graph rewiring**: Use discrete Ricci curvature to guide dynamic adjustment of graph/attention structure (negative-curvature edges indicate bottlenecks that need additional connections)
+
+## Engineering Feasibility
+
+GPU friendliness: the core difficulty of curvature is that "the full tensor cannot be materialized."
+- **Riemann tensor**: 4th order, $n^4$ components, materializing it exhausts memory -- **explicit computation is prohibited**
+- **Hessian-vector product (HVP)**: Via Pearlmutter's algorithm, one forward pass + one backward pass yields $Hv$, $O(N)$ time and $O(N)$ memory, GPU-friendly
+- **Monte Carlo estimation of Ricci/scalar curvature**: Randomly sample directions $v$, $\mathbb{E}[v^T H v] = \text{tr}(H)$, using Hutchinson's estimator, GPU-friendly
+- **Jacobi fields**: Require integrating a second-order ODE along a trajectory, serial recurrence, GPU-unfriendly; in practice, discrete finite-difference approximations are used
+- Low precision: second-order derivatives in HVP are noisy under fp16, requiring fp32 accumulation
+
+## Risks and Failure Conditions
+
+- **Materializing the full Riemann/Hessian tensor**: $O(N^2)$ to $O(N^4)$ memory, impossible when $N \sim 10^9$
+- **Low signal-to-noise ratio in curvature estimation**: Monte Carlo estimation of HVP has high variance; with small batches, the signal may be drowned in noise
+- **Treating curvature regularization as a panacea**: Curvature estimation itself is expensive (requiring additional forward and backward passes); benefits should be validated at small scale first when uncertain
+- **Discrete approximation errors**: When using finite differences to approximate Jacobi fields/HVP, the step size is sensitive -- too large causes truncation error, too small causes floating-point cancellation
+
+## Further References
+
+- Distillation notes: references/books/differential-geometry.md (Ch 12 Section 12.5/Section 12.10 Curvature, Ch 13 Section 13.2 Riemann Curvature, Section 13.7 Jacobi Fields, Section 13.11 Rauch Comparison)
+- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Section 13.2 Riemann Curvature Tensor, Section 13.7 Jacobi Fields

package/knowledge-base/differential-geometry/curvature.md

@@ -0,0 +1,49 @@
+# 曲率 (Curvature)
+
+## 最小定义
+
+曲率度量流形"弯不弯"以及"路径依不依赖"。Riemann 曲率张量 $R(X,Y)Z = \nabla_X\nabla_Y Z - \nabla_Y\nabla_X Z - \nabla_{[X,Y]}Z$ 描述向量绕无穷小平行四边形平行移动一圈后的偏转。截面曲率 $K(\sigma)$ 是其最简标量提取。
+
+## 核心公式
+
+- Riemann 曲率张量：$R^l_{ijk} = \partial_i \Gamma^l_{jk} - \partial_j \Gamma^l_{ik} + \Gamma^l_{im}\Gamma^m_{jk} - \Gamma^l_{jm}\Gamma^m_{ik}$
+- Ricci 曲率（缩并）：$R_{ij} = \sum_k R^k_{ikj}$
+- 标量曲率：$S = \sum_{ij} g^{ij} R_{ij}$
+- 截面曲率：$K(X,Y) = \frac{\langle R(X,Y)Y, X\rangle}{\|X\|^2\|Y\|^2 - \langle X,Y\rangle^2}$
+- Jacobi 方程：$\frac{D^2 J}{dt^2} + R(J, \dot\gamma)\dot\gamma = 0$（描述测地线的发散/汇聚）
+- Hessian-vector product：$Hv = \nabla(\nabla L \cdot v)$，$O(N)$ 估计曲率信息
+
+## 适用问题
+
+- 损失地形分析：曲率决定条件数与尖锐度，尖锐极小 vs 平坦极小的区分
+- 优化轨迹稳定性：Jacobi 场描述相邻优化轨迹的发散/汇聚
+- 泛化性诊断：平坦极小（低曲率）往往泛化更好
+- 流形学习：数据流形的曲率指导隐空间维数和度量选择
+
+## AI 设计翻译
+
+- **曲率正则化（SAM 的几何视角）**：用 Hessian-vector product 估计 $\max_v v^T H v$，惩罚尖锐极小，偏好 flat minima
+- **HVP-based 诊断器**：$\|Hv\|/\|v\|$ 作为 loss landscape 曲率的廉价代理，用于学习率自适应和早停
+- **Jacobi 场轨迹监控**：跟踪两条相邻优化轨迹的距离变化，$J''(t) + R(J,\dot\gamma)\dot\gamma = 0$ 的离散版，检测发散/收敛
+- **Ricci-flow 启发式图重连**：用离散 Ricci 曲率指导图/注意力结构的动态调整（负曲率边=瓶颈，需增连）
+
+## 工程可行性
+
+GPU 友好度：曲率的核心难点是"不能物化全张量"。
+- **Riemann 张量**：4 阶，$n^4$ 个分量，物化即爆显存，**禁止显式计算**
+- **Hessian-vector product (HVP)**：通过 Pearlmutter 算法，一次前向 + 一次反向即可得到 $Hv$，$O(N)$ 时间 $O(N)$ 显存，GPU 友好
+- **Ricci/标量曲率的 Monte Carlo 估计**：随机采样方向 $v$，$\mathbb{E}[v^T H v] = \text{tr}(H)$，用 Hutchinson 估计，GPU 友好
+- **Jacobi 场**：需要沿轨迹积分二阶 ODE，串行递推，GPU 不友好；工程上用离散差分近似
+- 低精度：HVP 中的二阶导在 fp16 下噪声大，需 fp32 累加
+
+## 风险与失效条件
+
+- **物化全 Riemann/Hessian 张量**：$O(N^2)$~$O(N^4)$ 显存，$N \sim 10^9$ 时不可能
+- **曲率估计信噪比低**：HVP 的 Monte Carlo 估计方差大，小 batch 下信号可能被噪声淹没
+- **把曲率正则当万能药**：曲率估计本身昂贵（每次需额外前向+反向），收益不确定时需先小规模验证
+- **离散近似误差**：用有限差分近似 Jacobi 场/HVP 时，步长选择敏感——太大截断误差大，太小浮点抵消
+
+## 深入参考
+
+- 蒸馏稿：references/books/differential-geometry.md（Ch 12 §12.5/§12.10 Curvature, Ch 13 §13.2 Riemann Curvature, §13.7 Jacobi Fields, §13.11 Rauch Comparison）
+- 原书：Jeffrey M. Lee, *Manifolds and Differential Geometry*, §13.2 Riemann Curvature Tensor, §13.7 Jacobi Fields

package/knowledge-base/differential-geometry/geodesic.en.md

@@ -0,0 +1,51 @@
+# Geodesic
+
+## Minimal Definition
+
+A geodesic is a locally length-minimizing (or extremal-length) path $\gamma: [0,1] \to M$ on a manifold, satisfying the auto-parallel condition $\nabla_{\dot\gamma} \dot\gamma = 0$ -- that is, the tangent vector is parallel-transported unchanged along itself. The exponential map $\exp_p(v)$ maps a tangent vector $v \in T_pM$ to the endpoint at $t=1$ of the geodesic starting at $p$ with initial velocity $v$.
+
+## Core Formulas
+
+- Geodesic equation: $\ddot\gamma^k + \sum_{ij} \Gamma^k_{ij} \dot\gamma^i \dot\gamma^j = 0$
+- Exponential map: $\exp_p(v) = \gamma_v(1)$, where $\gamma_v$ is the geodesic with $\gamma(0)=p, \dot\gamma(0)=v$
+- Logarithmic map: $\log_p(q) = v \in T_pM$ such that $\exp_p(v) = q$ (the inverse of the exponential map)
+- Retraction (engineering approximation): $R_p(v) \approx \exp_p(v)$, requiring only a first-order approximation for optimization
+- Closed-form on the sphere: $\exp_p(v) = \cos(\|v\|) p + \sin(\|v\|) \frac{v}{\|v\|}$
+
+## Applicable Problems
+
+- Constrained optimization: performing SGD on SPD/Stiefel/Grassmann/hyperbolic manifolds, where update steps must move along geodesics
+- Latent space interpolation: geodesics between two points in the latent space respect the data manifold structure better than Euclidean straight lines
+- Distance computation on manifolds: $d(p,q) = \|\log_p(q)\|_g$
+- Data augmentation: sampling along geodesics to generate new training samples
+
+## AI Design Translation
+
+- **Retraction-based optimizer**: Each step performs $x_{k+1} = R_{x_k}(-\eta \cdot \text{grad})$, using closed-form retractions instead of ODE integration; rotations for the sphere, QR/Cayley for Stiefel, Rodrigues for SO(3)
+- **Geodesic interpolation layer**: Use closed-form geodesics in spherical/hyperbolic latent spaces for mixup and interpolation, $\gamma(t) = \exp_p(t \cdot \log_p(q))$
+- **Momentum on manifolds**: Transport the momentum vector from $T_{x_k}M$ to $T_{x_{k+1}}M$ via vector transport (the discrete analog of parallel transport), then combine with the new gradient
+- **Exponential map output head**: The network makes unconstrained predictions in the tangent space $\mathbb{R}^n$, then projects back to the valid manifold via $\exp_p$, naturally satisfying constraints
+
+## Engineering Feasibility
+
+GPU friendliness depends on whether a closed-form retraction exists:
+- **Manifolds with closed forms** (sphere, hyperbolic, SO(3), Stiefel-QR): $\exp_p(v)$ is a finite algebraic expression, $O(1)$/sample, tensorizable in batches, GPU-friendly
+- **Manifolds without closed forms**: require numerical integration of the geodesic equation (second-order ODE), serial recurrence, GPU-unfriendly
+- Closed-form retractions as substitutes for exact exp: QR decomposition, Cayley transform, and other first-order approximations trade a small amount of precision for significant speedup
+- Small-matrix exp for 3x3/4x4 (SO(3)/SE(3)) can be fused into a single kernel, but cannot fully saturate Tensor Cores
+- **Low-precision critical issue**: $\sin\theta/\theta$ and $(1-\cos\theta)/\theta^2$ produce division by zero as $\theta \to 0$; $\log$ is singular as $\theta \to \pi$; fp16 yields NaN directly
+
+## Risks and Failure Conditions
+
+- **Step-by-step ODE integration for geodesics**: Serial recurrence kills GPU parallelism; closed-form retractions must be used instead
+- **Small-angle / large-angle singularities**: Numerical instability at $\theta \to 0$ and $\theta \to \pi$ is catastrophically amplified at low precision; Taylor expansion fallbacks are essential
+- **Retraction error accumulation**: First-order approximations may accumulate errors over multiple iterations, requiring occasional exact projection corrections
+- **Cut locus problem**: Beyond the cut locus, the exponential map no longer yields the shortest path; $\log_p(q)$ may not exist or may not be unique
+- **Forced manifold structure for geometric aesthetics**: Applying geodesics to tasks where Euclidean approximations suffice adds complexity and singularity risk
+
+## Further References
+
+- Distillation notes: references/books/differential-geometry.md (Ch 13 Section 13.4 Geodesics, Section 13.11 Rauch Comparison)
+- Distillation notes: references/books/smooth-manifolds.md (Ch 20 The Exponential Map)
+- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Section 13.4 Geodesics
+- Original text: John M. Lee, *Introduction to Smooth Manifolds*, Ch 20 (exponential map, retraction prototype)