math-skill 3.0.1 → 3.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en-US.md +72 -47
- package/README.md +40 -15
- package/agents/math-critic.en.md +6 -6
- package/agents/math-critic.md +6 -6
- package/commands/ask.en.md +6 -1
- package/commands/ask.md +6 -1
- package/design-patterns/attention/equivariant-attention.en.md +8 -2
- package/design-patterns/attention/equivariant-attention.md +7 -1
- package/design-patterns/attention/geometry-aware-attention.en.md +2 -2
- package/design-patterns/attention/geometry-aware-attention.md +1 -1
- package/design-patterns/attention/information-bottleneck-attention.en.md +32 -12
- package/design-patterns/attention/information-bottleneck-attention.md +32 -12
- package/design-patterns/attention/projection-attention.en.md +3 -3
- package/design-patterns/attention/projection-attention.md +2 -2
- package/design-patterns/attention/spectral-attention.en.md +19 -5
- package/design-patterns/attention/spectral-attention.md +18 -4
- package/design-patterns/compression/leverage-score-selection.en.md +13 -13
- package/design-patterns/compression/leverage-score-selection.md +13 -13
- package/design-patterns/compression/low-rank-kv-cache.en.md +39 -12
- package/design-patterns/compression/low-rank-kv-cache.md +38 -12
- package/design-patterns/compression/spectral-token-pruning.en.md +16 -14
- package/design-patterns/compression/spectral-token-pruning.md +16 -14
- package/design-patterns/compression/topology-preserving-compression.en.md +10 -10
- package/design-patterns/compression/topology-preserving-compression.md +10 -10
- package/design-patterns/loss/constraint-penalty.en.md +8 -4
- package/design-patterns/loss/constraint-penalty.md +9 -6
- package/design-patterns/loss/contrastive-loss.en.md +37 -11
- package/design-patterns/loss/contrastive-loss.md +38 -12
- package/design-patterns/loss/information-bottleneck-loss.en.md +5 -5
- package/design-patterns/loss/information-bottleneck-loss.md +6 -6
- package/design-patterns/loss/orthogonality-loss.en.md +24 -17
- package/design-patterns/loss/orthogonality-loss.md +24 -18
- package/design-patterns/loss/variational-loss.en.md +4 -4
- package/design-patterns/loss/variational-loss.md +5 -5
- package/design-patterns/overview.en.md +36 -0
- package/design-patterns/overview.md +36 -0
- package/design-patterns/representation/equivariant-split.en.md +4 -4
- package/design-patterns/representation/equivariant-split.md +4 -5
- package/design-patterns/representation/manifold-representation.en.md +12 -12
- package/design-patterns/representation/manifold-representation.md +13 -14
- package/design-patterns/representation/shared-private-decomposition.en.md +12 -7
- package/design-patterns/representation/shared-private-decomposition.md +12 -9
- package/design-patterns/representation/subspace-alignment.en.md +12 -12
- package/design-patterns/representation/subspace-alignment.md +12 -12
- package/design-patterns/routing/graph-routing.en.md +11 -11
- package/design-patterns/routing/graph-routing.md +11 -11
- package/design-patterns/routing/moe-routing.en.md +12 -12
- package/design-patterns/routing/moe-routing.md +12 -14
- package/design-patterns/routing/optimal-transport-routing.en.md +5 -5
- package/design-patterns/routing/optimal-transport-routing.md +6 -6
- package/design-patterns/routing/spectral-clustering-routing.en.md +26 -15
- package/design-patterns/routing/spectral-clustering-routing.md +26 -17
- package/knowledge-base/differential-geometry/connection.en.md +17 -2
- package/knowledge-base/differential-geometry/connection.md +17 -2
- package/knowledge-base/differential-geometry/curvature.en.md +14 -1
- package/knowledge-base/differential-geometry/curvature.md +14 -1
- package/knowledge-base/differential-geometry/geodesic.en.md +16 -2
- package/knowledge-base/differential-geometry/geodesic.md +16 -2
- package/knowledge-base/differential-geometry/index.en.md +52 -0
- package/knowledge-base/differential-geometry/index.md +52 -0
- package/knowledge-base/differential-geometry/manifold.en.md +15 -2
- package/knowledge-base/differential-geometry/manifold.md +15 -2
- package/knowledge-base/differential-geometry/metric-tensor.en.md +16 -2
- package/knowledge-base/differential-geometry/metric-tensor.md +16 -2
- package/knowledge-base/differential-geometry/tangent-space.en.md +17 -3
- package/knowledge-base/differential-geometry/tangent-space.md +17 -3
- package/knowledge-base/information-geometry/fisher-metric.en.md +24 -5
- package/knowledge-base/information-geometry/fisher-metric.md +24 -5
- package/knowledge-base/information-geometry/index.en.md +45 -0
- package/knowledge-base/information-geometry/index.md +45 -0
- package/knowledge-base/information-geometry/natural-gradient.en.md +17 -3
- package/knowledge-base/information-geometry/natural-gradient.md +17 -3
- package/knowledge-base/lie-theory/equivariance.en.md +18 -3
- package/knowledge-base/lie-theory/equivariance.md +18 -3
- package/knowledge-base/lie-theory/group-action.en.md +16 -2
- package/knowledge-base/lie-theory/group-action.md +16 -2
- package/knowledge-base/lie-theory/index.en.md +52 -0
- package/knowledge-base/lie-theory/index.md +52 -0
- package/knowledge-base/lie-theory/lie-algebra.en.md +17 -2
- package/knowledge-base/lie-theory/lie-algebra.md +17 -2
- package/knowledge-base/lie-theory/lie-group.en.md +17 -3
- package/knowledge-base/lie-theory/lie-group.md +17 -3
- package/knowledge-base/lie-theory/representation.en.md +17 -2
- package/knowledge-base/lie-theory/representation.md +17 -2
- package/knowledge-base/matrix-analysis/index.en.md +49 -0
- package/knowledge-base/matrix-analysis/index.md +49 -0
- package/knowledge-base/matrix-analysis/low-rank-approximation.en.md +20 -7
- package/knowledge-base/matrix-analysis/low-rank-approximation.md +20 -7
- package/knowledge-base/matrix-analysis/matrix-perturbation.en.md +13 -1
- package/knowledge-base/matrix-analysis/matrix-perturbation.md +13 -1
- package/knowledge-base/matrix-analysis/positive-semidefinite.en.md +20 -6
- package/knowledge-base/matrix-analysis/positive-semidefinite.md +20 -6
- package/knowledge-base/matrix-analysis/projection.en.md +14 -1
- package/knowledge-base/matrix-analysis/projection.md +14 -1
- package/knowledge-base/matrix-analysis/spectral-decomposition.en.md +14 -1
- package/knowledge-base/matrix-analysis/spectral-decomposition.md +14 -1
- package/knowledge-base/optimization/constrained-optimization.en.md +17 -3
- package/knowledge-base/optimization/constrained-optimization.md +17 -3
- package/knowledge-base/optimization/convex-optimization.en.md +15 -2
- package/knowledge-base/optimization/convex-optimization.md +15 -2
- package/knowledge-base/optimization/index.en.md +50 -0
- package/knowledge-base/optimization/index.md +50 -0
- package/knowledge-base/optimization/lagrangian-duality.en.md +18 -4
- package/knowledge-base/optimization/lagrangian-duality.md +18 -4
- package/knowledge-base/optimization/proximal-method.en.md +15 -1
- package/knowledge-base/optimization/proximal-method.md +15 -1
- package/knowledge-base/optimization/riemannian-optimization.en.md +18 -4
- package/knowledge-base/optimization/riemannian-optimization.md +18 -4
- package/knowledge-base/overview.en.md +20 -8
- package/knowledge-base/overview.md +20 -8
- package/knowledge-base/probability/concentration-inequality.en.md +16 -2
- package/knowledge-base/probability/concentration-inequality.md +16 -2
- package/knowledge-base/probability/entropy.en.md +19 -3
- package/knowledge-base/probability/entropy.md +18 -2
- package/knowledge-base/probability/fisher-information.en.md +17 -3
- package/knowledge-base/probability/fisher-information.md +17 -3
- package/knowledge-base/probability/index.en.md +51 -0
- package/knowledge-base/probability/index.md +51 -0
- package/knowledge-base/probability/information-bottleneck.en.md +18 -2
- package/knowledge-base/probability/information-bottleneck.md +18 -2
- package/knowledge-base/probability/kl-divergence.en.md +15 -2
- package/knowledge-base/probability/kl-divergence.md +15 -2
- package/knowledge-base/topology/euler-characteristic.en.md +15 -2
- package/knowledge-base/topology/euler-characteristic.md +15 -2
- package/knowledge-base/topology/fundamental-group.en.md +16 -2
- package/knowledge-base/topology/fundamental-group.md +16 -2
- package/knowledge-base/topology/index.en.md +48 -0
- package/knowledge-base/topology/index.md +48 -0
- package/knowledge-base/topology/persistent-homology.en.md +17 -2
- package/knowledge-base/topology/persistent-homology.md +17 -2
- package/lenses/spectral.en.md +1 -1
- package/lenses/spectral.md +1 -1
- package/lenses/topological.en.md +1 -1
- package/lenses/topological.md +1 -1
- package/package.json +6 -6
- package/references/agentic-workflow.en.md +2 -2
- package/references/books/abstract-algebra.en.md +12 -12
- package/references/books/abstract-algebra.md +9 -9
- package/references/books/algebraic-geometry-rising-sea.en.md +2 -2
- package/references/books/differential-geometry.en.md +16 -16
- package/references/books/differential-geometry.md +14 -14
- package/references/books/matrix-analysis.en.md +3 -3
- package/references/books/matrix-analysis.md +24 -24
- package/references/books/micro-lie-theory.en.md +1 -1
- package/references/books/optimization-ml.en.md +15 -15
- package/references/books/optimization-ml.md +14 -14
- package/references/books/smooth-manifolds.en.md +3 -3
- package/references/books/smooth-manifolds.md +6 -6
- package/references/gpu-friendly-math.en.md +21 -8
- package/references/gpu-friendly-math.md +19 -4
- package/references/skill-index.en.md +106 -0
- package/references/skill-index.md +106 -0
- package/skills/math-research-activator/SKILL.en.md +76 -118
- package/skills/math-research-activator/SKILL.md +69 -109
|
@@ -10,9 +10,9 @@ Use when two or more representation spaces need to be aligned to a common subspa
|
|
|
10
10
|
Core requirement: **find the optimal linear/nonlinear mapping between two spaces such that corresponding semantics are aligned**.
|
|
11
11
|
|
|
12
12
|
## Mathematical Inspiration
|
|
13
|
-
- Lenses: lenses/geometric.md (Grassmann manifold, principal angles), lenses/variational.md (Procrustes problem)
|
|
14
|
-
- Knowledge: knowledge-base/matrix-analysis/projection.md (SVD, orthogonal Procrustes),
|
|
15
|
-
knowledge-base/differential-geometry/manifold.md (Grassmann distance, geodesics)
|
|
13
|
+
- Lenses: ../../lenses/geometric.en.md (Grassmann manifold, principal angles), ../../lenses/variational.en.md (Procrustes problem)
|
|
14
|
+
- Knowledge: ../../knowledge-base/matrix-analysis/projection.en.md (SVD, orthogonal Procrustes),
|
|
15
|
+
../../knowledge-base/differential-geometry/manifold.en.md (Grassmann distance, geodesics)
|
|
16
16
|
|
|
17
17
|
## Required Mathematical Background
|
|
18
18
|
- **Orthogonal Procrustes Problem**: min_{W in O(d)} ||AW - B||_F^2
|
|
@@ -71,17 +71,17 @@ Online Subspace Tracking (adaptation at inference):
|
|
|
71
71
|
- **Chunked SVD**: Use randomized SVD approximation at large scale; sufficient precision and faster
|
|
72
72
|
|
|
73
73
|
## GPU Feasibility
|
|
74
|
-
- **
|
|
75
|
-
- **
|
|
76
|
-
- **
|
|
77
|
-
- **
|
|
78
|
-
- **
|
|
79
|
-
- **
|
|
80
|
-
- **
|
|
81
|
-
- **
|
|
74
|
+
- **D1[v]**: A^T @ B is standard GEMM (d x N) @ (N x d); SVD has cuSOLVER implementations
|
|
75
|
+
- **D2[v]**: Procrustes core requires 1 GEMM + 1 SVD; CCA requires 2 GEMMs + 1 SVD
|
|
76
|
+
- **D3[v]**: GEMM O(N * d^2); SVD O(d^3) (d typically < 1024, acceptable); online tracking O(d^2)
|
|
77
|
+
- **D4[v]**: Storing covariance matrix d x d (~4 MB for d = 1024); no KV-Cache overhead
|
|
78
|
+
- **D5[~]**: SVD strongly recommended in fp32; whitening matrix inversion requires fp32 + epsilon regularization
|
|
79
|
+
- **D6[v]**: Subspace angle computations for multiple expert pairs are independent and parallel; CCA GEMMs are highly parallel
|
|
80
|
+
- **D7[~]**: When source/target representations are sparse, covariance matrix Sigma is sparse, enabling sparse SVD
|
|
81
|
+
- **D8[~]**: Whitening (mean-sub -> cov -> inv_sqrt -> transform) can be partially fused
|
|
82
82
|
|
|
83
83
|
## Paper-Worthy Formulation
|
|
84
|
-
"Based on orthogonal Procrustes theory, we obtain the optimal isometric mapping W* = UV^T (where USV^T = SVD(A^T B)) between source and target representations. Extending this to Deep CCA for nonlinear subspace alignment, principal
|
|
84
|
+
"Based on orthogonal Procrustes theory, we obtain the optimal isometric mapping W* = UV^T (where USV^T = SVD(A^T B)) between source and target representations. Extending this to Deep CCA for nonlinear subspace alignment, principal angles on the Grassmann manifold can measure pre/post alignment distance. Convergence rates depend on sample independence, spectral gaps, and covariance-estimation assumptions; the Barlow Twins objective penalizes cross-correlation but does not by itself guarantee semantic disentanglement, so report principal angles, CCA correlations, and downstream transfer metrics."
|
|
85
85
|
|
|
86
86
|
## Risks
|
|
87
87
|
- SVD during backpropagation produces undefined gradients when singular values coincide; epsilon regularization is needed
|
|
@@ -10,9 +10,9 @@
|
|
|
10
10
|
核心诉求:**找到两个空间之间的最优线性/非线性映射,使对应语义对齐**。
|
|
11
11
|
|
|
12
12
|
## 数学思想来源
|
|
13
|
-
-
|
|
14
|
-
-
|
|
15
|
-
knowledge-base/differential-geometry/manifold.md(Grassmann 距离、测地线)
|
|
13
|
+
- 透镜:../../lenses/geometric.md(Grassmann 流形、主角度)、../../lenses/variational.md(Procrustes 问题)
|
|
14
|
+
- 知识:../../knowledge-base/matrix-analysis/projection.md(SVD、正交 Procrustes)、
|
|
15
|
+
../../knowledge-base/differential-geometry/manifold.md(Grassmann 距离、测地线)
|
|
16
16
|
|
|
17
17
|
## 需要的数学知识
|
|
18
18
|
- **正交 Procrustes 问题**:min_{W∈O(d)} ‖AW - B‖_F²
|
|
@@ -71,19 +71,19 @@
|
|
|
71
71
|
- **分块 SVD**:大规模时用 randomized SVD 近似,精度足够且更快
|
|
72
72
|
|
|
73
73
|
## GPU 可行性
|
|
74
|
-
-
|
|
75
|
-
- **
|
|
76
|
-
-
|
|
77
|
-
-
|
|
78
|
-
-
|
|
79
|
-
-
|
|
80
|
-
-
|
|
81
|
-
-
|
|
74
|
+
- **D1[v]**:A^T@B 为标准 GEMM (d×N)@(N×d);SVD 有 cuSOLVER 实现
|
|
75
|
+
- **D2[v]**:Procrustes 核心 1 次 GEMM + 1 次 SVD;CCA 2 次 GEMM + 1 次 SVD
|
|
76
|
+
- **D3[v]**:GEMM O(N·d²);SVD O(d³)(d 通常 <1024,可接受);在线追踪 O(d²)
|
|
77
|
+
- **D4[v]**:存储协方差矩阵 d×d(~4MB for d=1024),不增加 KV-Cache
|
|
78
|
+
- **D5[~]**:SVD 强烈建议 fp32;白化的矩阵逆需 fp32 + ε 正则化
|
|
79
|
+
- **D6[v]**:多专家对的子空间角度计算独立并行;CCA 的 GEMM 高度并行
|
|
80
|
+
- **D7[~]**:当源/目标表示稀疏时,协方差矩阵 Σ 稀疏,可用稀疏 SVD
|
|
81
|
+
- **D8[~]**:白化 (mean-sub → cov → inv_sqrt → transform) 可部分融合
|
|
82
82
|
|
|
83
83
|
## 论文表述方式
|
|
84
84
|
"基于正交 Procrustes 理论求得源-目标表示间的最优等距映射 W*=UV^T(USV^T=SVD(A^TB)),
|
|
85
85
|
将其扩展为深度 CCA 实现非线性子空间对齐,Grassmann 流形上的主角度分析表明
|
|
86
|
-
|
|
86
|
+
对齐前后的子空间距离可由主角度度量。收敛速率依赖样本独立性、谱间隙和协方差估计条件;Barlow Twins 目标只惩罚交叉相关,不能单独保证语义解耦,应报告主角度、CCA 相关系数和下游迁移指标。"
|
|
87
87
|
|
|
88
88
|
## 风险
|
|
89
89
|
- SVD 在反向传播时奇异值重合导致梯度未定义,需添加 ε 正则化
|
|
@@ -10,9 +10,9 @@ Use when there exists a known or learnable topological structure among modules/e
|
|
|
10
10
|
Core requirement: **leverage structural priors to constrain routing decisions and reduce the search space**.
|
|
11
11
|
|
|
12
12
|
## Mathematical Inspiration
|
|
13
|
-
- Lenses: lenses/geometric.md (graph Laplacian, spectral graph theory), lenses/probabilistic.md (message passing, information flow)
|
|
14
|
-
- Knowledge: knowledge-base/matrix-analysis/projection.md (adjacency matrix, spectral decomposition),
|
|
15
|
-
knowledge-base/optimization/lagrangian-duality.md (variational on graphs, diffusion processes)
|
|
13
|
+
- Lenses: ../../lenses/geometric.en.md (graph Laplacian, spectral graph theory), ../../lenses/probabilistic.en.md (message passing, information flow)
|
|
14
|
+
- Knowledge: ../../knowledge-base/matrix-analysis/projection.en.md (adjacency matrix, spectral decomposition),
|
|
15
|
+
../../knowledge-base/optimization/lagrangian-duality.en.md (variational on graphs, diffusion processes)
|
|
16
16
|
|
|
17
17
|
## Required Mathematical Background
|
|
18
18
|
- **Graph Laplacian**: L = D - A (combinatorial) or L_sym = D^{-1/2} L D^{-1/2} (normalized)
|
|
@@ -60,14 +60,14 @@ Method 3 - Hierarchical Tree Routing (O(log K) complexity):
|
|
|
60
60
|
- **Hierarchical tree implementation**: Represented as a complete binary tree array with level-wise vectorization
|
|
61
61
|
|
|
62
62
|
## GPU Feasibility
|
|
63
|
-
- **
|
|
64
|
-
- **
|
|
65
|
-
- **
|
|
66
|
-
- **
|
|
67
|
-
- **
|
|
68
|
-
- **
|
|
69
|
-
- **
|
|
70
|
-
- **
|
|
63
|
+
- **D1[v]**: GCN layers perform sparse matrix times dense matrix (SpMM), supported by both PyTorch and cuSPARSE
|
|
64
|
+
- **D2[v]**: score_init @ P_t in Method 1 is standard GEMM (N x K) @ (K x K)
|
|
65
|
+
- **D3[v]**: Method 1 O(N * K^2) diffusion + O(N * K * d) gate; Method 3 O(N * d * log K) significantly better than O(N * d * K)
|
|
66
|
+
- **D4[v]**: A matrix stored sparsely at K x K; hierarchical tree parameters d x log K are negligible
|
|
67
|
+
- **D5[v]**: Probability matrix P and softmax in fp16 require attention to normalization precision
|
|
68
|
+
- **D6[v]**: GNN message passing can be batched in parallel; hierarchical tree nodes at the same level can be evaluated independently in parallel
|
|
69
|
+
- **D7[v]**: Graph adjacency matrices are naturally sparse (degree << K); SpMM reduces from O(K^2) to O(K * avg_deg)
|
|
70
|
+
- **D8[v]**: L_norm @ H @ W in GCN can be fused into a single sparse GEMM
|
|
71
71
|
|
|
72
72
|
## Paper-Worthy Formulation
|
|
73
73
|
"Leveraging hierarchical tree/graph topology among experts, we compress routing decisions from a flat O(N * K) search to O(N * log K) tree traversal or O(N * K * avg_deg) graph diffusion. Fiedler spectral analysis reveals that the graph's algebraic connectivity lambda_2 directly governs the diversity-coherence trade-off in routing."
|
|
@@ -10,9 +10,9 @@
|
|
|
10
10
|
核心诉求:**利用结构化先验约束路由决策,减少搜索空间**。
|
|
11
11
|
|
|
12
12
|
## 数学思想来源
|
|
13
|
-
-
|
|
14
|
-
-
|
|
15
|
-
knowledge-base/optimization/lagrangian-duality.md(图上优化、扩散过程)
|
|
13
|
+
- 透镜:../../lenses/geometric.md(图拉普拉斯、谱图论)、../../lenses/probabilistic.md(消息传递、信息流)
|
|
14
|
+
- 知识:../../knowledge-base/matrix-analysis/projection.md(邻接矩阵、谱分解)、
|
|
15
|
+
../../knowledge-base/optimization/lagrangian-duality.md(图上优化、扩散过程)
|
|
16
16
|
|
|
17
17
|
## 需要的数学知识
|
|
18
18
|
- **图拉普拉斯**:L = D - A(组合)或 L_sym = D^{-1/2} L D^{-1/2}(归一化)
|
|
@@ -60,14 +60,14 @@
|
|
|
60
60
|
- **层次树实现**:用完全二叉树的数组表示,level-wise 向量化
|
|
61
61
|
|
|
62
62
|
## GPU 可行性
|
|
63
|
-
-
|
|
64
|
-
- **
|
|
65
|
-
-
|
|
66
|
-
-
|
|
67
|
-
-
|
|
68
|
-
-
|
|
69
|
-
-
|
|
70
|
-
-
|
|
63
|
+
- **D1[v]**:GCN 层为稀疏矩阵×稠密矩阵 (SpMM),PyTorch 和 cuSPARSE 均支持
|
|
64
|
+
- **D2[v]**:方法1 的 score_init@P_t 为标准 GEMM (N×K)@(K×K)
|
|
65
|
+
- **D3[v]**:方法1 O(N·K²) 扩散 + O(N·K·d) gate;方法3 O(N·d·log K) 显著优于 O(N·d·K)
|
|
66
|
+
- **D4[v]**:A 矩阵 K×K 稀疏存储;层次树参数 d×log K 极小
|
|
67
|
+
- **D5[v]**:概率矩阵 P 和 softmax 在 fp16 下需注意归一化精度
|
|
68
|
+
- **D6[v]**:GNN 消息传递可批量并行;层次树同层节点独立可并行判断
|
|
69
|
+
- **D7[v]**:图邻接矩阵天然稀疏(度 << K),SpMM 加速比 O(K²) 到 O(K·avg_deg)
|
|
70
|
+
- **D8[v]**:GCN 的 L_norm@H@W 可融合为单次稀疏 GEMM
|
|
71
71
|
|
|
72
72
|
## 论文表述方式
|
|
73
73
|
"利用专家间的层次树/图拓扑结构,将路由决策从 O(N·K) 的平铺搜索压缩为 O(N·log K) 的
|
|
@@ -9,9 +9,9 @@ Typical scenarios: (1) Sparse MoE layers -- each token selects top-k experts (k
|
|
|
9
9
|
Core requirement: **sparse activation, load balancing, end-to-end trainability**.
|
|
10
10
|
|
|
11
11
|
## Mathematical Inspiration
|
|
12
|
-
- Lenses: lenses/variational.md (discrete variational relaxation, Gumbel-Softmax), lenses/probabilistic.md (information-theoretic routing)
|
|
13
|
-
- Knowledge: knowledge-base/optimization/lagrangian-duality.md (combinatorial variational, integer programming relaxation),
|
|
14
|
-
knowledge-base/probability/entropy.md (entropy regularization, information bottleneck)
|
|
12
|
+
- Lenses: ../../lenses/variational.en.md (discrete variational relaxation, Gumbel-Softmax), ../../lenses/probabilistic.en.md (information-theoretic routing)
|
|
13
|
+
- Knowledge: ../../knowledge-base/optimization/lagrangian-duality.en.md (combinatorial variational, integer programming relaxation),
|
|
14
|
+
../../knowledge-base/probability/entropy.en.md (entropy regularization, information bottleneck)
|
|
15
15
|
|
|
16
16
|
## Required Mathematical Background
|
|
17
17
|
- **Mixture Model EM**: p(y|x) = sum_k pi_k(x) * p(y|x, theta_k)
|
|
@@ -66,17 +66,17 @@ Auxiliary loss:
|
|
|
66
66
|
- **Router Z-loss**: L_z = alpha * mean(logsumexp(logits)^2) stabilizes logit magnitudes
|
|
67
67
|
|
|
68
68
|
## GPU Feasibility
|
|
69
|
-
- **
|
|
70
|
-
- **
|
|
71
|
-
- **
|
|
72
|
-
- **
|
|
73
|
-
- **
|
|
74
|
-
- **
|
|
75
|
-
- **
|
|
76
|
-
- **
|
|
69
|
+
- **D1[v]**: Gate logits = X @ W_gate is standard GEMM (N x d) @ (d x K); expert computation is batched GEMM
|
|
70
|
+
- **D2[v]**: Gate requires 1 GEMM; each expert internally is a standard FFN (2 GEMMs + activation)
|
|
71
|
+
- **D3[v]**: Gate O(N * d * K); per expert O(N * d * d_ff / k); total FLOPs approximately standard FFN x k
|
|
72
|
+
- **D4[v]**: All K expert parameters stored but only k activated; activation memory approximately standard FFN x k
|
|
73
|
+
- **D5[v]**: Gate softmax + top-k is safe in fp16; Router Z-loss requires fp32 logsumexp
|
|
74
|
+
- **D6[~]**: Expert parallelism requires all-to-all communication (each GPU sends/receives N/K tokens); bandwidth-sensitive
|
|
75
|
+
- **D7[v]**: Top-k routing is naturally sparse (activates k/K fraction of parameters); sparsity = 1 - k/K
|
|
76
|
+
- **D8[v]**: Gate -> top-k -> softmax -> weighted-sum can be fused; intra-expert FFN can be fused
|
|
77
77
|
|
|
78
78
|
## Paper-Worthy Formulation
|
|
79
|
-
"We employ noisy top-k gating for sparse mixture-of-experts routing, activating
|
|
79
|
+
"We employ noisy top-k gating for sparse mixture-of-experts routing, activating only k experts per token to reduce activated compute while using the load-balancing auxiliary loss L_aux = K * <f, P> and Router Z-loss to stabilize routing logits. A paper should report measured expert utilization, overflow rate, all-to-all communication share, and quality deltas against dense baselines under matched FLOPs / parameter budgets; do not use unmeasured fixed-percentage placeholders."
|
|
80
80
|
|
|
81
81
|
## Risks
|
|
82
82
|
- Load imbalance: A few experts are over-selected (Matthew effect), while remaining experts receive insufficient training
|
|
@@ -9,9 +9,9 @@
|
|
|
9
9
|
核心诉求:**稀疏激活、负载均衡、端到端可训练**。
|
|
10
10
|
|
|
11
11
|
## 数学思想来源
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
knowledge-base/probability/entropy.md(熵正则化、信息瓶颈)
|
|
12
|
+
- 透镜:../../lenses/variational.md(离散优化松弛、Gumbel-Softmax)、../../lenses/probabilistic.md(信息论路由)
|
|
13
|
+
- 知识:../../knowledge-base/optimization/lagrangian-duality.md(对偶分解、拉格朗日松弛)、
|
|
14
|
+
../../knowledge-base/probability/entropy.md(熵正则化、信息瓶颈)
|
|
15
15
|
|
|
16
16
|
## 需要的数学知识
|
|
17
17
|
- **混合模型 EM**:p(y|x) = Σ_k π_k(x) · p(y|x,θ_k)
|
|
@@ -66,19 +66,17 @@
|
|
|
66
66
|
- **Router Z-loss**:L_z = α·mean(logsumexp(logits)²) 稳定 logits 幅度
|
|
67
67
|
|
|
68
68
|
## GPU 可行性
|
|
69
|
-
-
|
|
70
|
-
- **
|
|
71
|
-
-
|
|
72
|
-
-
|
|
73
|
-
-
|
|
74
|
-
-
|
|
75
|
-
-
|
|
76
|
-
-
|
|
69
|
+
- **D1[v]**:gate logits = X@W_gate 为标准 GEMM (N×d)@(d×K);专家计算为 batched GEMM
|
|
70
|
+
- **D2[v]**:gate 1 次 GEMM;每个专家内部为标准 FFN(2 次 GEMM + activation)
|
|
71
|
+
- **D3[v]**:gate O(N·d·K);每专家 O(N·d·d_ff/k);总 FLOPs ≈ 标准 FFN × k
|
|
72
|
+
- **D4[v]**:K 个专家参数全存储但仅 k 个激活,激活显存 ≈ 标准 FFN × k
|
|
73
|
+
- **D5[v]**:gate 的 softmax + top-k 在 fp16 安全;Router Z-loss 需 fp32 logsumexp
|
|
74
|
+
- **D6[~]**:专家并行需 all-to-all 通信(每个 GPU 收发 N/K 个 token),带宽敏感
|
|
75
|
+
- **D7[v]**:top-k 路由天然稀疏(激活 k/K 的参数),稀疏度 = 1 - k/K
|
|
76
|
+
- **D8[v]**:gate → top-k → softmax → weighted-sum 可融合;专家内 FFN 可 fuse
|
|
77
77
|
|
|
78
78
|
## 论文表述方式
|
|
79
|
-
"采用 noisy top-k
|
|
80
|
-
获得 O(K) 参数容量,配合负载均衡辅助损失 L_aux = K·⟨f,P⟩ 和 Router Z-loss 保证
|
|
81
|
-
专家利用率 >95%,在同等 FLOPs 预算下相比 dense 模型提升 X% 性能。"
|
|
79
|
+
"采用 noisy top-k 门控实现稀疏混合专家路由,每个 token 仅激活 k 个专家以降低激活计算,同时用负载均衡辅助损失 L_aux = K·⟨f,P⟩ 和 Router Z-loss 稳定路由 logits。论文中应报告专家利用率、溢出率、all-to-all 通信占比,以及同等 FLOPs/参数预算下相对 dense baseline 的实测质量差异;不得用未实测的固定百分比占位。"
|
|
82
80
|
|
|
83
81
|
## 风险
|
|
84
82
|
- 负载不均衡:少数专家被过度选择(马太效应),其余专家得不到训练
|
|
@@ -7,9 +7,9 @@ Typical scenarios: (1) Load-balanced MoE routing -- assigning N tokens to K expe
|
|
|
7
7
|
Core requirement: **globally optimal assignment, rather than greedy per-point decisions**.
|
|
8
8
|
|
|
9
9
|
## Mathematical Inspiration
|
|
10
|
-
- Lenses: lenses/variational.md (convex variational, duality theory), lenses/geometric.md (Wasserstein distance)
|
|
11
|
-
- Knowledge: knowledge-base/optimization/lagrangian-duality.md (duality theory, constrained optimization),
|
|
12
|
-
knowledge-base/probability/entropy.md (entropy regularization, marginal constraints)
|
|
10
|
+
- Lenses: ../../lenses/variational.en.md (convex variational, duality theory), ../../lenses/geometric.en.md (Wasserstein distance)
|
|
11
|
+
- Knowledge: ../../knowledge-base/optimization/lagrangian-duality.en.md (duality theory, constrained optimization),
|
|
12
|
+
../../knowledge-base/probability/entropy.en.md (entropy regularization, marginal constraints)
|
|
13
13
|
|
|
14
14
|
## Required Mathematical Background
|
|
15
15
|
- **Discrete Optimal Transport**: min_{P in Pi(mu,nu)} <C, P> = sum_{ij} C_{ij} P_{ij}
|
|
@@ -32,7 +32,7 @@ Sinkhorn routing (entropy-regularized):
|
|
|
32
32
|
for t = 1..T: // T = 5 ~ 20 iterations
|
|
33
33
|
u = a / (K_mat @ v) // row scaling, a = 1/N
|
|
34
34
|
v = b / (K_mat^T @ u) // column scaling, b = cap / sum(cap)
|
|
35
|
-
P = diag(u) @ K_mat @ diag(v) // optimal transport plan
|
|
35
|
+
P = diag(u) @ K_mat @ diag(v) // optimal transport plan (satisfies marginal constraints; doubly stochastic only when N = K with uniform marginals)
|
|
36
36
|
assignment = argmax(P, dim=1) // hard assignment (at inference)
|
|
37
37
|
// At training: weighted_features = P @ E (soft assignment, differentiable)
|
|
38
38
|
|
|
@@ -59,7 +59,7 @@ Capacity constraint (b vector):
|
|
|
59
59
|
- **Operator fusion**: The exp -> matvec -> division pipeline in a single Sinkhorn step can be fused into a CUDA kernel
|
|
60
60
|
|
|
61
61
|
## Paper-Worthy Formulation
|
|
62
|
-
"We formulate token-to-expert routing as an entropy-regularized optimal transport problem, obtaining an
|
|
62
|
+
"We formulate token-to-expert routing as an entropy-regularized optimal transport problem, obtaining an approximate transport plan via the Sinkhorn-Knopp algorithm within T = 10 iterations. Finite Sinkhorn iterations yield an approximate solution to the entropy-regularized problem (not the exact global optimum); approximation quality depends on the iteration count T and regularization parameter eps. Marginal constraints b control the load upper bound per expert."
|
|
63
63
|
|
|
64
64
|
## Risks
|
|
65
65
|
- Excessively small eps causes numerical instability in Sinkhorn (exp overflow); requires log-domain or increasing eps
|
|
@@ -9,9 +9,9 @@
|
|
|
9
9
|
核心诉求:**全局最优分配,而非贪心逐点决策**。
|
|
10
10
|
|
|
11
11
|
## 数学思想来源
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
knowledge-base/probability/entropy.md(熵正则化、边际约束)
|
|
12
|
+
- 透镜:../../lenses/variational.md(凸优化、对偶理论)、../../lenses/geometric.md(Wasserstein 距离)
|
|
13
|
+
- 知识:../../knowledge-base/optimization/lagrangian-duality.md(对偶理论、约束优化)、
|
|
14
|
+
../../knowledge-base/probability/entropy.md(熵正则化、边际约束)
|
|
15
15
|
|
|
16
16
|
## 需要的数学知识
|
|
17
17
|
- **离散最优传输**:min_{P∈Π(μ,ν)} ⟨C, P⟩ = Σ_{ij} C_{ij} P_{ij}
|
|
@@ -34,7 +34,7 @@ Sinkhorn 路由(熵正则化):
|
|
|
34
34
|
for t = 1..T: // T=5~20 次迭代
|
|
35
35
|
u = a / (K_mat @ v) // 行缩放,a = 1/N
|
|
36
36
|
v = b / (K_mat^T @ u) // 列缩放,b = cap/sum(cap)
|
|
37
|
-
P = diag(u) @ K_mat @ diag(v) //
|
|
37
|
+
P = diag(u) @ K_mat @ diag(v) // 最优传输计划(满足边际约束;仅当 N=K 且边际均匀时才是双随机矩阵)
|
|
38
38
|
assignment = argmax(P, dim=1) // 硬分配(推理时)
|
|
39
39
|
// 训练时:weighted_features = P @ E (软分配,可微)
|
|
40
40
|
|
|
@@ -62,8 +62,8 @@ Sinkhorn 路由(熵正则化):
|
|
|
62
62
|
|
|
63
63
|
## 论文表述方式
|
|
64
64
|
"将 token-to-expert 路由建模为熵正则化最优传输问题,通过 Sinkhorn-Knopp 算法在 T=10 次
|
|
65
|
-
|
|
66
|
-
|
|
65
|
+
迭代内求得近似最优的传输计划矩阵。有限次迭代给出熵正则化问题的近似解(非精确全局最优),
|
|
66
|
+
近似质量取决于迭代次数 T 和正则化参数 ε;同时通过边际约束 b 控制各专家的负载上限。"
|
|
67
67
|
|
|
68
68
|
## 风险
|
|
69
69
|
- ε 过小导致 Sinkhorn 数值不稳定(exp 溢出),需用 log-domain 或增大 ε
|
|
@@ -10,9 +10,9 @@ Use when routing needs to be based on the intrinsic similarity structure of toke
|
|
|
10
10
|
Core requirement: **discover the intrinsic cluster structure of data for routing or expert initialization**.
|
|
11
11
|
|
|
12
12
|
## Mathematical Inspiration
|
|
13
|
-
- Lenses: lenses/geometric.md (spectral graph theory, Laplacian eigenmaps), lenses/variational.md (relaxation and approximation)
|
|
14
|
-
- Knowledge: knowledge-base/matrix-analysis/projection.md (eigendecomposition, Rayleigh quotient),
|
|
15
|
-
knowledge-base/differential-geometry/manifold.md (manifold learning, graph cuts)
|
|
13
|
+
- Lenses: ../../lenses/geometric.en.md (spectral graph theory, Laplacian eigenmaps), ../../lenses/variational.en.md (relaxation and approximation)
|
|
14
|
+
- Knowledge: ../../knowledge-base/matrix-analysis/projection.en.md (eigendecomposition, Rayleigh quotient),
|
|
15
|
+
../../knowledge-base/differential-geometry/manifold.en.md (manifold learning, graph cuts)
|
|
16
16
|
|
|
17
17
|
## Required Mathematical Background
|
|
18
18
|
- **Spectral Clustering (Ng-Jordan-Weiss)**:
|
|
@@ -24,8 +24,12 @@ Core requirement: **discover the intrinsic cluster structure of data for routing
|
|
|
24
24
|
W approximately C * W_m^{-1} * C^T, reducing eigendecomposition to m x m
|
|
25
25
|
- **Spectral Relaxation Continuation**: Discrete cluster assignment => continuous eigenvectors => differentiable routing
|
|
26
26
|
Use softmax(U_k * W_proj) instead of hard k-means assignment
|
|
27
|
-
- **Power Iteration Acceleration**: Full eigendecomposition is unnecessary; only the
|
|
28
|
-
|
|
27
|
+
- **Power Iteration Acceleration**: Full eigendecomposition is unnecessary; only the $k$ **smallest** eigenvectors of Laplacian $L$ are needed
|
|
28
|
+
⚠ Power iteration naturally finds the **largest** eigenvectors, so it cannot be applied directly to $L$!
|
|
29
|
+
Correct approaches: (1) apply power iteration to the similarity matrix $W$ (or normalized $D^{-1/2}WD^{-1/2}$)
|
|
30
|
+
to find its largest eigenvectors (corresponding to $L$'s smallest); (2) use Lanczos on $L$ with which='SM';
|
|
31
|
+
(3) shifted inverse iteration $(L - \sigma I)^{-1}$
|
|
32
|
+
Lanczos/Arnoldi iteration O(N^2 * k * iter) or randomized SVD O(N^2 * k)
|
|
29
33
|
|
|
30
34
|
## AI Module Form
|
|
31
35
|
```
|
|
@@ -34,13 +38,16 @@ Input: X in R^{N x d}, number of clusters K
|
|
|
34
38
|
|
|
35
39
|
Method 1 - Online Spectral Clustering Routing (periodic updates during training):
|
|
36
40
|
// Update cluster centers every M steps; use nearest neighbor at inference
|
|
37
|
-
W = exp(-cdist(X_sample, X_sample) / (2 sigma^2)) // m x m
|
|
41
|
+
W = exp(-(cdist(X_sample, X_sample)**2) / (2 sigma^2)) // m x m RBF similarity
|
|
38
42
|
L = I - D^{-1/2} W D^{-1/2} // normalized Laplacian
|
|
39
43
|
U_k = eigsh(L, k=K, which='SM') // K smallest eigenvectors
|
|
40
44
|
centers = kmeans(U_k, K) // K cluster centers
|
|
41
|
-
// Routing:
|
|
42
|
-
|
|
43
|
-
|
|
45
|
+
// Routing: embed new tokens into spectral space and assign
|
|
46
|
+
// ⚠ X @ W_proj is just a learnable linear projection, NOT Nystrom extension!
|
|
47
|
+
// True Nystrom extension: v_new = (1/λ) * W(x_new, X_sample) @ v, where v is eigenvector, λ is eigenvalue
|
|
48
|
+
proj_nystrom = (1/λ_k) * W_new_sample @ U_k // Nystrom extension: similarity from x_new to m samples × eigenvectors
|
|
49
|
+
proj = X @ W_proj // alternative: learnable linear projection (not Nystrom, but end-to-end trainable)
|
|
50
|
+
assignment = argmin(cdist(proj_nystrom, centers)) // nearest center assignment
|
|
44
51
|
|
|
45
52
|
Method 2 - Differentiable Spectral Routing (end-to-end):
|
|
46
53
|
// Use softmax relaxation instead of hard assignment
|
|
@@ -48,18 +55,22 @@ Method 2 - Differentiable Spectral Routing (end-to-end):
|
|
|
48
55
|
A = exp(sim_matrix / tau) // similarity graph (learnable tau)
|
|
49
56
|
D_inv_sqrt = diag(1 / sqrt(sum(A, dim=1) + eps))
|
|
50
57
|
L_norm = I - D_inv_sqrt @ A @ D_inv_sqrt // normalized Laplacian
|
|
51
|
-
//
|
|
52
|
-
|
|
58
|
+
// ⚠ Power iteration finds largest eigenvectors, but spectral clustering needs L_norm's smallest!
|
|
59
|
+
// Apply power iteration to the normalized similarity matrix: its largest eigenvectors = L_norm's smallest
|
|
60
|
+
W_norm = D_inv_sqrt @ A @ D_inv_sqrt // = I - L_norm
|
|
61
|
+
U_k = power_iteration_approx(W_norm, K, steps=5) // N x K (W_norm's largest eigvecs = L_norm's smallest)
|
|
53
62
|
// Soft assignment
|
|
54
63
|
cluster_logits = U_k @ W_cluster // N x K (learnable projection)
|
|
55
64
|
route_probs = softmax(cluster_logits / tau_route) // soft routing probabilities
|
|
56
65
|
|
|
57
66
|
Method 3 - Anchor Spectral Clustering (large-scale):
|
|
58
67
|
anchors = kmeans_pp(X, m) // m anchor points, m << N
|
|
59
|
-
Z = exp(-cdist(X, anchors) / (2 sigma^2))
|
|
68
|
+
Z = exp(-(cdist(X, anchors)**2) / (2 sigma^2)) // N x m RBF affinity matrix
|
|
60
69
|
L_anchor = I - D_z^{-1/2} Z^T Z D_z^{-1/2} // m x m Laplacian
|
|
61
70
|
U_k = eigsh(L_anchor, K) // m x K eigenvectors
|
|
62
|
-
|
|
71
|
+
// Nystrom extension: Z @ U_k extends anchor eigenvectors to all N points (non-learnable part)
|
|
72
|
+
embedding_nystrom = Z @ U_k // N x K Nystrom extension embedding
|
|
73
|
+
route = embedding_nystrom @ W_proj // N x K routing scores after learnable projection
|
|
63
74
|
```
|
|
64
75
|
|
|
65
76
|
## Implementable Structures
|
|
@@ -75,11 +86,11 @@ Method 3 - Anchor Spectral Clustering (large-scale):
|
|
|
75
86
|
- **Memory & KV-Cache**: N x N similarity matrix exceeds 64 MB when N > 4096; sampling-based dimensionality reduction is essential
|
|
76
87
|
- **Low-precision stability**: Eigendecomposition recommended in fp32; exp(-dist/sigma^2) in fp16 requires distance clipping
|
|
77
88
|
- **Parallelism & Communication**: Power iteration matvec is highly parallel; k-means assign + update steps can be batch-parallelized
|
|
78
|
-
- **Sparse structure**: k-NN graph replaces fully connected graph; W sparsity
|
|
89
|
+
- **Sparse structure**: k-NN graph replaces the fully connected graph; W sparsity is about $1-k_{\text{nn}}/N$, enabling SpMM acceleration
|
|
79
90
|
- **Operator fusion**: Diagonal scaling in D^{-1/2} @ A @ D^{-1/2} can be fused; cdist + exp + normalize can be fused
|
|
80
91
|
|
|
81
92
|
## Paper-Worthy Formulation
|
|
82
|
-
"We
|
|
93
|
+
"We implement routing through a continuous relaxation of spectral clustering: construct the normalized Laplacian of the token similarity graph, use Nystrom / anchor approximations plus power iteration to avoid a full O(N^3) eigendecomposition, and express the main work as GEMM, matvecs, and k-means. Normalized Cut can be reported as a clustering-quality metric, but approximation ratios depend on graph-model, sampling, and solver assumptions and should not be claimed unconditionally."
|
|
83
94
|
|
|
84
95
|
## Risks
|
|
85
96
|
- The memory and computation cost of the N x N similarity matrix does not scale for long sequences; sampling or k-NN sparsification is mandatory
|
|
@@ -10,9 +10,9 @@
|
|
|
10
10
|
核心诉求:**发现数据内在的簇结构,用于路由或专家初始化**。
|
|
11
11
|
|
|
12
12
|
## 数学思想来源
|
|
13
|
-
-
|
|
14
|
-
-
|
|
15
|
-
knowledge-base/differential-geometry/manifold.md(流形学习、图割)
|
|
13
|
+
- 透镜:../../lenses/geometric.md(谱图论、拉普拉斯特征映射)、../../lenses/variational.md(松弛与近似)
|
|
14
|
+
- 知识:../../knowledge-base/matrix-analysis/projection.md(特征值分解、Rayleigh 商)、
|
|
15
|
+
../../knowledge-base/differential-geometry/manifold.md(流形学习、图割)
|
|
16
16
|
|
|
17
17
|
## 需要的数学知识
|
|
18
18
|
- **谱聚类 (Ng-Jordan-Weiss)**:
|
|
@@ -24,8 +24,11 @@
|
|
|
24
24
|
W ≈ C · W_m^{-1} · C^T,将特征分解降维到 m×m
|
|
25
25
|
- **谱松弛连续化**:离散聚类分配 → 连续特征向量 → 可微路由
|
|
26
26
|
用 softmax(U_k · W_proj) 替代硬 k-means 分配
|
|
27
|
-
-
|
|
28
|
-
|
|
27
|
+
- **幂迭代加速**:不需完整特征分解,只需拉普拉斯 $L$ 的前 $k$ 个**最小**特征向量
|
|
28
|
+
⚠ 幂迭代天然找**最大**特征向量,因此不能直接对 $L$ 做幂迭代!
|
|
29
|
+
正确做法:(1) 对相似度矩阵 $W$(或归一化 $D^{-1/2}WD^{-1/2}$)做幂迭代找最大特征向量
|
|
30
|
+
(对应 $L$ 的最小特征向量);(2) 对 $L$ 用 Lanczos + which='SM';(3) 移位逆迭代 $(L - \sigma I)^{-1}$
|
|
31
|
+
Lanczos/Arnoldi 迭代 O(N²·k·iter) 或 randomized SVD O(N²·k)
|
|
29
32
|
|
|
30
33
|
## AI 模块形式
|
|
31
34
|
```
|
|
@@ -34,13 +37,16 @@
|
|
|
34
37
|
|
|
35
38
|
方法1 - 在线谱聚类路由(训练时周期性更新):
|
|
36
39
|
// 每 M 步更新一次聚类中心,推理时用最近邻
|
|
37
|
-
W = exp(-cdist(X_sample, X_sample) / (2σ²)) // m×m
|
|
40
|
+
W = exp(-(cdist(X_sample, X_sample)**2) / (2σ²)) // m×m RBF 相似度
|
|
38
41
|
L = I - D^{-1/2} W D^{-1/2} // 归一化拉普拉斯
|
|
39
42
|
U_k = eigsh(L, k=K, which='SM') // 前 K 个最小特征向量
|
|
40
43
|
centers = kmeans(U_k, K) // K 个聚类中心
|
|
41
|
-
// 路由:将新 token
|
|
42
|
-
|
|
43
|
-
|
|
44
|
+
// 路由:将新 token 嵌入到谱空间后分配
|
|
45
|
+
// ⚠ X @ W_proj 仅为可学习线性投影,不是 Nyström 扩展!
|
|
46
|
+
// 真正的 Nyström 扩展:v_new = (1/λ) * W(x_new, X_sample) @ v,其中 v 为特征向量,λ 为特征值
|
|
47
|
+
proj_nystrom = (1/λ_k) * W_new_sample @ U_k // Nyström 扩展:x_new 到 m 个采样点的相似度 × 特征向量
|
|
48
|
+
proj = X @ W_proj // 替代方案:可学习线性投影(非 Nyström,但可端到端训练)
|
|
49
|
+
assignment = argmin(cdist(proj_nystrom, centers)) // 最近中心分配
|
|
44
50
|
|
|
45
51
|
方法2 - 可微谱路由(端到端):
|
|
46
52
|
// 用 softmax 松弛替代硬分配
|
|
@@ -48,18 +54,22 @@
|
|
|
48
54
|
A = exp(sim_matrix / τ) // 相似度图(可学习 τ)
|
|
49
55
|
D_inv_sqrt = diag(1 / sqrt(sum(A, dim=1) + ε))
|
|
50
56
|
L_norm = I - D_inv_sqrt @ A @ D_inv_sqrt // 归一化拉普拉斯
|
|
51
|
-
//
|
|
52
|
-
|
|
57
|
+
// ⚠ 幂迭代找最大特征向量,但谱聚类需要 L_norm 的最小特征向量!
|
|
58
|
+
// 对归一化相似度矩阵做幂迭代:其最大特征向量 = L_norm 的最小特征向量
|
|
59
|
+
W_norm = D_inv_sqrt @ A @ D_inv_sqrt // = I - L_norm
|
|
60
|
+
U_k = power_iteration_approx(W_norm, K, steps=5) // N×K(W_norm 的最大特征向量 = L_norm 的最小)
|
|
53
61
|
// 软分配
|
|
54
62
|
cluster_logits = U_k @ W_cluster // N×K(可学习投影)
|
|
55
63
|
route_probs = softmax(cluster_logits / τ_route) // 软路由概率
|
|
56
64
|
|
|
57
65
|
方法3 - 锚点谱聚类(大规模):
|
|
58
66
|
anchors = kmeans_pp(X, m) // m 个锚点,m << N
|
|
59
|
-
Z = exp(-cdist(X, anchors) / (2σ²))
|
|
67
|
+
Z = exp(-(cdist(X, anchors)**2) / (2σ²)) // N×m RBF 亲和矩阵
|
|
60
68
|
L_anchor = I - D_z^{-1/2} Z^T Z D_z^{-1/2} // m×m 拉普拉斯
|
|
61
69
|
U_k = eigsh(L_anchor, K) // m×K 特征向量
|
|
62
|
-
|
|
70
|
+
// Nyström 扩展:Z @ U_k 将锚点特征向量扩展到全部 N 个点(非可学习部分)
|
|
71
|
+
embedding_nystrom = Z @ U_k // N×K Nyström 扩展嵌入
|
|
72
|
+
route = embedding_nystrom @ W_proj // N×K 可学习投影后的路由分数
|
|
63
73
|
```
|
|
64
74
|
|
|
65
75
|
## 可实现结构
|
|
@@ -75,13 +85,12 @@
|
|
|
75
85
|
- **显存与 KV-Cache**:N×N 相似度矩阵在 N>4096 时 >64MB,必须采样降维
|
|
76
86
|
- **低精度稳定**:特征分解建议 fp32;exp(-dist/σ²) 在 fp16 下需 clip distance
|
|
77
87
|
- **并行与通信**:幂迭代的 matvec 高度并行;k-means 的 assign+update 可批并行
|
|
78
|
-
- **稀疏结构**:k-NN 图替代全连接图,W
|
|
88
|
+
- **稀疏结构**:k-NN 图替代全连接图,W 稀疏度约为 $1-k_{\text{nn}}/N$,可用 SpMM 加速
|
|
79
89
|
- **算子融合**:D^{-1/2}@A@D^{-1/2} 的对角缩放可融合;cdist+exp+normalize 可融合
|
|
80
90
|
|
|
81
91
|
## 论文表述方式
|
|
82
|
-
"
|
|
83
|
-
通过 Nyström
|
|
84
|
-
在线谱聚类,聚类质量以 Normalized Cut 衡量保证 O(√(log N/K)) 的近似比。"
|
|
92
|
+
"利用谱聚类的连续松弛实现路由:构造 token 相似度图的归一化拉普拉斯,
|
|
93
|
+
通过 Nyström / 锚点近似和幂迭代避免完整 O(N³) 特征分解,并将主要计算转化为 GEMM、matvec 与 k-means。Normalized Cut 可作为聚类质量指标,但近似比依赖图模型、采样策略和求解器假设,不能无条件宣称。"
|
|
85
94
|
|
|
86
95
|
## 风险
|
|
87
96
|
- N×N 相似度矩阵的显存和计算在长序列下不可扩展,必须采样或 k-NN 稀疏化
|
|
@@ -45,7 +45,22 @@ GPU friendliness: the core challenge of connections is "serial ODE integration."
|
|
|
45
45
|
|
|
46
46
|
## Further References
|
|
47
47
|
|
|
48
|
-
- Distillation notes: references/books/differential-geometry.md (Ch 12 Connections and Covariant Derivatives, Section 12.2 Connection Forms, Section 12.4 Ehresmann, Section 12.12 G-Connections)
|
|
49
|
-
- Distillation notes: references/books/differential-geometry.md (Section 6.8 Principal Bundles, Section 9.8 Electromagnetism)
|
|
48
|
+
- Distillation notes: ../../references/books/differential-geometry.en.md (Ch 12 Connections and Covariant Derivatives, Section 12.2 Connection Forms, Section 12.4 Ehresmann, Section 12.12 G-Connections)
|
|
49
|
+
- Distillation notes: ../../references/books/differential-geometry.en.md (Section 6.8 Principal Bundles, Section 9.8 Electromagnetism)
|
|
50
50
|
- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 12 (Section 12.1--Section 12.12, complete connection theory)
|
|
51
51
|
- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Section 13.1 Levi-Civita Connection
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## Routing Extensions
|
|
55
|
+
- If curvature definition is needed -> `curvature.en.md` (curvature tensor derived from connection)
|
|
56
|
+
- If parallel transport and geodesics are needed -> `geodesic.en.md` (geodesics are auto-parallel curves of parallel transport)
|
|
57
|
+
- If covariant derivative computation is needed -> `tangent-space.en.md` (covariant differentiation on tangent space)
|
|
58
|
+
|
|
59
|
+
## Extensible Directions
|
|
60
|
+
- Levi-Civita connection: unique torsion-free metric connection on Riemannian manifolds
|
|
61
|
+
- Christoffel symbols: connection components in coordinate basis
|
|
62
|
+
- Holonomy: effect of parallel transport around closed curves
|
|
63
|
+
- Torsion: antisymmetric part of a connection
|
|
64
|
+
- Affine connection: general affine connection theory
|
|
65
|
+
- Ehresmann connection: horizontal distributions on fiber bundles
|
|
66
|
+
- Gauge connection: gauge fields as connections in physics
|
|
@@ -45,7 +45,22 @@ GPU 友好度:联络的核心挑战是"串行 ODE 积分"。
|
|
|
45
45
|
|
|
46
46
|
## 深入参考
|
|
47
47
|
|
|
48
|
-
-
|
|
49
|
-
-
|
|
48
|
+
- 蒸馏稿:../../references/books/differential-geometry.md(Ch 12 Connections and Covariant Derivatives, §12.2 联络形式, §12.4 Ehresmann, §12.12 G-联络)
|
|
49
|
+
- 蒸馏稿:../../references/books/differential-geometry.md(§6.8 Principal Bundles, §9.8 Electromagnetism)
|
|
50
50
|
- 原书:Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 12(§12.1-§12.12 完整联络理论)
|
|
51
51
|
- 原书:Jeffrey M. Lee, *Manifolds and Differential Geometry*, §13.1 Levi-Civita Connection
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## 路由扩展
|
|
55
|
+
- 若需要曲率的定义 → `curvature.md`(曲率张量由联络导出)
|
|
56
|
+
- 若需要平行移动与测地线 → `geodesic.md`(测地线是平行移动的自平行曲线)
|
|
57
|
+
- 若需要协变导数的具体计算 → `tangent-space.md`(切空间上的协变微分)
|
|
58
|
+
|
|
59
|
+
## 可扩展方向
|
|
60
|
+
- Levi-Civita 联络:黎曼流形的唯一无挠度量联络
|
|
61
|
+
- Christoffel 符号:联络在坐标基下的分量
|
|
62
|
+
- 和乐(holonomy):平行移动绕闭曲线的效果
|
|
63
|
+
- 挠率(torsion):联络的非对称部分
|
|
64
|
+
- 仿射联络(affine connection):一般的仿射联络理论
|
|
65
|
+
- Ehresmann 联络:纤维丛上的水平分布
|
|
66
|
+
- 规范联络(gauge connection):物理中的规范场作为联络
|
|
@@ -45,5 +45,18 @@ GPU friendliness: the core difficulty of curvature is that "the full tensor cann
|
|
|
45
45
|
|
|
46
46
|
## Further References
|
|
47
47
|
|
|
48
|
-
- Distillation notes: references/books/differential-geometry.md (Ch 12 Section 12.5/Section 12.10 Curvature, Ch 13 Section 13.2 Riemann Curvature, Section 13.7 Jacobi Fields, Section 13.11 Rauch Comparison)
|
|
48
|
+
- Distillation notes: ../../references/books/differential-geometry.en.md (Ch 12 Section 12.5/Section 12.10 Curvature, Ch 13 Section 13.2 Riemann Curvature, Section 13.7 Jacobi Fields, Section 13.11 Rauch Comparison)
|
|
49
49
|
- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Section 13.2 Riemann Curvature Tensor, Section 13.7 Jacobi Fields
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
## Routing Extensions
|
|
53
|
+
- If local geometric analysis is needed -> `metric-tensor.en.md` (metric determines local curvature)
|
|
54
|
+
- If global topological analysis is needed -> `../topology/persistent-homology.md` (persistent homology captures global topology)
|
|
55
|
+
- If curvature's effect on stability is involved -> `../matrix-analysis/matrix-perturbation.md` (curvature-induced perturbation amplification)
|
|
56
|
+
|
|
57
|
+
## Extensible Directions
|
|
58
|
+
- Sectional / Ricci / scalar curvature: curvature concepts at different dimensions
|
|
59
|
+
- Gauss-Bonnet theorem: relationship between curvature and topological invariants
|
|
60
|
+
- Comparison theorems (Toponogov, Bishop-Gromov): geometric comparison under curvature constraints
|
|
61
|
+
- Curvature flow: mean curvature flow and Ricci flow
|
|
62
|
+
- Cartan-Hadamard theorem: global structure of non-positively curved manifolds
|
|
@@ -45,5 +45,18 @@ GPU 友好度:曲率的核心难点是"不能物化全张量"。
|
|
|
45
45
|
|
|
46
46
|
## 深入参考
|
|
47
47
|
|
|
48
|
-
-
|
|
48
|
+
- 蒸馏稿:../../references/books/differential-geometry.md(Ch 12 §12.5/§12.10 Curvature, Ch 13 §13.2 Riemann Curvature, §13.7 Jacobi Fields, §13.11 Rauch Comparison)
|
|
49
49
|
- 原书:Jeffrey M. Lee, *Manifolds and Differential Geometry*, §13.2 Riemann Curvature Tensor, §13.7 Jacobi Fields
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
## 路由扩展
|
|
53
|
+
- 若需要局部几何分析 → `metric-tensor.md`(度量决定局部曲率)
|
|
54
|
+
- 若需要全局拓扑分析 → `../topology/persistent-homology.md`(持续同调捕捉全局拓扑)
|
|
55
|
+
- 若涉及曲率对稳定性的影响 → `../matrix-analysis/matrix-perturbation.md`(曲率导致的扰动放大)
|
|
56
|
+
|
|
57
|
+
## 可扩展方向
|
|
58
|
+
- 截面/Ricci/标量曲率(sectional / Ricci / scalar curvature):不同维度的曲率概念
|
|
59
|
+
- Gauss-Bonnet 定理:曲率与拓扑不变量的关系
|
|
60
|
+
- 比较定理(Toponogov, Bishop-Gromov):曲率约束下的几何比较
|
|
61
|
+
- 曲率流(curvature flow):平均曲率流与 Ricci 流
|
|
62
|
+
- Cartan-Hadamard 定理:非正曲率流形的全局结构
|