math-skill 3.0.1 → 3.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en-US.md +72 -47
- package/README.md +40 -15
- package/agents/math-critic.en.md +6 -6
- package/agents/math-critic.md +6 -6
- package/commands/ask.en.md +6 -1
- package/commands/ask.md +6 -1
- package/design-patterns/attention/equivariant-attention.en.md +8 -2
- package/design-patterns/attention/equivariant-attention.md +7 -1
- package/design-patterns/attention/geometry-aware-attention.en.md +2 -2
- package/design-patterns/attention/geometry-aware-attention.md +1 -1
- package/design-patterns/attention/information-bottleneck-attention.en.md +32 -12
- package/design-patterns/attention/information-bottleneck-attention.md +32 -12
- package/design-patterns/attention/projection-attention.en.md +3 -3
- package/design-patterns/attention/projection-attention.md +2 -2
- package/design-patterns/attention/spectral-attention.en.md +19 -5
- package/design-patterns/attention/spectral-attention.md +18 -4
- package/design-patterns/compression/leverage-score-selection.en.md +13 -13
- package/design-patterns/compression/leverage-score-selection.md +13 -13
- package/design-patterns/compression/low-rank-kv-cache.en.md +39 -12
- package/design-patterns/compression/low-rank-kv-cache.md +38 -12
- package/design-patterns/compression/spectral-token-pruning.en.md +16 -14
- package/design-patterns/compression/spectral-token-pruning.md +16 -14
- package/design-patterns/compression/topology-preserving-compression.en.md +10 -10
- package/design-patterns/compression/topology-preserving-compression.md +10 -10
- package/design-patterns/loss/constraint-penalty.en.md +8 -4
- package/design-patterns/loss/constraint-penalty.md +9 -6
- package/design-patterns/loss/contrastive-loss.en.md +37 -11
- package/design-patterns/loss/contrastive-loss.md +38 -12
- package/design-patterns/loss/information-bottleneck-loss.en.md +5 -5
- package/design-patterns/loss/information-bottleneck-loss.md +6 -6
- package/design-patterns/loss/orthogonality-loss.en.md +24 -17
- package/design-patterns/loss/orthogonality-loss.md +24 -18
- package/design-patterns/loss/variational-loss.en.md +4 -4
- package/design-patterns/loss/variational-loss.md +5 -5
- package/design-patterns/overview.en.md +36 -0
- package/design-patterns/overview.md +36 -0
- package/design-patterns/representation/equivariant-split.en.md +4 -4
- package/design-patterns/representation/equivariant-split.md +4 -5
- package/design-patterns/representation/manifold-representation.en.md +12 -12
- package/design-patterns/representation/manifold-representation.md +13 -14
- package/design-patterns/representation/shared-private-decomposition.en.md +12 -7
- package/design-patterns/representation/shared-private-decomposition.md +12 -9
- package/design-patterns/representation/subspace-alignment.en.md +12 -12
- package/design-patterns/representation/subspace-alignment.md +12 -12
- package/design-patterns/routing/graph-routing.en.md +11 -11
- package/design-patterns/routing/graph-routing.md +11 -11
- package/design-patterns/routing/moe-routing.en.md +12 -12
- package/design-patterns/routing/moe-routing.md +12 -14
- package/design-patterns/routing/optimal-transport-routing.en.md +5 -5
- package/design-patterns/routing/optimal-transport-routing.md +6 -6
- package/design-patterns/routing/spectral-clustering-routing.en.md +26 -15
- package/design-patterns/routing/spectral-clustering-routing.md +26 -17
- package/knowledge-base/differential-geometry/connection.en.md +17 -2
- package/knowledge-base/differential-geometry/connection.md +17 -2
- package/knowledge-base/differential-geometry/curvature.en.md +14 -1
- package/knowledge-base/differential-geometry/curvature.md +14 -1
- package/knowledge-base/differential-geometry/geodesic.en.md +16 -2
- package/knowledge-base/differential-geometry/geodesic.md +16 -2
- package/knowledge-base/differential-geometry/index.en.md +52 -0
- package/knowledge-base/differential-geometry/index.md +52 -0
- package/knowledge-base/differential-geometry/manifold.en.md +15 -2
- package/knowledge-base/differential-geometry/manifold.md +15 -2
- package/knowledge-base/differential-geometry/metric-tensor.en.md +16 -2
- package/knowledge-base/differential-geometry/metric-tensor.md +16 -2
- package/knowledge-base/differential-geometry/tangent-space.en.md +17 -3
- package/knowledge-base/differential-geometry/tangent-space.md +17 -3
- package/knowledge-base/information-geometry/fisher-metric.en.md +24 -5
- package/knowledge-base/information-geometry/fisher-metric.md +24 -5
- package/knowledge-base/information-geometry/index.en.md +45 -0
- package/knowledge-base/information-geometry/index.md +45 -0
- package/knowledge-base/information-geometry/natural-gradient.en.md +17 -3
- package/knowledge-base/information-geometry/natural-gradient.md +17 -3
- package/knowledge-base/lie-theory/equivariance.en.md +18 -3
- package/knowledge-base/lie-theory/equivariance.md +18 -3
- package/knowledge-base/lie-theory/group-action.en.md +16 -2
- package/knowledge-base/lie-theory/group-action.md +16 -2
- package/knowledge-base/lie-theory/index.en.md +52 -0
- package/knowledge-base/lie-theory/index.md +52 -0
- package/knowledge-base/lie-theory/lie-algebra.en.md +17 -2
- package/knowledge-base/lie-theory/lie-algebra.md +17 -2
- package/knowledge-base/lie-theory/lie-group.en.md +17 -3
- package/knowledge-base/lie-theory/lie-group.md +17 -3
- package/knowledge-base/lie-theory/representation.en.md +17 -2
- package/knowledge-base/lie-theory/representation.md +17 -2
- package/knowledge-base/matrix-analysis/index.en.md +49 -0
- package/knowledge-base/matrix-analysis/index.md +49 -0
- package/knowledge-base/matrix-analysis/low-rank-approximation.en.md +20 -7
- package/knowledge-base/matrix-analysis/low-rank-approximation.md +20 -7
- package/knowledge-base/matrix-analysis/matrix-perturbation.en.md +13 -1
- package/knowledge-base/matrix-analysis/matrix-perturbation.md +13 -1
- package/knowledge-base/matrix-analysis/positive-semidefinite.en.md +20 -6
- package/knowledge-base/matrix-analysis/positive-semidefinite.md +20 -6
- package/knowledge-base/matrix-analysis/projection.en.md +14 -1
- package/knowledge-base/matrix-analysis/projection.md +14 -1
- package/knowledge-base/matrix-analysis/spectral-decomposition.en.md +14 -1
- package/knowledge-base/matrix-analysis/spectral-decomposition.md +14 -1
- package/knowledge-base/optimization/constrained-optimization.en.md +17 -3
- package/knowledge-base/optimization/constrained-optimization.md +17 -3
- package/knowledge-base/optimization/convex-optimization.en.md +15 -2
- package/knowledge-base/optimization/convex-optimization.md +15 -2
- package/knowledge-base/optimization/index.en.md +50 -0
- package/knowledge-base/optimization/index.md +50 -0
- package/knowledge-base/optimization/lagrangian-duality.en.md +18 -4
- package/knowledge-base/optimization/lagrangian-duality.md +18 -4
- package/knowledge-base/optimization/proximal-method.en.md +15 -1
- package/knowledge-base/optimization/proximal-method.md +15 -1
- package/knowledge-base/optimization/riemannian-optimization.en.md +18 -4
- package/knowledge-base/optimization/riemannian-optimization.md +18 -4
- package/knowledge-base/overview.en.md +20 -8
- package/knowledge-base/overview.md +20 -8
- package/knowledge-base/probability/concentration-inequality.en.md +16 -2
- package/knowledge-base/probability/concentration-inequality.md +16 -2
- package/knowledge-base/probability/entropy.en.md +19 -3
- package/knowledge-base/probability/entropy.md +18 -2
- package/knowledge-base/probability/fisher-information.en.md +17 -3
- package/knowledge-base/probability/fisher-information.md +17 -3
- package/knowledge-base/probability/index.en.md +51 -0
- package/knowledge-base/probability/index.md +51 -0
- package/knowledge-base/probability/information-bottleneck.en.md +18 -2
- package/knowledge-base/probability/information-bottleneck.md +18 -2
- package/knowledge-base/probability/kl-divergence.en.md +15 -2
- package/knowledge-base/probability/kl-divergence.md +15 -2
- package/knowledge-base/topology/euler-characteristic.en.md +15 -2
- package/knowledge-base/topology/euler-characteristic.md +15 -2
- package/knowledge-base/topology/fundamental-group.en.md +16 -2
- package/knowledge-base/topology/fundamental-group.md +16 -2
- package/knowledge-base/topology/index.en.md +48 -0
- package/knowledge-base/topology/index.md +48 -0
- package/knowledge-base/topology/persistent-homology.en.md +17 -2
- package/knowledge-base/topology/persistent-homology.md +17 -2
- package/lenses/spectral.en.md +1 -1
- package/lenses/spectral.md +1 -1
- package/lenses/topological.en.md +1 -1
- package/lenses/topological.md +1 -1
- package/package.json +6 -6
- package/references/agentic-workflow.en.md +2 -2
- package/references/books/abstract-algebra.en.md +12 -12
- package/references/books/abstract-algebra.md +9 -9
- package/references/books/algebraic-geometry-rising-sea.en.md +2 -2
- package/references/books/differential-geometry.en.md +16 -16
- package/references/books/differential-geometry.md +14 -14
- package/references/books/matrix-analysis.en.md +3 -3
- package/references/books/matrix-analysis.md +24 -24
- package/references/books/micro-lie-theory.en.md +1 -1
- package/references/books/optimization-ml.en.md +15 -15
- package/references/books/optimization-ml.md +14 -14
- package/references/books/smooth-manifolds.en.md +3 -3
- package/references/books/smooth-manifolds.md +6 -6
- package/references/gpu-friendly-math.en.md +21 -8
- package/references/gpu-friendly-math.md +19 -4
- package/references/skill-index.en.md +106 -0
- package/references/skill-index.md +106 -0
- package/skills/math-research-activator/SKILL.en.md +76 -118
- package/skills/math-research-activator/SKILL.md +69 -109
|
@@ -48,7 +48,22 @@ GPU 友好度取决于群的离散化程度:
|
|
|
48
48
|
|
|
49
49
|
## 深入参考
|
|
50
50
|
|
|
51
|
-
-
|
|
52
|
-
-
|
|
53
|
-
-
|
|
51
|
+
- 蒸馏稿:../../references/books/micro-lie-theory.md(§II-F 伴随 Ad_X,等变性的代数实现)
|
|
52
|
+
- 蒸馏稿:../../references/books/differential-geometry.md(§6.8 Principal Bundles, §12.12 G-Connections, 规范等变)
|
|
53
|
+
- 蒸馏稿:../../references/books/differential-geometry.md(Ch 5 Lie Groups, 连续对称作为先验)
|
|
54
54
|
- 原书:Jeffrey M. Lee, *Manifolds and Differential Geometry*, §6.8 + §12.12(规范等变的几何基础)
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
## 路由扩展
|
|
58
|
+
- 若需要表示理论基础 → `representation.md`(等变映射是表示间态射)
|
|
59
|
+
- 若需要群作用结构 → `group-action.md`(等变性定义依赖群作用)
|
|
60
|
+
- 若用于注意力机制设计 → `equivariant-attention`(设计模式层的等变注意力)
|
|
61
|
+
|
|
62
|
+
## 可扩展方向
|
|
63
|
+
- 可操纵特征(steerable features):SO(3) 下的可操纵特征表示
|
|
64
|
+
- 球谐函数(spherical harmonics):SO(3) 不可约表示的基函数
|
|
65
|
+
- Wigner D-矩阵:SO(3) 表示的矩阵元
|
|
66
|
+
- 等变映射代数(equivariant map algebra):等变线性映射的完全刻画
|
|
67
|
+
- 通用等变架构(universal equivariant architectures):等变函数的通用逼近
|
|
68
|
+
- 对称破缺(symmetry breaking):近似等变或可控对称破缺
|
|
69
|
+
- 近似等变(approximate equivariance):噪声或离散化下的近似等变性
|
|
@@ -45,7 +45,21 @@ GPU friendliness depends on the type of group:
|
|
|
45
45
|
|
|
46
46
|
## Further References
|
|
47
47
|
|
|
48
|
-
- Distillation notes: references/books/micro-lie-theory.md (Section II-B Group Actions)
|
|
49
|
-
- Distillation notes: references/books/smooth-manifolds.md (Ch 7 Lie Groups)
|
|
48
|
+
- Distillation notes: ../../references/books/micro-lie-theory.en.md (Section II-B Group Actions)
|
|
49
|
+
- Distillation notes: ../../references/books/smooth-manifolds.en.md (Ch 7 Lie Groups)
|
|
50
50
|
- Original text: Joan Sola et al., *A micro Lie theory*, Section II-B (group action definition and applications in robotics)
|
|
51
51
|
- Original text: John M. Lee, *Introduction to Smooth Manifolds*, Ch 7 (Lie groups and group actions)
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## Routing Extensions
|
|
55
|
+
- If equivariant map design is needed -> `equivariance.en.md` (equivariance under group actions)
|
|
56
|
+
- If linearization of group action is needed -> `representation.en.md` (representations are linear group actions)
|
|
57
|
+
- If invariant analysis is needed -> `symmetry` (design pattern layer for symmetry analysis)
|
|
58
|
+
|
|
59
|
+
## Extensible Directions
|
|
60
|
+
- Orbit-stabilizer theorem: relationship between orbits and stabilizers of group actions
|
|
61
|
+
- Transitive / free actions: special types of group actions
|
|
62
|
+
- Homogeneous space G/H: orbit space of group actions
|
|
63
|
+
- Quotient manifold: quotient structure under smooth group actions
|
|
64
|
+
- Slice theorem: local structure of compact group actions
|
|
65
|
+
- Momentum map: conserved quantities of Hamiltonian group actions
|
|
@@ -45,7 +45,21 @@ GPU 友好度取决于群的类型:
|
|
|
45
45
|
|
|
46
46
|
## 深入参考
|
|
47
47
|
|
|
48
|
-
-
|
|
49
|
-
-
|
|
48
|
+
- 蒸馏稿:../../references/books/micro-lie-theory.md(§II-B 群作用)
|
|
49
|
+
- 蒸馏稿:../../references/books/smooth-manifolds.md(Ch 7 Lie Groups)
|
|
50
50
|
- 原书:Joan Sola et al., *A micro Lie theory*, §II-B(群作用定义与机器人中的应用)
|
|
51
51
|
- 原书:John M. Lee, *Introduction to Smooth Manifolds*, Ch 7(李群与群作用)
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## 路由扩展
|
|
55
|
+
- 若需要等变映射的设计 → `equivariance.md`(群作用下的等变性)
|
|
56
|
+
- 若需要线性化群作用 → `representation.md`(群表示即线性群作用)
|
|
57
|
+
- 若需要不变量分析 → `symmetry`(设计模式层的对称性分析)
|
|
58
|
+
|
|
59
|
+
## 可扩展方向
|
|
60
|
+
- 轨道-稳定子定理(orbit-stabilizer theorem):群作用的轨道与稳定子关系
|
|
61
|
+
- 传递/自由作用(transitive / free actions):群作用的特殊类型
|
|
62
|
+
- 齐性空间 G/H:群作用的轨道空间
|
|
63
|
+
- 商流形(quotient manifold):光滑群作用下的商结构
|
|
64
|
+
- 切片定理(slice theorem):紧群作用的局部结构
|
|
65
|
+
- 动量映射(momentum map):Hamilton 群作用的守恒量
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Lie Theory Activation Index
|
|
2
|
+
|
|
3
|
+
## Domain Signals
|
|
4
|
+
Activate this domain direction when the problem involves:
|
|
5
|
+
- Symmetry / equivariance: model needs to be equivariant or invariant under group transformations
|
|
6
|
+
- Group parameterization: need to parameterize continuous symmetry groups (rotations, rigid body motions, etc.)
|
|
7
|
+
- Rotation / rigid body motion: involves specific Lie groups such as SO(3), SE(3)
|
|
8
|
+
- Conservation laws: correspondence between symmetries and conserved quantities
|
|
9
|
+
- Group representation: need to linearize group actions for network architecture design
|
|
10
|
+
- Orbit structure: need to analyze orbits and stabilizers of group actions
|
|
11
|
+
|
|
12
|
+
## Core Anchors
|
|
13
|
+
- `group-action.en.md` — Group actions
|
|
14
|
+
- `lie-group.en.md` — Lie groups
|
|
15
|
+
- `lie-algebra.en.md` — Lie algebras
|
|
16
|
+
- `representation.en.md` — Group representations
|
|
17
|
+
- `equivariance.en.md` — Equivariance
|
|
18
|
+
|
|
19
|
+
## Extended Concepts
|
|
20
|
+
When core anchors are insufficient, the following concepts may need temporary activation:
|
|
21
|
+
- Lie bracket: Lie bracket and commutation relations
|
|
22
|
+
- adjoint representation: adjoint representation and structure constants
|
|
23
|
+
- coadjoint orbit: coadjoint orbits and symplectic structure
|
|
24
|
+
- symplectic structure: symplectic forms and symplectic manifolds
|
|
25
|
+
- momentum map: momentum maps and conserved quantities
|
|
26
|
+
- geometric quantization: geometric quantization
|
|
27
|
+
- representation ring: representation ring and tensor product decomposition
|
|
28
|
+
- character theory: character theory
|
|
29
|
+
- Weyl group: Weyl group and root systems
|
|
30
|
+
- root system: root systems and Dynkin diagrams
|
|
31
|
+
- Dynkin diagram: Dynkin diagrams and classification
|
|
32
|
+
- highest weight classification: highest weight classification
|
|
33
|
+
- Peter-Weyl theorem: Peter-Weyl theorem and harmonic analysis
|
|
34
|
+
- harmonic analysis on groups: harmonic analysis on groups
|
|
35
|
+
- automorphic forms: automorphic forms
|
|
36
|
+
|
|
37
|
+
## Reference Book Directions
|
|
38
|
+
- `../../references/books/micro-lie-theory.en.md`: core content of Lie groups and Lie algebras
|
|
39
|
+
- `../../references/books/abstract-algebra.en.md`: abstract algebra foundations, group theory sections
|
|
40
|
+
|
|
41
|
+
## AI Translation Directions
|
|
42
|
+
- group action → data augmentation design / equivariant feature extraction
|
|
43
|
+
- lie group → rotation-equivariant layers / SO(3)-parameterized modules / pose estimation
|
|
44
|
+
- lie algebra → infinitesimal generators for continuous symmetries / steerable features
|
|
45
|
+
- representation → irreducible feature decomposition / spherical harmonics / Wigner-D layers
|
|
46
|
+
- equivariance → equivariant attention / weight sharing across orbits / orbit pooling
|
|
47
|
+
|
|
48
|
+
## Temporary Activation Rules
|
|
49
|
+
When the problem requires mathematics not in the core anchors:
|
|
50
|
+
1. First check whether extended concepts contain a match
|
|
51
|
+
2. If yes, generate a temporary knowledge card based on the lens
|
|
52
|
+
3. If no, enter the Knowledge Gap Protocol
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# 李理论 激活索引 / Lie Theory Activation Index
|
|
2
|
+
|
|
3
|
+
## 领域信号
|
|
4
|
+
当问题涉及以下信号时,激活本领域方向:
|
|
5
|
+
- 对称性/等变性:模型需要对群变换保持等变或不变
|
|
6
|
+
- 群参数化:需要参数化连续对称群(旋转、刚体运动等)
|
|
7
|
+
- 旋转/刚体运动:涉及 SO(3)、SE(3) 等具体李群
|
|
8
|
+
- 守恒律:对称性与守恒量的对应关系
|
|
9
|
+
- 群表示:需要将群作用线性化以设计网络结构
|
|
10
|
+
- 轨道结构:需要分析群作用的轨道与稳定子
|
|
11
|
+
|
|
12
|
+
## 核心锚点
|
|
13
|
+
- `group-action.md` — 群作用
|
|
14
|
+
- `lie-group.md` — 李群
|
|
15
|
+
- `lie-algebra.md` — 李代数
|
|
16
|
+
- `representation.md` — 群表示
|
|
17
|
+
- `equivariance.md` — 等变性
|
|
18
|
+
|
|
19
|
+
## 扩展概念
|
|
20
|
+
当核心锚点不够时,以下概念可能需要临时激活:
|
|
21
|
+
- Lie bracket:Lie 括号与对易关系
|
|
22
|
+
- adjoint representation:伴随表示与结构常数
|
|
23
|
+
- coadjoint orbit:余伴随轨道与辛结构
|
|
24
|
+
- symplectic structure:辛形式与辛流形
|
|
25
|
+
- momentum map:动量映射与守恒量
|
|
26
|
+
- geometric quantization:几何量子化
|
|
27
|
+
- representation ring:表示环与张量积分解
|
|
28
|
+
- character theory:特征标理论
|
|
29
|
+
- Weyl group:Weyl 群与根系
|
|
30
|
+
- root system:根系与 Dynkin 图
|
|
31
|
+
- Dynkin diagram:Dynkin 图与分类
|
|
32
|
+
- highest weight classification:最高权分类
|
|
33
|
+
- Peter-Weyl theorem:Peter-Weyl 定理与调和分析
|
|
34
|
+
- harmonic analysis on groups:群上的调和分析
|
|
35
|
+
- automorphic forms:自守形式
|
|
36
|
+
|
|
37
|
+
## 参考书方向
|
|
38
|
+
- `../../references/books/micro-lie-theory.md`:李群与李代数的核心内容
|
|
39
|
+
- `../../references/books/abstract-algebra.md`:抽象代数基础,群论部分
|
|
40
|
+
|
|
41
|
+
## AI 翻译方向
|
|
42
|
+
- group action → data augmentation design / equivariant feature extraction
|
|
43
|
+
- lie group → rotation-equivariant layers / SO(3)-parameterized modules / pose estimation
|
|
44
|
+
- lie algebra → infinitesimal generators for continuous symmetries / steerable features
|
|
45
|
+
- representation → irreducible feature decomposition / spherical harmonics / Wigner-D layers
|
|
46
|
+
- equivariance → equivariant attention / weight sharing across orbits / orbit pooling
|
|
47
|
+
|
|
48
|
+
## 临时激活规则
|
|
49
|
+
当问题需要的数学不在核心锚点中时:
|
|
50
|
+
1. 先检查扩展概念中是否有匹配
|
|
51
|
+
2. 若有,根据透镜生成临时知识卡
|
|
52
|
+
3. 若无,进入 Knowledge Gap Protocol
|
|
@@ -45,7 +45,22 @@ High GPU friendliness. The core advantage of the Lie algebra is that it is a "li
|
|
|
45
45
|
|
|
46
46
|
## Further References
|
|
47
47
|
|
|
48
|
-
- Distillation notes: references/books/micro-lie-theory.md (Section II-C Tangent Space and Lie Algebra, hat/vee operators)
|
|
49
|
-
- Distillation notes: references/books/differential-geometry.md (Ch 5 Lie Groups, Lie algebra section)
|
|
48
|
+
- Distillation notes: ../../references/books/micro-lie-theory.en.md (Section II-C Tangent Space and Lie Algebra, hat/vee operators)
|
|
49
|
+
- Distillation notes: ../../references/books/differential-geometry.en.md (Ch 5 Lie Groups, Lie algebra section)
|
|
50
50
|
- Original text: Joan Sola et al., *A micro Lie theory*, Section II-C (Lie algebra definition and hat/vee), Section II-D (exp/log bridge)
|
|
51
51
|
- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 5 (Lie groups and Lie algebras)
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## Routing Extensions
|
|
55
|
+
- If the corresponding global group is needed -> `lie-group.en.md` (integrating Lie algebra yields Lie group)
|
|
56
|
+
- If algebra representations are needed -> `representation.en.md` (representation theory of Lie algebras)
|
|
57
|
+
- If acting as the tangent space of a group -> `../differential-geometry/tangent-space.md` (Lie algebra structure on tangent space)
|
|
58
|
+
|
|
59
|
+
## Extensible Directions
|
|
60
|
+
- Structure constants: components of Lie bracket in a basis
|
|
61
|
+
- Jacobi identity: fundamental axiom of Lie algebras
|
|
62
|
+
- Ideal / subalgebra: substructures of Lie algebras
|
|
63
|
+
- Nilpotent / solvable / semisimple classification: structure theorems for Lie algebras
|
|
64
|
+
- Killing form: invariant bilinear form of Lie algebras
|
|
65
|
+
- Cartan subalgebra: maximal toral subalgebra of semisimple Lie algebras
|
|
66
|
+
- Root system: root system classification of semisimple Lie algebras
|
|
@@ -45,7 +45,22 @@ GPU 友好度高。李代数的核心优势是"线性空间":
|
|
|
45
45
|
|
|
46
46
|
## 深入参考
|
|
47
47
|
|
|
48
|
-
-
|
|
49
|
-
-
|
|
48
|
+
- 蒸馏稿:../../references/books/micro-lie-theory.md(§II-C 切空间与李代数, hat/vee 算子)
|
|
49
|
+
- 蒸馏稿:../../references/books/differential-geometry.md(Ch 5 Lie Groups, 李代数部分)
|
|
50
50
|
- 原书:Joan Sola et al., *A micro Lie theory*, §II-C(李代数定义与 hat/vee)、§II-D(exp/log 桥梁)
|
|
51
51
|
- 原书:Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 5(李群与李代数)
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## 路由扩展
|
|
55
|
+
- 若需要对应的全局群 → `lie-group.md`(李代数积分得到李群)
|
|
56
|
+
- 若需要代数表示 → `representation.md`(李代数的表示理论)
|
|
57
|
+
- 若作为群的切空间 → `../differential-geometry/tangent-space.md`(切空间的李代数结构)
|
|
58
|
+
|
|
59
|
+
## 可扩展方向
|
|
60
|
+
- 结构常数(structure constants):李括号在基下的分量
|
|
61
|
+
- Jacobi 恒等式:李代数的基本公理
|
|
62
|
+
- 理想/子代数(ideal / subalgebra):李代数的子结构
|
|
63
|
+
- 幂零/可解/半单分类(nilpotent / solvable / semisimple):李代数的结构定理
|
|
64
|
+
- Killing 型(Killing form):李代数的不变双线性型
|
|
65
|
+
- Cartan 子代数(Cartan subalgebra):半单李代数的极大环面子代数
|
|
66
|
+
- 根系(root system):半单李代数的根系分类
|
|
@@ -45,7 +45,21 @@ GPU friendliness: the key factor is whether exp/log has a closed form.
|
|
|
45
45
|
|
|
46
46
|
## Further References
|
|
47
47
|
|
|
48
|
-
- Distillation notes: references/books/micro-lie-theory.md (Section II-A Lie Group Definition, Section II-D Exponential Map, Section II-E Plus/Minus Operators)
|
|
49
|
-
- Distillation notes: references/books/differential-geometry.md (Ch 5 Lie Groups)
|
|
50
|
-
- Distillation notes: references/books/smooth-manifolds.md (Ch 7 Lie Groups)
|
|
48
|
+
- Distillation notes: ../../references/books/micro-lie-theory.en.md (Section II-A Lie Group Definition, Section II-D Exponential Map, Section II-E Plus/Minus Operators)
|
|
49
|
+
- Distillation notes: ../../references/books/differential-geometry.en.md (Ch 5 Lie Groups)
|
|
50
|
+
- Distillation notes: ../../references/books/smooth-manifolds.en.md (Ch 7 Lie Groups)
|
|
51
51
|
- Original text: Joan Sola et al., *A micro Lie theory*, Section II-A through Section II-F (complete Lie group toolkit)
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## Routing Extensions
|
|
55
|
+
- If infinitesimal structure is needed -> `lie-algebra.en.md` (Lie algebra is the tangent space of a Lie group)
|
|
56
|
+
- If linear representations are needed -> `representation.en.md` (finite-dimensional representations of Lie groups)
|
|
57
|
+
- If optimization on Lie groups is needed -> `../optimization/riemannian-optimization.md` (Riemannian optimization on Lie groups)
|
|
58
|
+
|
|
59
|
+
## Extensible Directions
|
|
60
|
+
- Simply connected Lie group: universal covering group
|
|
61
|
+
- Covering group: covering maps between Lie groups
|
|
62
|
+
- Maximal torus: maximal abelian subgroup in compact Lie groups
|
|
63
|
+
- Compact / semisimple / solvable: structural classification of Lie groups
|
|
64
|
+
- Nilpotent: nilpotent Lie groups and nilpotent Lie algebras
|
|
65
|
+
- Exponential map properties: properties of the map from Lie algebra to Lie group
|
|
@@ -45,7 +45,21 @@ GPU 友好度:关键看 exp/log 是否有闭式。
|
|
|
45
45
|
|
|
46
46
|
## 深入参考
|
|
47
47
|
|
|
48
|
-
-
|
|
49
|
-
-
|
|
50
|
-
-
|
|
48
|
+
- 蒸馏稿:../../references/books/micro-lie-theory.md(§II-A 李群定义, §II-D 指数映射, §II-E 加减算子)
|
|
49
|
+
- 蒸馏稿:../../references/books/differential-geometry.md(Ch 5 Lie Groups)
|
|
50
|
+
- 蒸馏稿:../../references/books/smooth-manifolds.md(Ch 7 Lie Groups)
|
|
51
51
|
- 原书:Joan Sola et al., *A micro Lie theory*, §II-A ~ §II-F(完整李群工具链)
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## 路由扩展
|
|
55
|
+
- 若需要无穷小结构 → `lie-algebra.md`(李代数是李群的切空间)
|
|
56
|
+
- 若需要线性表示 → `representation.md`(李群的有限维表示)
|
|
57
|
+
- 若需要在李群上优化 → `../optimization/riemannian-optimization.md`(李群上的黎曼优化)
|
|
58
|
+
|
|
59
|
+
## 可扩展方向
|
|
60
|
+
- 单连通李群(simply connected Lie group):万有覆叠群
|
|
61
|
+
- 覆叠群(covering group):李群之间的覆叠映射
|
|
62
|
+
- 极大环面(maximal torus):紧李群中的极大交换子群
|
|
63
|
+
- 紧/半单/可解(compact / semisimple / solvable):李群的结构分类
|
|
64
|
+
- 幂零(nilpotent):幂零李群与幂零李代数
|
|
65
|
+
- 指数映射性质(exponential map properties):李代数到李群的映射特性
|
|
@@ -46,7 +46,22 @@ GPU friendliness depends on the group size and representation dimension:
|
|
|
46
46
|
|
|
47
47
|
## Further References
|
|
48
48
|
|
|
49
|
-
- Distillation notes: references/books/micro-lie-theory.md (Section II-F Adjoint $\text{Ad}_X$ and Adjoint Matrix)
|
|
50
|
-
- Distillation notes: references/books/differential-geometry.md (Ch 5 Lie Groups, adjoint representation section)
|
|
49
|
+
- Distillation notes: ../../references/books/micro-lie-theory.en.md (Section II-F Adjoint $\text{Ad}_X$ and Adjoint Matrix)
|
|
50
|
+
- Distillation notes: ../../references/books/differential-geometry.en.md (Ch 5 Lie Groups, adjoint representation section)
|
|
51
51
|
- Original text: Joan Sola et al., *A micro Lie theory*, Section II-F (adjoint representation, equations 30--35)
|
|
52
52
|
- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 5 (representations of Lie groups)
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
## Routing Extensions
|
|
56
|
+
- If equivariant network design is needed -> `equivariance.en.md` (representation theory drives equivariant network construction)
|
|
57
|
+
- If the specific form of group action is needed -> `group-action.en.md` (representations are linear group actions)
|
|
58
|
+
- If irreducible decomposition is needed -> `../matrix-analysis/spectral-decomposition.md` (analogous to spectral decomposition of matrices)
|
|
59
|
+
|
|
60
|
+
## Extensible Directions
|
|
61
|
+
- Irreducible representation: basic building blocks of representations
|
|
62
|
+
- Character: trace function and classification of representations
|
|
63
|
+
- Schur's lemma: morphisms between irreducible representations
|
|
64
|
+
- Peter-Weyl theorem: decomposition of regular representation for compact groups
|
|
65
|
+
- Induced representation: constructing representations of large groups from subgroups
|
|
66
|
+
- Tensor product of representations: combining multi-particle systems
|
|
67
|
+
- Clebsch-Gordan coefficients: transformation coefficients for tensor product decomposition into irreducibles
|
|
@@ -46,7 +46,22 @@ GPU 友好度取决于群的规模和表示维数:
|
|
|
46
46
|
|
|
47
47
|
## 深入参考
|
|
48
48
|
|
|
49
|
-
-
|
|
50
|
-
-
|
|
49
|
+
- 蒸馏稿:../../references/books/micro-lie-theory.md(§II-F 伴随 Ad_X 与伴随矩阵)
|
|
50
|
+
- 蒸馏稿:../../references/books/differential-geometry.md(Ch 5 Lie Groups, 伴随表示部分)
|
|
51
51
|
- 原书:Joan Sola et al., *A micro Lie theory*, §II-F(伴随表示,式 30-35)
|
|
52
52
|
- 原书:Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 5(李群的表示)
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
## 路由扩展
|
|
56
|
+
- 若需要等变网络设计 → `equivariance.md`(表示理论驱动等变网络构造)
|
|
57
|
+
- 若需要群作用的具体形式 → `group-action.md`(表示是线性群作用)
|
|
58
|
+
- 若需要不可约分解 → `../matrix-analysis/spectral-decomposition.md`(类比为矩阵的谱分解)
|
|
59
|
+
|
|
60
|
+
## 可扩展方向
|
|
61
|
+
- 不可约表示(irreducible representation):表示的基本构建块
|
|
62
|
+
- 特征标(character):表示的迹函数与分类
|
|
63
|
+
- Schur 引理:不可约表示间的态射
|
|
64
|
+
- Peter-Weyl 定理:紧群的正则表示分解
|
|
65
|
+
- 诱导表示(induced representation):从子群构造大群表示
|
|
66
|
+
- 表示的张量积(tensor product of representations):多粒子系统的组合
|
|
67
|
+
- Clebsch-Gordan 系数:张量积分解为不可约的变换系数
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Matrix Analysis Activation Index
|
|
2
|
+
|
|
3
|
+
## Domain Signals
|
|
4
|
+
Activate this domain direction when the problem involves:
|
|
5
|
+
- Dimensionality reduction / compression: need to compress high-dimensional data or parameters into a low-dimensional subspace
|
|
6
|
+
- Low-rank approximation: need to approximate large matrices with low-rank ones
|
|
7
|
+
- Condition number / ill-conditioning: numerical stability issues in matrix computations
|
|
8
|
+
- Orthogonal decomposition: need to decompose space into orthogonal subspaces
|
|
9
|
+
- Spectral structure: need to analyze eigenvalue / singular value distributions
|
|
10
|
+
- Subspace separation: need to measure or control relationships between subspaces
|
|
11
|
+
|
|
12
|
+
## Core Anchors
|
|
13
|
+
- `projection.en.md` — Projection operators and subspace projection
|
|
14
|
+
- `spectral-decomposition.en.md` — Spectral decomposition and eigendecomposition
|
|
15
|
+
- `low-rank-approximation.en.md` — Low-rank approximation and truncated SVD
|
|
16
|
+
- `positive-semidefinite.en.md` — Positive semidefinite matrices and PSD cone
|
|
17
|
+
- `matrix-perturbation.en.md` — Matrix perturbation theory and error bounds
|
|
18
|
+
|
|
19
|
+
## Extended Concepts
|
|
20
|
+
When core anchors are insufficient, the following concepts may need temporary activation:
|
|
21
|
+
- SVD variants (truncated SVD, randomized SVD): fast decomposition methods for large-scale matrices
|
|
22
|
+
- PCA / kernel PCA: principal component analysis and its kernelized version
|
|
23
|
+
- condition number: computation and control of condition numbers
|
|
24
|
+
- pseudoinverse: Moore-Penrose generalized inverse and its applications
|
|
25
|
+
- matrix equation (Sylvester / Lyapunov): solution methods and stability for matrix equations
|
|
26
|
+
- Schur decomposition: Schur decomposition and invariant subspaces
|
|
27
|
+
- Jordan form: Jordan canonical form and generalized eigenspaces
|
|
28
|
+
- matrix function: definition and computation of matrix functions
|
|
29
|
+
- Kronecker product / vectorization: tensor product and vectorization operations
|
|
30
|
+
- randomized linear algebra: randomized linear algebra methods
|
|
31
|
+
- CUR decomposition: column/row sampling-based matrix approximation
|
|
32
|
+
- Nystrom approximation: low-rank approximation of kernel matrices
|
|
33
|
+
- matrix concentration inequalities: concentration inequalities for random matrices
|
|
34
|
+
|
|
35
|
+
## Reference Book Directions
|
|
36
|
+
- `../../references/books/matrix-analysis.en.md`: comprehensive coverage of matrix analysis, especially spectral decomposition, perturbation theory, and matrix function chapters
|
|
37
|
+
|
|
38
|
+
## AI Translation Directions
|
|
39
|
+
- projection → subspace attention / conflict removal / shared-private split
|
|
40
|
+
- spectral decomposition → low-rank KV cache / token pruning / stability monitor
|
|
41
|
+
- low-rank approximation → LoRA / adapter modules / memory-efficient attention
|
|
42
|
+
- positive-semidefinite → covariance-aware regularization / PSD-constrained loss
|
|
43
|
+
- matrix perturbation → condition number monitoring / robustness-aware training
|
|
44
|
+
|
|
45
|
+
## Temporary Activation Rules
|
|
46
|
+
When the problem requires mathematics not in the core anchors:
|
|
47
|
+
1. First check whether extended concepts contain a match
|
|
48
|
+
2. If yes, generate a temporary knowledge card based on the lens
|
|
49
|
+
3. If no, enter the Knowledge Gap Protocol
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# 矩阵分析 激活索引 / Matrix Analysis Activation Index
|
|
2
|
+
|
|
3
|
+
## 领域信号
|
|
4
|
+
当问题涉及以下信号时,激活本领域方向:
|
|
5
|
+
- 降维/压缩:需要将高维数据或参数压缩到低维子空间
|
|
6
|
+
- 低秩近似:需要用低秩矩阵近似大规模矩阵
|
|
7
|
+
- 条件数/病态:矩阵运算的数值稳定性出现问题
|
|
8
|
+
- 正交分解:需要将空间分解为正交子空间
|
|
9
|
+
- 谱结构:需要分析矩阵的特征值/奇异值分布
|
|
10
|
+
- 子空间分离:需要度量或控制子空间之间的关系
|
|
11
|
+
|
|
12
|
+
## 核心锚点
|
|
13
|
+
- `projection.md` — 投影算子与子空间投影
|
|
14
|
+
- `spectral-decomposition.md` — 谱分解与特征分解
|
|
15
|
+
- `low-rank-approximation.md` — 低秩近似与截断 SVD
|
|
16
|
+
- `positive-semidefinite.md` — 半正定矩阵与 PSD 锥
|
|
17
|
+
- `matrix-perturbation.md` — 矩阵扰动理论与误差界
|
|
18
|
+
|
|
19
|
+
## 扩展概念
|
|
20
|
+
当核心锚点不够时,以下概念可能需要临时激活:
|
|
21
|
+
- SVD 变体(truncated SVD, randomized SVD):大规模矩阵的快速分解方法
|
|
22
|
+
- PCA / kernel PCA:主成分分析及其核化版本
|
|
23
|
+
- condition number:条件数的计算与控制
|
|
24
|
+
- pseudoinverse:Moore-Penrose 广义逆及其应用
|
|
25
|
+
- matrix equation(Sylvester / Lyapunov):矩阵方程的解法与稳定性
|
|
26
|
+
- Schur decomposition:Schur 分解与不变子空间
|
|
27
|
+
- Jordan form:Jordan 标准形与广义特征空间
|
|
28
|
+
- matrix function:矩阵函数的定义与计算
|
|
29
|
+
- Kronecker product / vectorization:张量积与向量化运算
|
|
30
|
+
- randomized linear algebra:随机化线性代数方法
|
|
31
|
+
- CUR decomposition:基于列/行采样的矩阵近似
|
|
32
|
+
- Nystrom approximation:核矩阵的低秩近似
|
|
33
|
+
- matrix concentration inequalities:随机矩阵的集中不等式
|
|
34
|
+
|
|
35
|
+
## 参考书方向
|
|
36
|
+
- `../../references/books/matrix-analysis.md`:矩阵分析全面覆盖,特别是谱分解、扰动理论和矩阵函数章节
|
|
37
|
+
|
|
38
|
+
## AI 翻译方向
|
|
39
|
+
- projection → subspace attention / conflict removal / shared-private split
|
|
40
|
+
- spectral decomposition → low-rank KV cache / token pruning / stability monitor
|
|
41
|
+
- low-rank approximation → LoRA / adapter modules / memory-efficient attention
|
|
42
|
+
- positive-semidefinite → covariance-aware regularization / PSD-constrained loss
|
|
43
|
+
- matrix perturbation → condition number monitoring / robustness-aware training
|
|
44
|
+
|
|
45
|
+
## 临时激活规则
|
|
46
|
+
当问题需要的数学不在核心锚点中时:
|
|
47
|
+
1. 先检查扩展概念中是否有匹配
|
|
48
|
+
2. 若有,根据透镜生成临时知识卡
|
|
49
|
+
3. 若无,进入 Knowledge Gap Protocol
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
## Minimal Definition
|
|
4
4
|
|
|
5
|
-
Given a matrix $A \in \mathbb{R}^{m \times n}$, find a matrix $B$ with rank at most $k$ that minimizes $\|A - B\|$. The Eckart-Young-Mirsky theorem guarantees that the truncated SVD provides
|
|
5
|
+
Given a matrix $A \in \mathbb{R}^{m \times n}$, find a matrix $B$ with rank at most $k$ that minimizes $\|A - B\|$. The Eckart-Young-Mirsky theorem guarantees that the truncated SVD provides an optimal solution under both the Frobenius norm and the spectral norm; the principal subspace is unique when $\sigma_k > \sigma_{k+1}$: $B_k = \sum_{i=1}^k \sigma_i u_i v_i^H$.
|
|
6
6
|
|
|
7
7
|
## Core Formulas
|
|
8
8
|
|
|
@@ -16,7 +16,7 @@ Given a matrix $A \in \mathbb{R}^{m \times n}$, find a matrix $B$ with rank at m
|
|
|
16
16
|
## Applicable Problems
|
|
17
17
|
|
|
18
18
|
- LoRA weight compression: $W \approx W_0 + BA$, $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times d}$, $r \ll d$
|
|
19
|
-
- KV-Cache compression: projecting Key/Value caches into a low-
|
|
19
|
+
- KV-Cache compression: projecting Key/Value caches into a low-rank factor format, reducing memory from $O(Ld)$ to $O(Lk + kd)$; standard softmax still has sequence length $L$
|
|
20
20
|
- PCA / whitening: the top $k$ principal components of the data covariance matrix correspond to the truncated SVD
|
|
21
21
|
- Gradient compression: the effective rank of gradient matrices in large models is often much lower than the nominal rank, allowing safe truncation
|
|
22
22
|
- Recommender systems / matrix completion: low-rank factorization $R \approx UV^H$
|
|
@@ -25,16 +25,16 @@ Given a matrix $A \in \mathbb{R}^{m \times n}$, find a matrix $B$ with rank at m
|
|
|
25
25
|
|
|
26
26
|
- **LoRA (Low-Rank Adaptation)**: Freeze $W_0$, train $\Delta W = BA$ ($r \ll d$), and merge $W = W_0 + BA$ at inference time. Forward pass = two matmul operations ($x \to Ax \to BAx$), reducing trainable parameters from $O(d^2)$ to $O(dr)$. Implemented via `torch.mm(B, torch.mm(A, x))` or merged into a single matmul.
|
|
27
27
|
- **Randomized SVD operator**: For a large matrix $A \in \mathbb{R}^{m \times n}$, first sample $Y = A\Omega$ ($\Omega \in \mathbb{R}^{n \times (k+p)}$ random Gaussian), compute QR decomposition $Y = QR$, then form $B = Q^HA$ (small matrix $O(k \times n)$), and perform SVD on $B$. Total complexity $O(mnk)$ instead of $O(mn^2)$; all core operations are matmul.
|
|
28
|
-
- **KV-Cache low-rank reduction**: Maintain $
|
|
28
|
+
- **KV-Cache low-rank reduction**: Maintain $K$ in factored low-rank form $K \approx U_k \Sigma_k V_k^H$ (storing $U_k \in \mathbb{R}^{L \times k}$ and $\Sigma_k V_k^H \in \mathbb{R}^{k \times d}$, totaling $O(Lk + kd)$ instead of $O(Ld)$). Perform incremental PCA or streaming SVD updates for each new token. **Note**: for standard softmax attention, low-rank factors cannot be treated as $k$ compressed tokens; softmax is still normalized over length $L$. However, one can compute $qK^T = (q(\Sigma_k V_k^H)^T)U_k^T$ in factored form, avoiding materialized $L \times d$ reconstruction and reducing the QK inner dimension from $d$ to $k$. Only in linear attention, and only when compressing additive statistics such as $\phi(K)^T V$ and $\phi(K)^T\mathbf{1}$, can the historical state truly shrink from $L$ tokens to $k$ statistical factors.
|
|
29
29
|
- **Nuclear norm regularization**: $\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \|W\|_*$ promotes low-rank solutions. However, nuclear norm computation requires full SVD ($O(n^3)$). Alternatives: (1) approximate with truncated SVD; (2) factorize $\|W\|_* = \min_{W=UV^H} \frac{1}{2}(\|U\|_F^2 + \|V\|_F^2)$, converting to Frobenius regularization on $U, V$.
|
|
30
|
-
- **Gradient low-rank compression (distributed training)**:
|
|
30
|
+
- **Gradient low-rank compression (distributed training)**: For a gradient matrix $G \in \mathbb{R}^{m \times n}$, transmit top-$k$ SVD factors before all-reduce, reducing communication from $O(mn)$ to $O(k(m+n))$ (if $m \approx n \approx d$, this is $O(d^2) \to O(kd)$). Use randomized SVD locally on each device, then merge.
|
|
31
31
|
|
|
32
32
|
## Engineering Feasibility
|
|
33
33
|
|
|
34
|
-
- **Primary operations**: matmul + small-matrix SVD. LoRA forward = two matmul operations; randomized SVD =
|
|
34
|
+
- **Primary operations**: matmul + small-matrix SVD. LoRA forward = two matmul operations; randomized SVD = several matmuls + one thin QR + one small SVD, typically $O(mnk)$ (also depending on oversampling, power iterations, and spectral gaps). Lanczos / randomized truncated SVD is usually far cheaper than full SVD, but should not be summarized as an $O(k/n)$ fraction of full SVD.
|
|
35
35
|
- **GPU friendliness**: Extremely high. LoRA forward/backward are all tensor core matmul operations; the dominant cost of randomized SVD is also matmul. Small-matrix SVD has cuSOLVER batched implementations.
|
|
36
36
|
- **Complexity**: LoRA forward $O(dk)$ per sample vs. $O(d^2)$ full rank; randomized SVD $O(mnk)$; full SVD $O(\min(m^2n, mn^2))$.
|
|
37
|
-
- **Memory**: LoRA storage $O(dr)$ vs. $O(d^2)$; KV-Cache low-rank
|
|
37
|
+
- **Memory**: LoRA storage $O(dr)$ vs. $O(d^2)$; KV-Cache low-rank factor storage $O(Lk + kd)$ vs. $O(Ld)$.
|
|
38
38
|
|
|
39
39
|
## Risks and Failure Conditions
|
|
40
40
|
|
|
@@ -45,5 +45,18 @@ Given a matrix $A \in \mathbb{R}^{m \times n}$, find a matrix $B$ with rank at m
|
|
|
45
45
|
|
|
46
46
|
## Further References
|
|
47
47
|
|
|
48
|
-
- Distilled notes: references/books/matrix-analysis.md (Section 2.6 SVD, Section 7.4 Polar Decomposition and SVD, nuclear norm-spectral norm duality Section 5.5)
|
|
48
|
+
- Distilled notes: ../../references/books/matrix-analysis.en.md (Section 2.6 SVD, Section 7.4 Polar Decomposition and SVD, nuclear norm-spectral norm duality Section 5.5)
|
|
49
49
|
- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 2 Section 2.6 (SVD) + Chapter 7 Section 7.3-7.4 (Polar Decomposition & SVD)
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
## Routing Extensions
|
|
53
|
+
- If subspace projection implementation is needed -> `projection.en.md` (projection operators)
|
|
54
|
+
- If detailed decomposition tools are needed -> `spectral-decomposition.en.md` (SVD/EVD)
|
|
55
|
+
- If the goal is information-preserving compression -> `../probability/information-bottleneck.en.md` (information bottleneck theory)
|
|
56
|
+
|
|
57
|
+
## Extensible Directions
|
|
58
|
+
- Tensor decomposition (CP / Tucker / TT): low-rank decomposition of higher-order tensors
|
|
59
|
+
- Structured low-rank (Toeplitz / Hankel): structure-preserving low-rank approximation
|
|
60
|
+
- Online / streaming low-rank: incrementally updated low-rank estimation
|
|
61
|
+
- Matrix completion: recovering low-rank matrices from partial observations
|
|
62
|
+
- Robust PCA: low-rank + sparse decomposition
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
## 最小定义
|
|
4
4
|
|
|
5
|
-
给定矩阵 $A \in \mathbb{R}^{m \times n}$,寻找秩不超过 $k$ 的矩阵 $B$ 使得 $\|A - B\|$ 最小。Eckart-Young-Mirsky 定理保证截断 SVD 给出 Frobenius
|
|
5
|
+
给定矩阵 $A \in \mathbb{R}^{m \times n}$,寻找秩不超过 $k$ 的矩阵 $B$ 使得 $\|A - B\|$ 最小。Eckart-Young-Mirsky 定理保证截断 SVD 给出 Frobenius 范数和谱范数下的一个最优解;当 $\sigma_k > \sigma_{k+1}$ 时,对应主子空间唯一:$B_k = \sum_{i=1}^k \sigma_i u_i v_i^H$。
|
|
6
6
|
|
|
7
7
|
## 核心公式
|
|
8
8
|
|
|
@@ -16,7 +16,7 @@
|
|
|
16
16
|
## 适用问题
|
|
17
17
|
|
|
18
18
|
- LoRA 权重压缩:$W \approx W_0 + BA$,$B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times d}$,$r \ll d$
|
|
19
|
-
- KV-Cache 压缩:将 Key/Value
|
|
19
|
+
- KV-Cache 压缩:将 Key/Value 缓存投影到低秩因子格式,显存从 $O(Ld)$ 变为 $O(Lk + kd)$;标准 softmax 下序列长度仍为 $L$
|
|
20
20
|
- PCA / 白化:数据协方差的前 $k$ 个主成分即截断 SVD
|
|
21
21
|
- 梯度压缩:大模型梯度矩阵的有效秩通常远低于名义秩,可安全截断
|
|
22
22
|
- 推荐系统 / 矩阵补全:低秩因子分解 $R \approx UV^H$
|
|
@@ -25,16 +25,16 @@
|
|
|
25
25
|
|
|
26
26
|
- **LoRA (Low-Rank Adaptation)**:冻结 $W_0$,训练 $\Delta W = BA$($r \ll d$),推理时合并 $W = W_0 + BA$。前向传播 = 两次 matmul($x \to Ax \to BAx$),训练参数量从 $O(d^2)$ 降到 $O(dr)$。用 `torch.mm(B, torch.mm(A, x))` 或合并为单次 matmul。
|
|
27
27
|
- **随机化 SVD 算子**:对大矩阵 $A \in \mathbb{R}^{m \times n}$,先采样 $Y = A\Omega$($\Omega \in \mathbb{R}^{n \times (k+p)}$ 随机高斯),QR 分解 $Y = QR$,再算 $B = Q^HA$(小矩阵 $O(k \times n)$),对 $B$ 做 SVD。总复杂度 $O(mnk)$ 而非 $O(mn^2)$,核心操作全是 matmul。
|
|
28
|
-
- **KV-Cache 低秩化**:维护 $
|
|
28
|
+
- **KV-Cache 低秩化**:维护 $K$ 的低秩因子形式 $K \approx U_k \Sigma_k V_k^H$(存储 $U_k \in \mathbb{R}^{L \times k}$ 和 $\Sigma_k V_k^H \in \mathbb{R}^{k \times d}$,共 $O(Lk + kd)$ 而非 $O(Ld)$)。每新到 token 做增量 PCA 或 streaming SVD 更新。**注意**:对标准 softmax attention,低秩因子不能当作 $k$ 个压缩 token;softmax 仍在长度 $L$ 上归一化。但可用因子化计算 $qK^T = (q(\Sigma_k V_k^H)^T)U_k^T$,避免物化 $L \times d$ 重构,并将 QK 内维从 $d$ 降到 $k$。仅在线性注意力中,且压缩 $\phi(K)^T V$、$\phi(K)^T\mathbf{1}$ 等可加统计量时,历史状态才可真正从 $L$ 降至 $k$ 个统计因子。
|
|
29
29
|
- **核范数正则化**:$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \|W\|_*$ 促进低秩解。但核范数计算需完整 SVD($O(n^3)$),替代方案:(1) 用截断 SVD 近似;(2) 因子化 $\|W\|_* = \min_{W=UV^H} \frac{1}{2}(\|U\|_F^2 + \|V\|_F^2)$ 转为对 $U, V$ 的 Frobenius 正则。
|
|
30
|
-
- **梯度低秩压缩 (分布式训练)
|
|
30
|
+
- **梯度低秩压缩 (分布式训练)**:对梯度矩阵 $G \in \mathbb{R}^{m \times n}$ 做 top-$k$ SVD 后传输因子,all-reduce 通信量从 $O(mn)$ 降到 $O(k(m+n))$(若 $m \approx n \approx d$,即 $O(d^2) \to O(kd)$)。用随机化 SVD 在每卡本地算,再合并。
|
|
31
31
|
|
|
32
32
|
## 工程可行性
|
|
33
33
|
|
|
34
|
-
- **主要操作**:matmul + 小矩阵 SVD。LoRA 前向 = 两次 matmul;随机化 SVD =
|
|
34
|
+
- **主要操作**:matmul + 小矩阵 SVD。LoRA 前向 = 两次 matmul;随机化 SVD = 若干次 matmul + 一次 thin QR + 小矩阵 SVD,典型复杂度 $O(mnk)$(还依赖 oversampling、power iteration 和谱间隙);Lanczos/随机化截断 SVD 通常远低于完整 SVD,但不能简单写成完整 SVD 的 $O(k/n)$ 倍。
|
|
35
35
|
- **GPU 友好度**:极高。LoRA 前向/反向全是 tensor core matmul;随机化 SVD 的主要开销也是 matmul。小矩阵 SVD 有 cuSOLVER 的 batched 实现。
|
|
36
36
|
- **复杂度**:LoRA 前向 $O(dk)$ per sample vs. $O(d^2)$ 全秩;随机化 SVD $O(mnk)$;完整 SVD $O(\min(m^2n, mn^2))$。
|
|
37
|
-
- **显存**:LoRA 存储 $O(dr)$ vs. $O(d^2)$;KV-Cache
|
|
37
|
+
- **显存**:LoRA 存储 $O(dr)$ vs. $O(d^2)$;KV-Cache 低秩因子存储 $O(Lk + kd)$ vs. $O(Ld)$。
|
|
38
38
|
|
|
39
39
|
## 风险与失效条件
|
|
40
40
|
|
|
@@ -45,5 +45,18 @@
|
|
|
45
45
|
|
|
46
46
|
## 深入参考
|
|
47
47
|
|
|
48
|
-
-
|
|
48
|
+
- 蒸馏稿:../../references/books/matrix-analysis.md(§2.6 SVD、§7.4 极分解与 SVD、核范数-谱范数对偶 §5.5)
|
|
49
49
|
- 原书:Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 2 §2.6 (SVD) + Chapter 7 §7.3-7.4 (Polar Decomposition & SVD)
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
## 路由扩展
|
|
53
|
+
- 若需要子空间投影的具体实现 → `projection.md`(投影算子)
|
|
54
|
+
- 若需要分解工具的详细分析 → `spectral-decomposition.md`(SVD/EVD)
|
|
55
|
+
- 若目标是信息保持压缩 → `../probability/information-bottleneck.md`(信息瓶颈理论)
|
|
56
|
+
|
|
57
|
+
## 可扩展方向
|
|
58
|
+
- 张量分解(CP / Tucker / TT):高阶张量的低秩分解
|
|
59
|
+
- 结构化低秩(Toeplitz / Hankel):保持结构的低秩近似
|
|
60
|
+
- 在线/流式低秩(online / streaming low-rank):增量更新的低秩估计
|
|
61
|
+
- 矩阵补全(matrix completion):从部分观测恢复低秩矩阵
|
|
62
|
+
- 鲁棒 PCA(robust PCA):低秩+稀疏分解
|
|
@@ -45,5 +45,17 @@ Studies how the eigenvalues, singular values, and eigenspaces of a matrix $A$ ch
|
|
|
45
45
|
|
|
46
46
|
## Further References
|
|
47
47
|
|
|
48
|
-
- Distilled notes: references/books/matrix-analysis.md (Section 4.3 Eigenvalue Inequalities Weyl/Interlacing, Section 6.1-6.3 Geršgorin Discs and Perturbation Theorems, Section 5.8 Condition Numbers)
|
|
48
|
+
- Distilled notes: ../../references/books/matrix-analysis.en.md (Section 4.3 Eigenvalue Inequalities Weyl/Interlacing, Section 6.1-6.3 Geršgorin Discs and Perturbation Theorems, Section 5.8 Condition Numbers)
|
|
49
49
|
- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 4 Section 4.3 (Eigenvalue Inequalities) + Chapter 6 (Location and Perturbation of Eigenvalues Section 6.1-6.3)
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
## Routing Extensions
|
|
53
|
+
- If focusing on eigenvalue sensitivity -> `spectral-decomposition.en.md` (perturbation analysis of spectral decomposition)
|
|
54
|
+
- If random perturbation bounds are involved -> `../probability/concentration-inequality.md` (random matrix concentration inequalities)
|
|
55
|
+
|
|
56
|
+
## Extensible Directions
|
|
57
|
+
- Pseudospectra: spectral sensitivity analysis for non-normal matrices
|
|
58
|
+
- Structured perturbation: perturbation analysis preserving matrix structure
|
|
59
|
+
- Davis-Kahan theorem variants: multiple bounds for subspace perturbation
|
|
60
|
+
- Eigenvalue interlacing: relationship between submatrix and eigenvalues
|
|
61
|
+
- Wilkinson polynomial: condition numbers and sensitivity of polynomial roots
|
|
@@ -45,5 +45,17 @@
|
|
|
45
45
|
|
|
46
46
|
## 深入参考
|
|
47
47
|
|
|
48
|
-
-
|
|
48
|
+
- 蒸馏稿:../../references/books/matrix-analysis.md(§4.3 特征值不等式 Weyl/交错、§6.1-6.3 Ger\v{s}gorin 圆盘与扰动定理、§5.8 条件数)
|
|
49
49
|
- 原书:Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 4 §4.3 (Eigenvalue Inequalities) + Chapter 6 (Location and Perturbation of Eigenvalues §6.1-6.3)
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
## 路由扩展
|
|
53
|
+
- 若关注特征值灵敏度 → `spectral-decomposition.md`(谱分解的扰动分析)
|
|
54
|
+
- 若涉及随机扰动界 → `../probability/concentration-inequality.md`(随机矩阵集中不等式)
|
|
55
|
+
|
|
56
|
+
## 可扩展方向
|
|
57
|
+
- 伪谱(pseudospectra):非正规矩阵的谱灵敏度分析
|
|
58
|
+
- 结构化扰动(structured perturbation):保持矩阵结构的扰动分析
|
|
59
|
+
- Davis-Kahan 定理变体:子空间扰动的多种界
|
|
60
|
+
- 特征值交错(eigenvalue interlacing):主子矩阵与特征值关系
|
|
61
|
+
- Wilkinson 多项式:多项式根的条件数与扰动灵敏度
|