math-skill 3.0.0 → 3.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en-US.md +9 -1
- package/README.md +9 -1
- package/agents/math-critic.en.md +27 -25
- package/agents/math-critic.md +27 -27
- package/commands/ask.en.md +10 -0
- package/commands/ask.md +1 -1
- package/design-patterns/attention/equivariant-attention.en.md +9 -9
- package/design-patterns/attention/equivariant-attention.md +9 -9
- package/design-patterns/attention/geometry-aware-attention.en.md +9 -9
- package/design-patterns/attention/geometry-aware-attention.md +9 -9
- package/design-patterns/attention/information-bottleneck-attention.en.md +9 -9
- package/design-patterns/attention/information-bottleneck-attention.md +9 -9
- package/design-patterns/attention/projection-attention.en.md +9 -9
- package/design-patterns/attention/projection-attention.md +9 -9
- package/design-patterns/attention/spectral-attention.en.md +9 -9
- package/design-patterns/attention/spectral-attention.md +9 -9
- package/design-patterns/compression/leverage-score-selection.en.md +1 -1
- package/design-patterns/compression/leverage-score-selection.md +1 -1
- package/design-patterns/compression/low-rank-kv-cache.en.md +1 -1
- package/design-patterns/compression/low-rank-kv-cache.md +1 -1
- package/design-patterns/compression/spectral-token-pruning.en.md +1 -1
- package/design-patterns/compression/spectral-token-pruning.md +1 -1
- package/design-patterns/compression/topology-preserving-compression.en.md +1 -1
- package/design-patterns/compression/topology-preserving-compression.md +1 -1
- package/design-patterns/loss/constraint-penalty.en.md +1 -1
- package/design-patterns/loss/constraint-penalty.md +1 -1
- package/design-patterns/loss/contrastive-loss.en.md +1 -1
- package/design-patterns/loss/contrastive-loss.md +1 -1
- package/design-patterns/loss/information-bottleneck-loss.en.md +1 -1
- package/design-patterns/loss/information-bottleneck-loss.md +1 -1
- package/design-patterns/loss/orthogonality-loss.en.md +1 -1
- package/design-patterns/loss/orthogonality-loss.md +1 -1
- package/design-patterns/loss/variational-loss.en.md +1 -1
- package/design-patterns/loss/variational-loss.md +1 -1
- package/design-patterns/representation/equivariant-split.en.md +1 -1
- package/design-patterns/representation/equivariant-split.md +1 -1
- package/design-patterns/representation/manifold-representation.en.md +1 -1
- package/design-patterns/representation/manifold-representation.md +1 -1
- package/design-patterns/representation/shared-private-decomposition.en.md +1 -1
- package/design-patterns/representation/shared-private-decomposition.md +1 -1
- package/design-patterns/representation/subspace-alignment.en.md +1 -1
- package/design-patterns/representation/subspace-alignment.md +1 -1
- package/design-patterns/routing/graph-routing.en.md +1 -1
- package/design-patterns/routing/graph-routing.md +1 -1
- package/design-patterns/routing/moe-routing.en.md +1 -1
- package/design-patterns/routing/moe-routing.md +1 -1
- package/design-patterns/routing/optimal-transport-routing.en.md +1 -1
- package/design-patterns/routing/optimal-transport-routing.md +1 -1
- package/design-patterns/routing/spectral-clustering-routing.en.md +1 -1
- package/design-patterns/routing/spectral-clustering-routing.md +1 -1
- package/knowledge-base/differential-geometry/metric-tensor.md +1 -1
- package/knowledge-base/information-geometry/fisher-metric.en.md +8 -8
- package/knowledge-base/information-geometry/fisher-metric.md +8 -8
- package/knowledge-base/information-geometry/natural-gradient.en.md +7 -7
- package/knowledge-base/information-geometry/natural-gradient.md +7 -7
- package/knowledge-base/overview.en.md +2 -2
- package/knowledge-base/overview.md +2 -2
- package/knowledge-base/probability/concentration-inequality.en.md +5 -5
- package/knowledge-base/probability/concentration-inequality.md +5 -5
- package/knowledge-base/probability/entropy.en.md +5 -5
- package/knowledge-base/probability/entropy.md +5 -5
- package/knowledge-base/probability/fisher-information.en.md +7 -7
- package/knowledge-base/probability/fisher-information.md +7 -7
- package/knowledge-base/probability/information-bottleneck.en.md +6 -6
- package/knowledge-base/probability/information-bottleneck.md +6 -6
- package/knowledge-base/probability/kl-divergence.en.md +6 -6
- package/knowledge-base/probability/kl-divergence.md +6 -6
- package/lenses/algorithmic.en.md +1 -1
- package/lenses/algorithmic.md +1 -1
- package/lenses/axiomatization.en.md +1 -1
- package/lenses/axiomatization.md +1 -1
- package/lenses/categorical.en.md +1 -1
- package/lenses/categorical.md +1 -1
- package/lenses/causal.en.md +1 -1
- package/lenses/causal.md +1 -1
- package/lenses/duality.en.md +1 -1
- package/lenses/duality.md +1 -1
- package/lenses/game.en.md +2 -2
- package/lenses/game.md +2 -2
- package/lenses/geometric.en.md +1 -1
- package/lenses/geometric.md +1 -1
- package/lenses/local-to-global.en.md +1 -1
- package/lenses/local-to-global.md +1 -1
- package/lenses/perturbation.en.md +1 -1
- package/lenses/perturbation.md +1 -1
- package/lenses/probabilistic.en.md +1 -1
- package/lenses/probabilistic.md +1 -1
- package/lenses/projection.en.md +1 -1
- package/lenses/projection.md +1 -1
- package/lenses/spectral.en.md +1 -1
- package/lenses/spectral.md +1 -1
- package/lenses/symmetry.en.md +1 -1
- package/lenses/symmetry.md +1 -1
- package/lenses/topological.en.md +1 -1
- package/lenses/topological.md +1 -1
- package/lenses/variational.en.md +1 -1
- package/lenses/variational.md +1 -1
- package/package.json +1 -1
- package/references/agentic-workflow.en.md +1 -1
- package/references/agentic-workflow.md +1 -1
- package/references/books/abstract-algebra.en.md +124 -0
- package/references/books/abstract-algebra.md +17 -17
- package/references/books/algebraic-geometry-rising-sea.en.md +171 -0
- package/references/books/algebraic-geometry-rising-sea.md +7 -7
- package/references/books/differential-geometry.en.md +140 -0
- package/references/books/differential-geometry.md +18 -18
- package/references/books/matrix-analysis.en.md +146 -0
- package/references/books/matrix-analysis.md +8 -8
- package/references/books/micro-lie-theory.en.md +116 -0
- package/references/books/micro-lie-theory.md +16 -16
- package/references/books/optimization-ml.en.md +163 -0
- package/references/books/optimization-ml.md +6 -7
- package/references/books/smooth-manifolds.en.md +105 -0
- package/references/books/smooth-manifolds.md +12 -12
- package/references/gpu-friendly-math.en.md +15 -2
- package/references/gpu-friendly-math.md +15 -2
- package/skills/math-research-activator/SKILL.en.md +27 -1
- package/skills/math-research-activator/SKILL.md +26 -3
package/README.en-US.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
<a href="README.md">中文</a> | <a href="README.en-US.md">English</a>
|
|
3
3
|
</p>
|
|
4
4
|
|
|
5
|
-
#
|
|
5
|
+
# 📐 Math Skill — A Mathematical Research Operating System for AI Architecture Design
|
|
6
6
|
|
|
7
7
|
> **The thinking system does not hand out theorems. The knowledge system does not improvise inspiration. The design layer does not fake profundity.**
|
|
8
8
|
|
|
@@ -225,6 +225,14 @@ Distillation files ship with the npm package. For full-fidelity lookups, place P
|
|
|
225
225
|
|
|
226
226
|
## Changelog
|
|
227
227
|
|
|
228
|
+
### v3.0.1 — Token Optimization & Bilingual Completion
|
|
229
|
+
|
|
230
|
+
- **Emoji cleanup**: removed all emoji characters from skill files; GPU rating markers ✅⚠️❌ replaced with text `[v]`/`[~]`/`[x]`, saving ~1,400 tokens
|
|
231
|
+
- **English distillation files**: added EN translations for all 7 book distillation notes (`references/books/*.en.md`) and `commands/ask.en.md`
|
|
232
|
+
- **Mixed-language routing**: added 5-rule decision system for code-switched input (technical terms excluded from language detection; sentence frame determines primary language)
|
|
233
|
+
- **GPU dimension abbreviations**: verbose labels `**Dimension N full-name**` compressed to `**DN**` (D1-D8), defined in `gpu-friendly-math.md`, saving ~800-1,000 tokens
|
|
234
|
+
- Terminology unification, cross-reference fixes, and other minor improvements
|
|
235
|
+
|
|
228
236
|
### v3.0.0 — Mathematical Research Operating System
|
|
229
237
|
|
|
230
238
|
**Architecture overhaul**: from "thinking weapon arsenal" to "math general staff" — three-layer orthogonal architecture:
|
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
<a href="README.md">中文</a> | <a href="README.en-US.md">English</a>
|
|
3
3
|
</p>
|
|
4
4
|
|
|
5
|
-
#
|
|
5
|
+
# 📐 Math Skill — 面向 AI 架构创新的数学研究操作系统
|
|
6
6
|
|
|
7
7
|
> **思想系统不负责给定理,知识系统不负责乱启发,设计层不负责装深刻。**
|
|
8
8
|
|
|
@@ -225,6 +225,14 @@ math-skill/
|
|
|
225
225
|
|
|
226
226
|
## 变更日志
|
|
227
227
|
|
|
228
|
+
### v3.0.1 — Token 优化与双语补全
|
|
229
|
+
|
|
230
|
+
- **Emoji 清理**:移除所有 skill 文件中的 emoji 符号,GPU 评级标记 ✅⚠️❌ 替换为文本 `[v]`/`[~]`/`[x]`,节约 ~1,400 tokens
|
|
231
|
+
- **英文蒸馏稿补全**:7 本书蒸馏稿新增英文翻译(`references/books/*.en.md`),`commands/ask.en.md` 新增英文入口
|
|
232
|
+
- **混合语言路由**:新增 5 条判定规则,解决中英混杂输入的路由问题(技术词不计入语言判定,按句式主框架判定)
|
|
233
|
+
- **GPU 维度缩写**:八维标签从 `**维度 N 全称**` 压缩为 `**DN**`(D1-D8),定义于 `gpu-friendly-math.md`,额外节约 ~800-1,000 tokens
|
|
234
|
+
- 术语统一、交叉引用修正及其他小问题修复
|
|
235
|
+
|
|
228
236
|
### v3.0.0 — 数学研究操作系统
|
|
229
237
|
|
|
230
238
|
**架构重构**:从"思想武器库"升级为"数学参谋部"——三层正交架构:
|
package/agents/math-critic.en.md
CHANGED
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
# Math Critic Sub-Agent
|
|
2
2
|
|
|
3
|
+
> **File routing**: Follow the language routing rules in `../skills/math-research-activator/SKILL.en.md`. Chinese primary → load `math-critic.md`; English primary → use this file.
|
|
4
|
+
|
|
3
5
|
## Role
|
|
4
6
|
|
|
5
|
-
You are a mathematical assistant with both critical review and implementation capabilities. Your primary task is to evaluate the reliability and applicability of arguments, proposals, or conclusions from a mathematical perspective, while also providing concrete implementation strategies, problem-solving approaches, or proof steps when necessary.
|
|
7
|
+
You are a mathematical assistant with both critical review and implementation capabilities. Your primary task is to evaluate the reliability and applicability of arguments, proposals, or conclusions from a mathematical perspective, while also providing concrete implementation strategies, problem-solving approaches, or proof steps when necessary. You additionally serve as the **dual-acceptance gatekeeper**: ensuring every deliverable simultaneously satisfies "mathematical beauty x GPU friendliness."
|
|
6
8
|
|
|
7
9
|
You should give equal weight to two categories of responsibility: on one hand, scrutinize the chain of reasoning, the basis of assumptions, model applicability, computational feasibility, and **GPU feasibility**; on the other hand, when the user explicitly requires it, provide practical mathematical derivations, implementation frameworks, or proof details.
|
|
8
10
|
|
|
@@ -24,113 +26,113 @@ You should give equal weight to two categories of responsibility: on one hand, s
|
|
|
24
26
|
|
|
25
27
|
## Review Dimensions
|
|
26
28
|
|
|
27
|
-
The
|
|
29
|
+
The first 15 dimensions cover the core review angles — assumptions, logic, models, computation — most of which correspond to the v3 thinking lenses in `../lenses/`; Dimensions 16–18 are cross-cutting: tool selection, GPU feasibility, and modern math activation. **Do not mechanically check every dimension one by one** -- select the most relevant dimensions for in-depth review based on the nature of the problem and the user's focus; the rest may be briefly mentioned or skipped. If the deliverable involves algorithm/operator/GPU design, **Dimensions 17 (GPU) and 18 (Modern Math Activation) are mandatory checkpoints**.
|
|
28
30
|
|
|
29
|
-
### 1. Assumption Review
|
|
31
|
+
### 1. Assumption Review -> axiomatization lens
|
|
30
32
|
|
|
31
33
|
- What fundamental assumptions does the argument rely on?
|
|
32
34
|
- Are these assumptions reasonable? Are they explicitly stated?
|
|
33
35
|
- Are there hidden assumptions? If the assumptions fail, does the conclusion still hold?
|
|
34
36
|
|
|
35
|
-
### 2. Abstraction Level Assessment
|
|
37
|
+
### 2. Abstraction Level Assessment -> categorical lens
|
|
36
38
|
|
|
37
39
|
- Is the argument operating at the right level of abstraction? Too concrete (missing general structure) or too abstract (losing critical details)?
|
|
38
40
|
- Is there a more suitable abstraction perspective?
|
|
39
41
|
- Does the abstraction process preserve key information? Has over-abstraction rendered the conclusions vacuous?
|
|
40
|
-
-
|
|
42
|
+
- Once the abstract structure is concretized, is it computable? Does it land on a tensorizable representation?
|
|
41
43
|
|
|
42
|
-
### 3. Logic Check
|
|
44
|
+
### 3. Logic Check (general dimension)
|
|
43
45
|
|
|
44
46
|
- Are there logical leaps in the reasoning process?
|
|
45
47
|
- Are sufficient conditions and necessary conditions conflated?
|
|
46
48
|
- Is there circular reasoning?
|
|
47
49
|
- Does the conclusion genuinely follow from the premises?
|
|
48
50
|
|
|
49
|
-
### 4. Model Applicability
|
|
51
|
+
### 4. Model Applicability (general dimension)
|
|
50
52
|
|
|
51
53
|
- If a model is used, are its assumptions reasonable?
|
|
52
54
|
- Is the model oversimplified or overcomplicated?
|
|
53
55
|
- Has the model been validated?
|
|
54
56
|
|
|
55
|
-
### 5. Optimization Quality
|
|
57
|
+
### 5. Optimization Quality -> variational lens
|
|
56
58
|
|
|
57
59
|
- If optimization is involved, is the objective function clearly defined? Are the constraints complete?
|
|
58
60
|
- Has convexity been verified? Are local optima mistaken for global optima?
|
|
59
61
|
- Does the dual perspective provide additional insight or a simpler solution?
|
|
60
62
|
|
|
61
|
-
### 6. Quantitative Evaluation
|
|
63
|
+
### 6. Quantitative Evaluation -> probabilistic lens
|
|
62
64
|
|
|
63
65
|
- Can the concepts in the argument be quantified?
|
|
64
66
|
- If probability or statistics is involved, are the methods correct?
|
|
65
67
|
- Is the sample size sufficient? Is there selection bias?
|
|
66
68
|
|
|
67
|
-
### 7. Transformation Opportunity
|
|
69
|
+
### 7. Transformation Opportunity -> duality lens
|
|
68
70
|
|
|
69
71
|
- Is there a simpler equivalent representation of the problem? Is the current representation the most natural one?
|
|
70
72
|
- Does a transformation (Fourier/Laplace/generating function, etc.) exist that could simplify the problem?
|
|
71
73
|
- Does a dual or equivalent reformulation reveal hidden structure?
|
|
72
|
-
-
|
|
74
|
+
- Does the transformation convert unfriendly operations into GPU-friendly ones (e.g., convolution -> GEMM)?
|
|
73
75
|
|
|
74
|
-
### 8. Symmetry Exploitation
|
|
76
|
+
### 8. Symmetry Exploitation -> symmetry lens
|
|
75
77
|
|
|
76
78
|
- Does the problem have hidden symmetries that could reduce complexity?
|
|
77
79
|
- Can invariants simplify the analysis or classification?
|
|
78
80
|
- Are symmetry-breaking cases overlooked?
|
|
79
|
-
-
|
|
81
|
+
- Can group actions be tensorized and reduced to linear representations/GEMM?
|
|
80
82
|
|
|
81
|
-
### 9. Induction and Analogy Review
|
|
83
|
+
### 9. Induction and Analogy Review -> local-to-global lens
|
|
82
84
|
|
|
83
85
|
- Does the inductive reasoning start from a sufficient number of instances? Is there overgeneralization?
|
|
84
86
|
- Are the analogies structurally similar in substance, not merely on the surface?
|
|
85
87
|
- Are there counterexamples that weaken the inductive conclusion?
|
|
86
88
|
|
|
87
|
-
### 10. Computational Feasibility
|
|
89
|
+
### 10. Computational Feasibility -> algorithmic lens
|
|
88
90
|
|
|
89
91
|
- If computation is involved, is the procedure guaranteed to terminate? Are the time/space complexity acceptable?
|
|
90
92
|
- Does the problem belong to the NP-hard or undecidable class? Are approximation or heuristics needed?
|
|
91
93
|
- Do the numerical methods converge? Is the precision sufficient?
|
|
92
94
|
|
|
93
|
-
### 11. Information Structure Review
|
|
95
|
+
### 11. Information Structure Review -> probabilistic lens + `knowledge-base/probability/`
|
|
94
96
|
|
|
95
97
|
- Is the information structure of the problem clear? Is there redundant or missing information?
|
|
96
98
|
- Can uncertainty be quantified using information entropy, mutual information, etc.?
|
|
97
99
|
- Have bottlenecks in information transmission or compression been identified?
|
|
98
100
|
|
|
99
|
-
### 12. Game and Strategy Review
|
|
101
|
+
### 12. Game and Strategy Review -> game lens
|
|
100
102
|
|
|
101
103
|
- If multi-party interactions are involved, are strategic dependencies taken into account?
|
|
102
104
|
- Do Nash equilibria exist? Is the mechanism design incentive-compatible?
|
|
103
105
|
- Is information asymmetry in the game being overlooked?
|
|
104
106
|
|
|
105
|
-
### 13. Causal Chain Review
|
|
107
|
+
### 13. Causal Chain Review -> causal lens
|
|
106
108
|
|
|
107
109
|
- Does the argument conflate correlation with causation?
|
|
108
110
|
- Can an intervention framework or counterfactual reasoning be used to verify the causal direction?
|
|
109
111
|
- Are confounding or mediating variables being ignored?
|
|
110
112
|
|
|
111
|
-
### 14. Topological Structure Review
|
|
113
|
+
### 14. Topological Structure Review -> topological lens
|
|
112
114
|
|
|
113
115
|
- Are the connectivity structure and boundary behavior of the problem handled correctly?
|
|
114
116
|
- Do topological obstructions (holes, entanglements) affect the conclusion?
|
|
115
117
|
- Do properties invariant under continuous deformation provide a simplifying perspective?
|
|
116
|
-
-
|
|
118
|
+
- Can local topological invariants be computed in batch-parallel fashion? Is global exact homology being erroneously forced into training?
|
|
117
119
|
|
|
118
|
-
### 15. Counterexample and Boundary
|
|
120
|
+
### 15. Counterexample and Boundary (general dimension)
|
|
119
121
|
|
|
120
122
|
- Can a counterexample be constructed to refute the conclusion?
|
|
121
123
|
- Does the conclusion hold in boundary cases (limiting scenarios)?
|
|
122
124
|
- What is the scope of applicability of the conclusion? Are there undeclared exceptions?
|
|
123
125
|
- Is the enumeration of finite cases exhaustive?
|
|
124
126
|
|
|
125
|
-
### 16. Tool-Selection and Flow Review
|
|
127
|
+
### 16. Tool-Selection and Flow Review -> math-research-activator
|
|
126
128
|
|
|
127
129
|
- Is the selection of review dimensions itself optimal? Have any critical dimensions been overlooked?
|
|
128
130
|
- Were the thinking toolkits best suited to the nature of the problem selected, rather than merely the most familiar ones?
|
|
129
|
-
-
|
|
131
|
+
- Is the activator main flow followed: **Intent diagnosis -> Lens routing -> Knowledge lookup -> Design translation -> GPU screening**? Was diagnosis skipped in favor of jumping straight to math exposition?
|
|
130
132
|
- Were multiple candidate structures enumerated (rather than only one)?
|
|
131
133
|
- If uncertain about which dimensions to review, first invoke `/ask` to have the activator recommend the 3-5 most suitable review dimensions.
|
|
132
134
|
|
|
133
|
-
### 17. GPU-Feasibility Review
|
|
135
|
+
### 17. GPU-Feasibility Review -> `references/gpu-friendly-math.en.md`
|
|
134
136
|
|
|
135
137
|
> **Mandatory** when the deliverable involves algorithm/operator/training/GPU design. Corresponds to the second gate of the "dual-acceptance gate."
|
|
136
138
|
|
|
@@ -139,7 +141,7 @@ The following dimensions correspond to the fifteen mathematical thinking toolkit
|
|
|
139
141
|
- Are the inverse transforms and numerical components stable (condition number, ill-conditioning)?
|
|
140
142
|
- Have memory and communication been assessed (KV-Cache, distributed all-reduce, optimizer state precision)?
|
|
141
143
|
|
|
142
|
-
### 18. Modern-Math Activation Review
|
|
144
|
+
### 18. Modern-Math Activation Review -> `references/books/*`
|
|
143
145
|
|
|
144
146
|
> **Mandatory** when the deliverable claims to "activate modern mathematics into algorithms." Corresponds to the first gate of the "dual-acceptance gate" (mathematical correctness) + cross-domain activation quality.
|
|
145
147
|
|
package/agents/math-critic.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# 数学审视子 Agent / Math Critic Sub-Agent
|
|
2
2
|
|
|
3
|
-
>
|
|
3
|
+
> **语言路由**:按 `../skills/math-research-activator/SKILL.md` 中的语言路由规则判定主语言。英文主语言 → 读取 `math-critic.en.md`,英文输出;中文主语言 → 继续使用本文件。
|
|
4
4
|
|
|
5
5
|
## 角色
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
你是一个兼具审查与实现能力的数学助手。主要任务是从数学角度评估论点、方案或结论的可靠性与适用性,同时在必要时提供具体的实现思路、解题方案或证明步骤。你还兼任**双验收门把关人**:确保每个产出同时满足「math beautiful × GPU friendly」。
|
|
8
8
|
|
|
9
9
|
你应并列考虑两类职责:一方面审视论证链条、假设依据、模型适用性、计算可行性与 **GPU 可行性**;另一方面在用户明确需要时,给出实际的数学推导、实现框架或证明细节。
|
|
10
10
|
|
|
@@ -26,113 +26,113 @@
|
|
|
26
26
|
|
|
27
27
|
## 审视维度
|
|
28
28
|
|
|
29
|
-
|
|
29
|
+
前 15 个维度覆盖假设、逻辑、模型、计算等核心审视角度(多数对应 `../lenses/` 中的 v3 思想透镜),第 16–18 维为横切维度:工具选择、GPU 可行性、现代数学激活。**不需要机械地逐一检查所有维度**——应根据问题性质和用户关注点,选择最相关的维度深入审视,其余可简略提及或跳过。若产出涉及算法/算子/GPU 设计,**第 17(GPU)与第 18(现代数学激活)维为强制检查项**。
|
|
30
30
|
|
|
31
|
-
### 1. 假设审查 / Assumption Review
|
|
31
|
+
### 1. 假设审查 / Assumption Review -> axiomatization(公理化透镜)
|
|
32
32
|
|
|
33
33
|
- 论点依赖了哪些基本假设?
|
|
34
34
|
- 这些假设是否合理?是否被明确声明?
|
|
35
35
|
- 是否存在隐含假设?如果假设不成立,结论是否仍然成立?
|
|
36
36
|
|
|
37
|
-
### 2. 抽象层级评估 / Abstraction Level Assessment
|
|
37
|
+
### 2. 抽象层级评估 / Abstraction Level Assessment -> categorical(范畴化透镜)
|
|
38
38
|
|
|
39
39
|
- 论证是否在正确的抽象层级?过具体(缺少一般结构)还是过抽象(丢失关键细节)?
|
|
40
40
|
- 是否存在更适合的抽象视角?
|
|
41
41
|
- 抽象过程是否保留了关键信息?是否过度抽象导致结论空洞?
|
|
42
|
-
-
|
|
42
|
+
- 抽象结构具体化后是否可计算?是否落到了可张量化的表示?
|
|
43
43
|
|
|
44
|
-
### 3. 逻辑检查 / Logic Check
|
|
44
|
+
### 3. 逻辑检查 / Logic Check (通用维度)
|
|
45
45
|
|
|
46
46
|
- 推理过程是否存在逻辑跳跃?
|
|
47
47
|
- 是否混淆了充分条件和必要条件?
|
|
48
48
|
- 是否存在循环论证?
|
|
49
49
|
- 结论是否真的能从前提推导出来?
|
|
50
50
|
|
|
51
|
-
### 4. 模型适用性 / Model Applicability
|
|
51
|
+
### 4. 模型适用性 / Model Applicability (通用维度)
|
|
52
52
|
|
|
53
53
|
- 如果使用了模型,模型的假设是否合理?
|
|
54
54
|
- 模型是否过度简化或过度复杂化?
|
|
55
55
|
- 模型是否经过验证?
|
|
56
56
|
|
|
57
|
-
### 5. 优化质量 / Optimization Quality
|
|
57
|
+
### 5. 优化质量 / Optimization Quality -> variational(变分透镜)
|
|
58
58
|
|
|
59
59
|
- 若涉及优化,目标函数是否明确定义?约束是否完备?
|
|
60
60
|
- 是否验证了凸性?局部最优是否被误认为全局最优?
|
|
61
61
|
- 对偶视角是否提供了额外的信息或更简的解法?
|
|
62
62
|
|
|
63
|
-
### 6. 量化评估 / Quantitative Evaluation
|
|
63
|
+
### 6. 量化评估 / Quantitative Evaluation -> probabilistic(概率统计透镜)
|
|
64
64
|
|
|
65
65
|
- 论点中的概念是否可以量化?
|
|
66
66
|
- 如果涉及概率或统计,方法是否正确?
|
|
67
67
|
- 样本量是否足够?是否存在选择偏差?
|
|
68
68
|
|
|
69
|
-
### 7. 变换机会 / Transformation Opportunity
|
|
69
|
+
### 7. 变换机会 / Transformation Opportunity -> duality(对偶透镜)
|
|
70
70
|
|
|
71
71
|
- 问题是否有更简的等价表示?当前表示是否最自然?
|
|
72
72
|
- 是否存在变换(傅里叶/拉普拉斯/生成函数等)可简化问题?
|
|
73
73
|
- 对偶或等价重述是否揭示了隐藏结构?
|
|
74
|
-
-
|
|
74
|
+
- 变换是否把不友好的运算变成 GPU 友好的运算(如卷积→GEMM)?
|
|
75
75
|
|
|
76
|
-
### 8. 对称性利用 / Symmetry Exploitation
|
|
76
|
+
### 8. 对称性利用 / Symmetry Exploitation -> symmetry(对称透镜)
|
|
77
77
|
|
|
78
78
|
- 问题是否有隐藏对称性可降低复杂度?
|
|
79
79
|
- 不变量是否可简化分析或分类?
|
|
80
80
|
- 是否遗漏了对称破缺的情况?
|
|
81
|
-
-
|
|
81
|
+
- 群作用是否可张量化、落到线性表示/GEMM?
|
|
82
82
|
|
|
83
|
-
### 9. 归纳与类比审视 / Induction & Analogy Review
|
|
83
|
+
### 9. 归纳与类比审视 / Induction & Analogy Review -> local-to-global(局部到整体透镜)
|
|
84
84
|
|
|
85
85
|
- 归纳推理是否从足够多的实例出发?是否存在过度推广?
|
|
86
86
|
- 类比是否在结构上真正相似,而非仅表面相似?
|
|
87
87
|
- 是否有反例削弱归纳结论?
|
|
88
88
|
|
|
89
|
-
### 10. 计算可行性 / Computational Feasibility
|
|
89
|
+
### 10. 计算可行性 / Computational Feasibility -> algorithmic(算法透镜)
|
|
90
90
|
|
|
91
91
|
- 若涉及计算,程序是否可保证终止?时间/空间复杂度是否可接受?
|
|
92
92
|
- 问题是否属于NP-hard或不可判定类别?是否需要近似或启发式?
|
|
93
93
|
- 数值方法是否收敛?精度是否足够?
|
|
94
94
|
|
|
95
|
-
### 11. 信息结构审视 / Information Structure Review
|
|
95
|
+
### 11. 信息结构审视 / Information Structure Review -> probabilistic(概率统计透镜)+ `knowledge-base/probability/`
|
|
96
96
|
|
|
97
97
|
- 问题的信息结构是否清晰?是否存在冗余或缺失的信息?
|
|
98
98
|
- 是否可以用信息熵、互信息等量化不确定性?
|
|
99
99
|
- 信息传输或压缩的瓶颈是否被识别?
|
|
100
100
|
|
|
101
|
-
### 12. 博弈与策略审视 / Game & Strategy Review
|
|
101
|
+
### 12. 博弈与策略审视 / Game & Strategy Review -> game(博弈透镜)
|
|
102
102
|
|
|
103
103
|
- 若涉及多方互动,是否考虑了策略依赖性?
|
|
104
104
|
- 是否存在纳什均衡?机制设计是否激励相容?
|
|
105
105
|
- 是否忽略了博弈中的信息不对称?
|
|
106
106
|
|
|
107
|
-
### 13. 因果链条审视 / Causal Chain Review
|
|
107
|
+
### 13. 因果链条审视 / Causal Chain Review -> causal(因果透镜)
|
|
108
108
|
|
|
109
109
|
- 论证是否混淆了相关与因果?
|
|
110
110
|
- 是否可以用干预框架或反事实推理验证因果方向?
|
|
111
111
|
- 是否存在混杂变量或中介变量被忽略?
|
|
112
112
|
|
|
113
|
-
### 14. 拓扑结构审视 / Topological Structure Review
|
|
113
|
+
### 14. 拓扑结构审视 / Topological Structure Review -> topological(拓扑透镜)
|
|
114
114
|
|
|
115
115
|
- 问题的连通结构、边界行为是否被正确处理?
|
|
116
116
|
- 是否存在拓扑障碍(空洞、纠缠)影响结论?
|
|
117
117
|
- 连续变形下的不变性质是否提供了简化视角?
|
|
118
|
-
-
|
|
118
|
+
- 局部拓扑量是否可批量并行?全局精确同调是否被误塞进训练?
|
|
119
119
|
|
|
120
|
-
### 15. 反例与边界 / Counterexample & Boundary
|
|
120
|
+
### 15. 反例与边界 / Counterexample & Boundary (通用维度)
|
|
121
121
|
|
|
122
122
|
- 能否构造一个反例来推翻结论?
|
|
123
123
|
- 在边界情况(极限情形)下结论是否成立?
|
|
124
124
|
- 结论的适用范围是什么?是否存在未声明的例外?
|
|
125
125
|
- 有限情况下的枚举是否穷尽?
|
|
126
126
|
|
|
127
|
-
### 16. 工具选择与流程审视 / Tool-Selection & Flow Review
|
|
127
|
+
### 16. 工具选择与流程审视 / Tool-Selection & Flow Review -> math-research-activator
|
|
128
128
|
|
|
129
129
|
- 审视维度的选择本身是否最优?是否遗漏了关键维度?
|
|
130
130
|
- 是否选择了最适合问题性质的思想透镜,而非仅仅选择最熟悉的?
|
|
131
|
-
-
|
|
131
|
+
- 是否遵循激活器主流程:**意图诊断 → 透镜路由 → 知识调用 → 设计翻译 → GPU 筛选**?是否跳过了诊断就堆数学科普?
|
|
132
132
|
- 是否枚举了多个候选结构(而非只给一个)?
|
|
133
133
|
- 若不确定该审视哪些维度,先调用 `/ask` 让激活器推荐最合适的 3–5 个审视维度。
|
|
134
134
|
|
|
135
|
-
### 17. GPU 可行性审视 / GPU-Feasibility Review
|
|
135
|
+
### 17. GPU 可行性审视 / GPU-Feasibility Review -> `references/gpu-friendly-math.md`
|
|
136
136
|
|
|
137
137
|
> 产出涉及算法/算子/训练/GPU 设计时为**强制项**。对应「双验收门」的第二道门。
|
|
138
138
|
|
|
@@ -141,7 +141,7 @@
|
|
|
141
141
|
- 逆变换/数值部分是否稳定(条件数、病态)?
|
|
142
142
|
- 显存与通信是否被评估(KV-Cache、分布式 all-reduce、optimizer state 精度)?
|
|
143
143
|
|
|
144
|
-
### 18. 现代数学激活审视 / Modern-Math Activation Review
|
|
144
|
+
### 18. 现代数学激活审视 / Modern-Math Activation Review -> `references/books/*`
|
|
145
145
|
|
|
146
146
|
> 产出声称「把现代数学激活进算法」时为**强制项**。对应「双验收门」的第一道门(数学正确)+ 跨域激活质量。
|
|
147
147
|
|
|
@@ -184,7 +184,7 @@
|
|
|
184
184
|
- [列出本次选择深入审视的 3–5 个维度及原因;若涉算法/GPU,标注 17/18 已纳入]
|
|
185
185
|
|
|
186
186
|
#### 双验收门结果
|
|
187
|
-
- [候选1]: 数学正确[
|
|
187
|
+
- [候选1]: 数学正确[v/x] | GPU 八维[友好/可改造/不友好] | 通过[是/否]
|
|
188
188
|
- [候选2]: ...
|
|
189
189
|
|
|
190
190
|
#### 做得好的地方
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ask
|
|
3
|
+
description: |
|
|
4
|
+
Math Research OS entry: auto-diagnose user intent, route to thinking lenses, math knowledge base, or design translation layer.
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
Read and follow `../skills/math-research-activator/SKILL.en.md`.
|
|
8
|
+
|
|
9
|
+
Current question:
|
|
10
|
+
$ARGUMENTS
|
package/commands/ask.md
CHANGED
|
@@ -5,7 +5,7 @@ description: |
|
|
|
5
5
|
English: Math Research OS entry: auto-diagnose user intent, route to thinking lenses, math knowledge base, or design translation layer.
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
按 `../skills/math-research-activator/SKILL.md` 中的语言路由与混合输入规则判定主语言。英文主语言 → 读取 `../skills/math-research-activator/SKILL.en.md`;中文主语言 → 读取 `../skills/math-research-activator/SKILL.md`。
|
|
9
9
|
|
|
10
10
|
当前问题:
|
|
11
11
|
$ARGUMENTS
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Equivariant Attention
|
|
2
|
-
> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as
|
|
2
|
+
> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
|
|
3
3
|
|
|
4
4
|
## Applicable Problems
|
|
5
5
|
When the input possesses an **explicit symmetry group $G$ action** (rotation, translation, permutation, reflection, etc.), and the desired model output should be **equivariant** (covariant) rather than invariant under the same transformations, equivariant constraints must be directly encoded into the attention mechanism. Typical scenarios include: 3D point clouds / molecules ($E(3)$ rigid-body group), image classification ($D_n$ rotation/reflection group), set data ($S_n$ permutation group), and multi-view / multi-sensor fusion.
|
|
@@ -55,14 +55,14 @@ output = mean(softmax((rho(g)@X@W_q) @ (rho(g)@X@W_k).T/sqrt(d)) @ (rho(g)@X@W_v
|
|
|
55
55
|
- **G-CNN Attention**: $D_n$ rotation/reflection equivariance for remote sensing imagery and medical imaging
|
|
56
56
|
|
|
57
57
|
## GPU Feasibility
|
|
58
|
-
- **
|
|
59
|
-
- **
|
|
60
|
-
- **
|
|
61
|
-
- **
|
|
62
|
-
- **
|
|
63
|
-
- **
|
|
64
|
-
- **
|
|
65
|
-
- **
|
|
58
|
+
- **D1**: Group actions implemented as $\rho(g)$ matrix multiplications; equivariant features stored as batched tensors
|
|
59
|
+
- **D2**: $\rho(g) X$ and $Q K^T$ are both GEMM operations; $|G|$ group elements map to batched GEMM
|
|
60
|
+
- **D3**: $|G|$-fold computation overhead; acceptable for small groups ($|D_4|=8$), infeasible for large groups ($|S_n|=n!$). Remedy: use generators + Cayley graph propagation instead of full group enumeration
|
|
61
|
+
- **D4**: Requires storing $|G|$ copies of intermediate features; mitigated by chunking + gradient checkpointing
|
|
62
|
+
- **D5**: Orthogonal representation matrices are numerically stable under bf16
|
|
63
|
+
- **D6**: $|G|$ group elements are naturally parallelizable (along the batch dimension)
|
|
64
|
+
- **D7**: Permutation $\rho(g)$ is extremely sparse and can be encoded as gather indices
|
|
65
|
+
- **D8**: Group action + linear duality can be fused into a single batched GEMM
|
|
66
66
|
|
|
67
67
|
## Paper Phrasing
|
|
68
68
|
"We propose an equivariant attention mechanism that constrains attention weights to be group invariants and attention outputs to be group equivariants, directly encoding the inductive bias of symmetry group $G$ into the model architecture without additional data augmentation, achieving a $|G|/|stab|$-fold improvement in parameter efficiency."
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# 等变注意力 / Equivariant Attention
|
|
2
|
-
> **严谨性声明**:本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache
|
|
2
|
+
> **严谨性声明**:本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行,需工程验证。
|
|
3
3
|
|
|
4
4
|
## 适用问题
|
|
5
5
|
当输入具有**明确的对称群 $G$ 作用**(旋转、平移、置换、反射等),且期望模型输出在相同变换下**协变**(equivariant)而非不变时,需要将等变约束直接编入注意力机制。典型场景:3D 点云/分子($E(3)$ 刚体群)、图像分类($D_n$ 旋转/镜像群)、集合数据($S_n$ 置换群)、多视角/多传感器融合。
|
|
@@ -55,14 +55,14 @@ output = mean(softmax((rho(g)@X@W_q) @ (rho(g)@X@W_k).T/sqrt(d)) @ (rho(g)@X@W_v
|
|
|
55
55
|
- **G-CNN 注意力**:$D_n$ 旋转/镜像等变,用于遥感图像和医学影像
|
|
56
56
|
|
|
57
57
|
## GPU 可行性
|
|
58
|
-
-
|
|
59
|
-
-
|
|
60
|
-
-
|
|
61
|
-
-
|
|
62
|
-
-
|
|
63
|
-
-
|
|
64
|
-
-
|
|
65
|
-
-
|
|
58
|
+
- **D1[v]**:群作用实现为 $\rho(g)$ 矩阵乘,等变特征为 batched 张量
|
|
59
|
+
- **D2[v]**:$\rho(g) X$ 和 $Q K^T$ 均为 GEMM;$|G|$ 个群元素 → batched GEMM
|
|
60
|
+
- **D3[~]**:$|G|$ 倍计算量,小群($|D_4|=8$)可接受,大群($|S_n|=n!$)不可。改造:用生成元 + Cayley 图传播替代整群枚举
|
|
61
|
+
- **D4[~]**:需存储 $|G|$ 份中间特征,可分块 + gradient checkpointing
|
|
62
|
+
- **D5[v]**:正交表示矩阵在 bf16 下数值稳定
|
|
63
|
+
- **D6[v]**:$|G|$ 个群元素天然并行(batch 维)
|
|
64
|
+
- **D7[~]**:置换 $\rho(g)$ 极度稀疏,可编码为 gather 索引
|
|
65
|
+
- **D8[v]**:群作用 + 线性变换可融合为单次 batched GEMM
|
|
66
66
|
|
|
67
67
|
## 论文表述方式
|
|
68
68
|
"我们提出等变注意力机制,通过将注意力权重约束为群不变量、注意力输出约束为群等变量,在不增加数据增强的情况下将对称群 $G$ 的归纳偏置直接编码进模型结构,参数效率提升 $|G|/|stab|$ 倍。"
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Geometry-Aware Attention
|
|
2
|
-
> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as
|
|
2
|
+
> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
|
|
3
3
|
|
|
4
4
|
## Applicable Problems
|
|
5
5
|
When **known geometric relationships** exist between tokens/keys (spatial distance, manifold geodesic distance, hierarchical structure, temporal interval), the position-agnostic inner product of standard attention cannot exploit this structural information. Geometry-aware attention **directly injects geometric priors into attention weight computation**, enabling the model to naturally respect the metric structure of the underlying space. Typical scenarios include: 3D scene understanding, molecular conformation, temporal forecasting, hierarchical text structure (paragraph-sentence-word), and knowledge graphs.
|
|
@@ -56,14 +56,14 @@ def manifold_attention(Q, K, V, manifold):
|
|
|
56
56
|
- **Molecular Conformation Attention**: 3D atomic coordinates -> distance matrix -> geometric bias, used in molecular GNNs and protein structure prediction
|
|
57
57
|
|
|
58
58
|
## GPU Feasibility
|
|
59
|
-
- **
|
|
60
|
-
- **
|
|
61
|
-
- **
|
|
62
|
-
- **
|
|
63
|
-
- **
|
|
64
|
-
- **
|
|
65
|
-
- **
|
|
66
|
-
- **
|
|
59
|
+
- **D1**: Distance matrices and bias matrices are dense tensors with element-wise operations
|
|
60
|
+
- **D2**: The main body $Q K^T$ is standard GEMM; geometric bias is additive and does not interrupt GEMM
|
|
61
|
+
- **D3**: Pairwise distance matrix construction and storage are $O(n^2)$; however, block-wise computation + online softmax can avoid materializing the full matrix
|
|
62
|
+
- **D4**: The $n \times n$ distance matrix occupies GPU memory; mitigated by block/streaming computation (compatible with FlashAttention)
|
|
63
|
+
- **D5**: Distance computation and bias addition are stable under bf16; the exp in RBF requires overflow caution (clamp distances)
|
|
64
|
+
- **D6**: Distance computation and attention can be pipelined; pairwise distances can be computed in parallel blocks
|
|
65
|
+
- **D7**: Distant biases tend toward $-\infty$ (approaching zero after softmax), naturally inducing structured sparsity (local attention windows)
|
|
66
|
+
- **D8**: Geometric bias can be fused into FlashAttention's online softmax loop (added after $QK^T$, before softmax)
|
|
67
67
|
|
|
68
68
|
## Paper Phrasing
|
|
69
69
|
"We propose geometry-aware attention, which explicitly injects learnable geometric distance bias terms into attention scores, enabling the model to naturally respect the metric structure of the input space. This achieves exponential decay of attention to distant tokens without increasing parameter count, while maintaining compatibility with FlashAttention."
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# 几何感知注意力 / Geometry-Aware Attention
|
|
2
|
-
> **严谨性声明**:本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache
|
|
2
|
+
> **严谨性声明**:本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行,需工程验证。
|
|
3
3
|
|
|
4
4
|
## 适用问题
|
|
5
5
|
当 token/key 之间存在**已知的几何关系**(空间距离、流形测地距、层级结构、时序间隔)时,标准注意力的位置无关内积无法利用这些结构信息。几何感知注意力将**几何先验直接注入注意力权重计算**,使模型天然尊重底空间的度量结构。典型场景:3D 场景理解、分子构象、时序预测、层级文本结构(段落-句子-词)、知识图谱。
|
|
@@ -55,14 +55,14 @@ def manifold_attention(Q, K, V, manifold):
|
|
|
55
55
|
- **分子构象注意力**:3D 原子坐标 → 距离矩阵 → 几何偏置,用于分子 GNN 和蛋白质结构预测
|
|
56
56
|
|
|
57
57
|
## GPU 可行性
|
|
58
|
-
-
|
|
59
|
-
-
|
|
60
|
-
-
|
|
61
|
-
-
|
|
62
|
-
-
|
|
63
|
-
-
|
|
64
|
-
-
|
|
65
|
-
-
|
|
58
|
+
- **D1[v]**:距离矩阵和偏置矩阵均为稠密张量,逐元素运算
|
|
59
|
+
- **D2[v]**:主体 $Q K^T$ 为标准 GEMM;几何偏置为加法,不阻断 GEMM
|
|
60
|
+
- **D3[~]**:成对距离矩阵 $O(n^2)$ 构建和存储;但可用分块计算 + online softmax 避免物化完整矩阵
|
|
61
|
+
- **D4[~]**:$n \times n$ 距离矩阵占用显存;可分块/流式计算(与 FlashAttention 兼容)
|
|
62
|
+
- **D5[v]**:距离计算和偏置加法在 bf16 下稳定;RBF 的 exp 需注意上溢(clamp 距离)
|
|
63
|
+
- **D6[v]**:距离计算和注意力可流水线并行,成对距离可分块并行
|
|
64
|
+
- **D7[v]**:远距离偏置趋于 $-\infty$(softmax 后趋零),天然诱导结构化稀疏(局部注意力窗口)
|
|
65
|
+
- **D8[v]**:几何偏置可融入 FlashAttention 的 online softmax 循环中(加在 $QK^T$ 之后、softmax 之前)
|
|
66
66
|
|
|
67
67
|
## 论文表述方式
|
|
68
68
|
"我们提出几何感知注意力,通过在注意力分数中显式注入可学习的几何距离偏置项,使模型天然尊重输入空间的度量结构,在不增加参数量的情况下将远距离 token 的注意力指数衰减,同时保持与 FlashAttention 的兼容性。"
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Information Bottleneck Attention
|
|
2
|
-
> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as
|
|
2
|
+
> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
|
|
3
3
|
|
|
4
4
|
## Applicable Problems
|
|
5
5
|
When the attention mechanism needs to **selectively transmit useful information while suppressing redundant/noisy information**, information bottleneck theory can guide the learning of attention weights -- maximizing the mutual information $I(Z;Y)$ of the attention distribution with respect to the target $Y$, while minimizing the mutual information $I(X;Z)$ with respect to the input $X$. Typical scenarios include: long-document summarization (filtering large numbers of irrelevant tokens), multimodal alignment (cross-modal noise suppression), and interpretability (attention weights as visualization of information flow).
|
|
@@ -55,14 +55,14 @@ loss = -info_nce + beta * kl_bottleneck
|
|
|
55
55
|
- **Multi-Head Information Allocation**: Different heads learn different information bottlenecks (different $\beta$); some heads transmit global information, others only local information
|
|
56
56
|
|
|
57
57
|
## GPU Feasibility
|
|
58
|
-
- **
|
|
59
|
-
- **
|
|
60
|
-
- **
|
|
61
|
-
- **
|
|
62
|
-
- **
|
|
63
|
-
- **
|
|
64
|
-
- **
|
|
65
|
-
- **
|
|
58
|
+
- **D1**: KL regularization involves element-wise operations; VIB reparameterization sampling is element-wise
|
|
59
|
+
- **D2**: The main body $QK^T$ and $attn \cdot V$ are standard GEMM; regularization introduces no new GEMM operations
|
|
60
|
+
- **D3**: KL regularization is $O(n)$ per token, adding no asymptotic complexity
|
|
61
|
+
- **D4**: VIB requires additional $\mu_z$ and $\log\sigma_z$, approximately doubling attention weight memory
|
|
62
|
+
- **D5**: log/exp in KL computations are stable under bf16 (standard log-softmax tricks)
|
|
63
|
+
- **D6**: Regularization can be computed in parallel with forward propagation, introducing no serial dependencies
|
|
64
|
+
- **D7**: KL regularization implicitly induces attention sparsity, amenable to block-sparse acceleration
|
|
65
|
+
- **D8**: KL can be fused into the softmax kernel (FusedSoftmaxKL)
|
|
66
66
|
|
|
67
67
|
## Paper Phrasing
|
|
68
68
|
"We propose information bottleneck attention, which models attention as an information bottleneck variational problem. By maximizing the mutual information between output and target while minimizing redundant information transmission from the input, it achieves theoretically optimal information selection and demonstrates enhanced sparsity and interpretability in experiments."
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# 信息瓶颈注意力 / Information Bottleneck Attention
|
|
2
|
-
> **严谨性声明**:本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache
|
|
2
|
+
> **严谨性声明**:本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行,需工程验证。
|
|
3
3
|
|
|
4
4
|
## 适用问题
|
|
5
5
|
当注意力机制需要**选择性传递有用信息、抑制冗余/噪声信息**时,用信息瓶颈理论指导注意力权重的学习——使注意力分布最大化关于目标 $Y$ 的互信息 $I(Z;Y)$,同时最小化关于输入 $X$ 的互信息 $I(X;Z)$。典型场景:长文档摘要(大量无关 token 需被过滤)、多模态对齐(跨模态噪声抑制)、可解释性(注意力权重作为信息流的可视化)。
|
|
@@ -55,14 +55,14 @@ loss = -info_nce + beta * kl_bottleneck
|
|
|
55
55
|
- **Multi-Head 信息分配**:不同 head 学习不同的信息瓶颈(不同 $\beta$),有的 head 传递全局信息,有的只传递局部信息
|
|
56
56
|
|
|
57
57
|
## GPU 可行性
|
|
58
|
-
-
|
|
59
|
-
-
|
|
60
|
-
-
|
|
61
|
-
-
|
|
62
|
-
-
|
|
63
|
-
-
|
|
64
|
-
-
|
|
65
|
-
-
|
|
58
|
+
- **D1[v]**:KL 正则项为逐元素运算,VIB 重参数化采样为逐元素
|
|
59
|
+
- **D2[v]**:主体 $QK^T$ 和 $attn \cdot V$ 为标准 GEMM,正则项不引入新 GEMM
|
|
60
|
+
- **D3[v]**:KL 正则 $O(n)$ 逐 token,不增加渐近复杂度
|
|
61
|
+
- **D4[~]**:VIB 需额外 $\mu_z$ 和 $\log\sigma_z$,约 2x 注意力权重显存
|
|
62
|
+
- **D5[v]**:KL 中 log/exp 在 bf16 下稳定(标准 log-softmax 技巧)
|
|
63
|
+
- **D6[v]**:正则项可与前向传播并行,不引入串行依赖
|
|
64
|
+
- **D7[v]**:KL 正则隐式诱导注意力稀疏性,可利用 block-sparse 加速
|
|
65
|
+
- **D8[v]**:KL 可融入 softmax kernel(FusedSoftmaxKL)
|
|
66
66
|
|
|
67
67
|
## 论文表述方式
|
|
68
68
|
"我们提出信息瓶颈注意力机制,通过将注意力建模为信息瓶颈优化问题,在最大化输出与目标的互信息的同时最小化输入信息的冗余传递,实现理论上最优的信息选择,并在实验中展现出更强的稀疏性和可解释性。"
|