math-skill 3.0.0 → 3.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en-US.md +9 -1
- package/README.md +9 -1
- package/agents/math-critic.en.md +27 -25
- package/agents/math-critic.md +27 -27
- package/commands/ask.en.md +10 -0
- package/commands/ask.md +1 -1
- package/design-patterns/attention/equivariant-attention.en.md +9 -9
- package/design-patterns/attention/equivariant-attention.md +9 -9
- package/design-patterns/attention/geometry-aware-attention.en.md +9 -9
- package/design-patterns/attention/geometry-aware-attention.md +9 -9
- package/design-patterns/attention/information-bottleneck-attention.en.md +9 -9
- package/design-patterns/attention/information-bottleneck-attention.md +9 -9
- package/design-patterns/attention/projection-attention.en.md +9 -9
- package/design-patterns/attention/projection-attention.md +9 -9
- package/design-patterns/attention/spectral-attention.en.md +9 -9
- package/design-patterns/attention/spectral-attention.md +9 -9
- package/design-patterns/compression/leverage-score-selection.en.md +1 -1
- package/design-patterns/compression/leverage-score-selection.md +1 -1
- package/design-patterns/compression/low-rank-kv-cache.en.md +1 -1
- package/design-patterns/compression/low-rank-kv-cache.md +1 -1
- package/design-patterns/compression/spectral-token-pruning.en.md +1 -1
- package/design-patterns/compression/spectral-token-pruning.md +1 -1
- package/design-patterns/compression/topology-preserving-compression.en.md +1 -1
- package/design-patterns/compression/topology-preserving-compression.md +1 -1
- package/design-patterns/loss/constraint-penalty.en.md +1 -1
- package/design-patterns/loss/constraint-penalty.md +1 -1
- package/design-patterns/loss/contrastive-loss.en.md +1 -1
- package/design-patterns/loss/contrastive-loss.md +1 -1
- package/design-patterns/loss/information-bottleneck-loss.en.md +1 -1
- package/design-patterns/loss/information-bottleneck-loss.md +1 -1
- package/design-patterns/loss/orthogonality-loss.en.md +1 -1
- package/design-patterns/loss/orthogonality-loss.md +1 -1
- package/design-patterns/loss/variational-loss.en.md +1 -1
- package/design-patterns/loss/variational-loss.md +1 -1
- package/design-patterns/representation/equivariant-split.en.md +1 -1
- package/design-patterns/representation/equivariant-split.md +1 -1
- package/design-patterns/representation/manifold-representation.en.md +1 -1
- package/design-patterns/representation/manifold-representation.md +1 -1
- package/design-patterns/representation/shared-private-decomposition.en.md +1 -1
- package/design-patterns/representation/shared-private-decomposition.md +1 -1
- package/design-patterns/representation/subspace-alignment.en.md +1 -1
- package/design-patterns/representation/subspace-alignment.md +1 -1
- package/design-patterns/routing/graph-routing.en.md +1 -1
- package/design-patterns/routing/graph-routing.md +1 -1
- package/design-patterns/routing/moe-routing.en.md +1 -1
- package/design-patterns/routing/moe-routing.md +1 -1
- package/design-patterns/routing/optimal-transport-routing.en.md +1 -1
- package/design-patterns/routing/optimal-transport-routing.md +1 -1
- package/design-patterns/routing/spectral-clustering-routing.en.md +1 -1
- package/design-patterns/routing/spectral-clustering-routing.md +1 -1
- package/knowledge-base/differential-geometry/metric-tensor.md +1 -1
- package/knowledge-base/information-geometry/fisher-metric.en.md +8 -8
- package/knowledge-base/information-geometry/fisher-metric.md +8 -8
- package/knowledge-base/information-geometry/natural-gradient.en.md +7 -7
- package/knowledge-base/information-geometry/natural-gradient.md +7 -7
- package/knowledge-base/overview.en.md +2 -2
- package/knowledge-base/overview.md +2 -2
- package/knowledge-base/probability/concentration-inequality.en.md +5 -5
- package/knowledge-base/probability/concentration-inequality.md +5 -5
- package/knowledge-base/probability/entropy.en.md +5 -5
- package/knowledge-base/probability/entropy.md +5 -5
- package/knowledge-base/probability/fisher-information.en.md +7 -7
- package/knowledge-base/probability/fisher-information.md +7 -7
- package/knowledge-base/probability/information-bottleneck.en.md +6 -6
- package/knowledge-base/probability/information-bottleneck.md +6 -6
- package/knowledge-base/probability/kl-divergence.en.md +6 -6
- package/knowledge-base/probability/kl-divergence.md +6 -6
- package/lenses/algorithmic.en.md +1 -1
- package/lenses/algorithmic.md +1 -1
- package/lenses/axiomatization.en.md +1 -1
- package/lenses/axiomatization.md +1 -1
- package/lenses/categorical.en.md +1 -1
- package/lenses/categorical.md +1 -1
- package/lenses/causal.en.md +1 -1
- package/lenses/causal.md +1 -1
- package/lenses/duality.en.md +1 -1
- package/lenses/duality.md +1 -1
- package/lenses/game.en.md +2 -2
- package/lenses/game.md +2 -2
- package/lenses/geometric.en.md +1 -1
- package/lenses/geometric.md +1 -1
- package/lenses/local-to-global.en.md +1 -1
- package/lenses/local-to-global.md +1 -1
- package/lenses/perturbation.en.md +1 -1
- package/lenses/perturbation.md +1 -1
- package/lenses/probabilistic.en.md +1 -1
- package/lenses/probabilistic.md +1 -1
- package/lenses/projection.en.md +1 -1
- package/lenses/projection.md +1 -1
- package/lenses/spectral.en.md +1 -1
- package/lenses/spectral.md +1 -1
- package/lenses/symmetry.en.md +1 -1
- package/lenses/symmetry.md +1 -1
- package/lenses/topological.en.md +1 -1
- package/lenses/topological.md +1 -1
- package/lenses/variational.en.md +1 -1
- package/lenses/variational.md +1 -1
- package/package.json +1 -1
- package/references/agentic-workflow.en.md +1 -1
- package/references/agentic-workflow.md +1 -1
- package/references/books/abstract-algebra.en.md +124 -0
- package/references/books/abstract-algebra.md +17 -17
- package/references/books/algebraic-geometry-rising-sea.en.md +171 -0
- package/references/books/algebraic-geometry-rising-sea.md +7 -7
- package/references/books/differential-geometry.en.md +140 -0
- package/references/books/differential-geometry.md +18 -18
- package/references/books/matrix-analysis.en.md +146 -0
- package/references/books/matrix-analysis.md +8 -8
- package/references/books/micro-lie-theory.en.md +116 -0
- package/references/books/micro-lie-theory.md +16 -16
- package/references/books/optimization-ml.en.md +163 -0
- package/references/books/optimization-ml.md +6 -7
- package/references/books/smooth-manifolds.en.md +105 -0
- package/references/books/smooth-manifolds.md +12 -12
- package/references/gpu-friendly-math.en.md +15 -2
- package/references/gpu-friendly-math.md +15 -2
- package/skills/math-research-activator/SKILL.en.md +27 -1
- package/skills/math-research-activator/SKILL.md +26 -3
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
# Optimization (Including ML Applications)
|
|
2
|
+
|
|
3
|
+
> **An Introduction to Optimization, With Applications to Machine Learning** (5th Edition, 2024)
|
|
4
|
+
> Edwin K. P. Chong, Wu-Sheng Lu, Stanislaw H. Zak -- John Wiley & Sons (Hardback ISBN 9781119877639)
|
|
5
|
+
>
|
|
6
|
+
> This file is an "activation reference": synthesized distillation, engineering-oriented, not a verbatim excerpt.
|
|
7
|
+
> Goal -- extract the structures from this classic optimization textbook that **can be applied to algorithms / GPU / training Infra**.
|
|
8
|
+
|
|
9
|
+
## Overview
|
|
10
|
+
|
|
11
|
+
The fifth edition of a classic continuous optimization textbook. Compared to earlier editions, the largest addition is a completely new **Part V "Optimization in Machine Learning"**, which fully integrates first-order/second-order optimality, duality, KKT, and convergence theory with training practices such as SGD/SVM/PCA. The book has 31 chapters organized into five parts.
|
|
12
|
+
|
|
13
|
+
Actual chapter map:
|
|
14
|
+
|
|
15
|
+
- **Part I -- Mathematical Review (Ch 1-5)**
|
|
16
|
+
- Ch 1 Proof methods and notation; Ch 2 Vector spaces and matrices (rank, inner products and norms).
|
|
17
|
+
- Ch 3 Transformations: eigenvalues/eigenvectors, orthogonal projections, quadratic forms S3.4, matrix norms S3.5.
|
|
18
|
+
- Ch 4 Geometry: hyperplanes, **convex sets S4.3**, polytopes and polyhedra.
|
|
19
|
+
- Ch 5 Calculus: derivative matrices S5.3, **level sets and gradients S5.5**, Taylor series S5.6.
|
|
20
|
+
- **Part II -- Unconstrained Optimization (Ch 6-14)**
|
|
21
|
+
- Ch 6 First-order/second-order necessary and sufficient conditions for local minima.
|
|
22
|
+
- Ch 7 One-dimensional search: golden section, Fibonacci, bisection, Newton, secant, line search.
|
|
23
|
+
- **Ch 8 Gradient methods** (steepest descent S8.2, convergence analysis S8.3).
|
|
24
|
+
- **Ch 9 Newton's method** (Levenberg-Marquardt S9.3, nonlinear least squares).
|
|
25
|
+
- Ch 10 Conjugate direction/conjugate gradient methods; **Ch 11 Quasi-Newton methods** (inverse Hessian approximation, DFP S11.4, **BFGS S11.5**).
|
|
26
|
+
- Ch 12 Solving linear equations (least squares, RLS, Kaczmarz, minimum-norm solutions).
|
|
27
|
+
- **Ch 13 Neural networks and backpropagation** (single-neuron training S13.2, Backprop S13.3).
|
|
28
|
+
- Ch 14 Global search: Nelder-Mead, simulated annealing, PSO, genetic algorithms.
|
|
29
|
+
- **Part III -- Linear Programming (Ch 15-19)**
|
|
30
|
+
- Ch 15 LP fundamentals and geometry; Ch 16 Simplex method.
|
|
31
|
+
- **Ch 17 LP duality** (dual LP, matrix games).
|
|
32
|
+
- Ch 18 Non-simplex/interior-point methods (Khachiyan, affine scaling, Karmarkar); Ch 19 Integer programming.
|
|
33
|
+
- **Part IV -- Nonlinear Constrained Optimization (Ch 20-25)**
|
|
34
|
+
- Ch 20 Equality constraints: tangent/normal spaces S20.3, **Lagrange conditions S20.4**, second-order conditions.
|
|
35
|
+
- **Ch 21 Inequality constraints and KKT** (KKT S21.1, second-order conditions S21.2).
|
|
36
|
+
- **Ch 22 Convex optimization** (convex functions S22.2, convex problems S22.3, SDP/LMI S22.4).
|
|
37
|
+
- **Ch 23 Lagrangian duality** (weak/strong duality, duality gap S23.4.6, Slater S23.6.3, saddle points).
|
|
38
|
+
- Ch 24 Constrained optimization algorithms (projection S24.2, **projected gradient S24.3**, Armijo S24.4.4, augmented Lagrangian S24.5, penalty methods S24.6).
|
|
39
|
+
- Ch 25 Multi-objective / robust LP (Pareto, uncertainty).
|
|
40
|
+
- **Part V -- Optimization in ML (Ch 26-31)**
|
|
41
|
+
- Ch 26 Feature engineering, PCA, SVD, linear autoencoders.
|
|
42
|
+
- **Ch 27 SGD algorithms** (SGD S27.1, variance reduction **SVRG S27.2**, **distributed SVRG and communication/computation tradeoffs S27.3**).
|
|
43
|
+
- Ch 28 Linear regression (regularization S28.3, cross-validation); Ch 29 Logistic regression / Softmax.
|
|
44
|
+
- **Ch 30 SVM** (hinge loss, hard/soft margin); **Ch 31 Kernel methods and K-Means**.
|
|
45
|
+
|
|
46
|
+
## Core Structures Transferable to AI/Infra
|
|
47
|
+
|
|
48
|
+
- **First-order optimality + gradient geometry (Ch 6, S5.5)**
|
|
49
|
+
- Gradient is perpendicular to level sets and is the steepest ascent direction -- the axiomatic root of all training optimizers.
|
|
50
|
+
- `nabla f = 0` is the stopping criterion; saddle points / non-convexity are the central pain points of modern deep learning.
|
|
51
|
+
- **Convergence analysis framework (Ch 8 S8.3)**
|
|
52
|
+
- Uses the condition number kappa = lambda_max/lambda_min to characterize the convergence rate of steepest descent.
|
|
53
|
+
- Explains "why ill-conditioned networks are hard to train, why normalization / preconditioning is needed."
|
|
54
|
+
- **Second-order curvature and damping (Ch 9)**
|
|
55
|
+
- Newton's method uses the Hessian to provide affine-invariant step sizes and directions.
|
|
56
|
+
- Levenberg-Marquardt's `(H+mu I)` damping = trust-region idea, the ancestor of Adam's `epsilon` and second-order optimizers.
|
|
57
|
+
- **Quasi-Newton low-rank updates (Ch 11)**
|
|
58
|
+
- BFGS/DFP uses only gradient differences, maintaining an inverse Hessian approximation via rank-1/rank-2 updates.
|
|
59
|
+
- Direct intellectual ancestor of L-BFGS, K-FAC, and Shampoo.
|
|
60
|
+
- **Matrix-free second-order methods (Ch 10)**
|
|
61
|
+
- Conjugate gradient requires only Hessian-vector products without storing the full Hessian; corresponds to autodiff HVP (Hessian-free).
|
|
62
|
+
- **Constraint geometry (Ch 20-21, 24)**
|
|
63
|
+
- Tangent/normal spaces -> Lagrange/KKT -> projected gradient + penalty / augmented Lagrangian.
|
|
64
|
+
- Forms a complete toolkit for "training with constraints."
|
|
65
|
+
- **Duality and saddle points (Ch 17, 23)**
|
|
66
|
+
- Strong duality <=> minimax = maximin; KKT, complementary slackness, Slater conditions.
|
|
67
|
+
- Min-max training (GAN / adversarial robustness) and SVM duality both fall here.
|
|
68
|
+
- **Stochastic and distributed methods (Ch 27)**
|
|
69
|
+
- SGD's unbiased gradient estimation, SVRG variance reduction.
|
|
70
|
+
- **Explicit "communication vs. computation" tradeoff in distributed settings** -- the theoretical root of data-parallel training.
|
|
71
|
+
- **Low-rank / spectral methods (Ch 26, S3.4)**
|
|
72
|
+
- SVD/PCA/linear autoencoders, quadratic forms and spectra -- the mathematical foundation of LoRA, KV compression, and spectral normalization.
|
|
73
|
+
|
|
74
|
+
## Problem Types Suited for Activation
|
|
75
|
+
|
|
76
|
+
- Selecting / designing optimizers, or explaining training dynamics (why divergence, oscillation, or stagnation at saddle points occurs).
|
|
77
|
+
- Diagnosing ill-conditioning and slow convergence: quantifying "hard to train" via the condition number kappa.
|
|
78
|
+
- Training with constraints: weight-norm balls, spectral norms, safety / budget constraints, requiring projection or penalty methods.
|
|
79
|
+
- Min-max / adversarial / dual perspectives: replacing a hard primal with an easier dual (e.g., kernel Gram matrix).
|
|
80
|
+
- Communication-computation tradeoffs, gradient compression, and variance reduction in distributed training.
|
|
81
|
+
- Scenarios requiring second-order information but unable to compute the full Hessian (curvature-adaptive preconditioning).
|
|
82
|
+
|
|
83
|
+
## Possible Algorithmic Inspirations
|
|
84
|
+
|
|
85
|
+
- **Adaptive optimizers**
|
|
86
|
+
- Levenberg-Marquardt's `(H+mu I)` damping -> Adam's `epsilon`, trust regions, adaptive learning rates.
|
|
87
|
+
- Making "curvature / damping" explicit rather than staying with purely heuristic hyperparameter tuning.
|
|
88
|
+
- **Feasible second-order for large models**
|
|
89
|
+
- L-BFGS: Store only the most recent m gradient pairs (low-rank history, memory O(md)).
|
|
90
|
+
- Hessian-free: Autodiff HVP + CG (matrix-free, no need to materialize the Hessian).
|
|
91
|
+
- K-FAC / Shampoo: Approximate the Hessian as **block-diagonal / Kronecker factors**, reducing each block to a small GEMM.
|
|
92
|
+
- **Efficient solving via the dual perspective**
|
|
93
|
+
- SVM converts a high-dimensional primal into a dual that depends only on the kernel Gram matrix (Ch 30-31), a big win when sample count << dimensionality.
|
|
94
|
+
- Dual decomposition of constrained problems is naturally parallelizable.
|
|
95
|
+
- **Constrained training**
|
|
96
|
+
- Projected gradient (S24.3) for norm-ball / spectral-norm constraints (weight clipping, spectral normalization).
|
|
97
|
+
- Augmented Lagrangian / penalty methods (S24.5-S24.6) soften hard constraints into differentiable regularization.
|
|
98
|
+
- **Variance reduction / large batch**
|
|
99
|
+
- SVRG's (S27.2) control variate idea -> gradient accumulation, stability analysis of large-batch training.
|
|
100
|
+
- **Distributed training**
|
|
101
|
+
- S27.3 "Communication vs. Computation" directly corresponds to all-reduce and computation overlap, gradient compression, federated / data security.
|
|
102
|
+
|
|
103
|
+
## GPU Friendliness Warning
|
|
104
|
+
|
|
105
|
+
> The following item-by-item evaluation uses the **eight-dimension scorecard** from `../gpu-friendly-math.md` (read that first).
|
|
106
|
+
> Conclusion: The book's first-order / stochastic / low-rank content is naturally GPU-friendly; second-order methods and search-based methods require heavy adaptation or elimination.
|
|
107
|
+
|
|
108
|
+
- **Full Hessian in second-order methods (Ch 9, 11)**
|
|
109
|
+
- Violates **Dimension 4 (Memory)**: Dense Hessian is O(d^2), impossible to materialize for hundreds of millions of parameters.
|
|
110
|
+
- Violates **Dimension 2 (GEMM-mappability)**: Inversion / factorization is not batched-GEMM friendly.
|
|
111
|
+
- Violates **Dimension 5 (Low-precision stability)**: Ill-conditioned inversion causes catastrophic cancellation under bf16/fp16.
|
|
112
|
+
- Adaptation: Matrix-free HVP + CG, L-BFGS low-rank, K-FAC block-diagonal -> restores **Dimension 2/4** friendliness.
|
|
113
|
+
- **Line search (Ch 7 golden section / Armijo S24.4.4)**
|
|
114
|
+
- Violates **Dimension 1 (Tensorization)**: Essentially a scalar loop + data-dependent branching (step-by-step comparison of function values).
|
|
115
|
+
- Violates **Dimension 8 (Operator fusion)**: Frequent small kernels, control-flow divergence.
|
|
116
|
+
- Adaptation: Large models almost universally use scheduled step sizes (warmup + cosine) instead of step-by-step line search.
|
|
117
|
+
- **First-order / SGD (Ch 8, 27)**
|
|
118
|
+
- **Dimension 1/2 friendly**: Gradients = tensor algebra, expressible as GEMM.
|
|
119
|
+
- **Dimension 6 (Parallelism and communication)** is good: Data-parallel all-reduce can overlap with backprop -- the fundamental reason this is the default approach for large models.
|
|
120
|
+
- **Global search (Ch 14: GA/SA/PSO)**
|
|
121
|
+
- Violates **Dimension 1**: Non-differentiable, discrete stochastic search, blocks end-to-end gradient training.
|
|
122
|
+
- **Dimension 6** is also poor: Inter-population dependencies, hard to parallelize. Only suitable for black-box hyperparameter search, not for the training inner loop.
|
|
123
|
+
- **LP simplex / interior-point methods (Ch 16, 18)**
|
|
124
|
+
- Simplex row operations are acceptable; interior-point methods require solving a linear system at each step, **Dimension 2/6** limited by factorization and communication.
|
|
125
|
+
- Suitable for medium-scale constrained subproblems, not for placement inside every training step.
|
|
126
|
+
- **Projections (S24.2)**
|
|
127
|
+
- Norm-ball / box projections are cheap elementwise operations (**Dimension 1/8** friendly).
|
|
128
|
+
- General polyhedral projections require solving a QP (**Dimension 2** degrades).
|
|
129
|
+
- **Convergence depends on condition number kappa**
|
|
130
|
+
- Large kappa -> **Dimension 5** is further amplified at low precision (ill-conditioned gradients).
|
|
131
|
+
- Must pair with normalization / preconditioning; otherwise bf16 training diverges.
|
|
132
|
+
|
|
133
|
+
## Which Thinking Lens to Invoke
|
|
134
|
+
|
|
135
|
+
- **variational** (primary): The complete objective-constraint-optimality-convergence framework; this book is its core source; translate real-world tasks (classification / regression / constraints) into solvable optimization problems.
|
|
136
|
+
- **algorithmic**: Convergence properties, complexity, step size / stopping criteria for iterative algorithms.
|
|
137
|
+
- **duality**: Duality, kernel methods, variable substitution, SVD/PCA -- "equivalence transformations to simplify problems."
|
|
138
|
+
- **probabilistic**: SGD stochastic gradients, SVRG variance reduction, cross-validation / regularization.
|
|
139
|
+
|
|
140
|
+
## Anti-patterns
|
|
141
|
+
|
|
142
|
+
- Applying **full-Hessian Newton** directly to large models -- O(d^2) memory and inversion blow up immediately (must use matrix-free / low-rank / block-diagonal).
|
|
143
|
+
- Using **line search** to tune step sizes at every step of a large model -- scalar serial, control-flow divergence, burns through compute; should switch to scheduled learning rates.
|
|
144
|
+
- Using **GA/SA/PSO to train neural networks** -- non-differentiable, cannot parallelize, extremely sample-inefficient; limited to black-box hyperparameter search only.
|
|
145
|
+
- Discussing convergence while **ignoring the condition number kappa** -- expecting low-precision convergence without normalization / preconditioning is wishful thinking.
|
|
146
|
+
- Forcing **LP simplex thinking** onto continuously differentiable problems.
|
|
147
|
+
- Treating **duality / KKT as universal** -- non-convex problems typically lack strong duality and have duality gaps (S23.4.6); KKT is only a necessary condition.
|
|
148
|
+
- Treating this "activation reference" as a source of rigorous proofs -- details and theorem conditions must be verified against the original book.
|
|
149
|
+
|
|
150
|
+
## Deep-dive Entry
|
|
151
|
+
|
|
152
|
+
> **Bibliographic information**: Edwin K. P. Chong, Wu-Sheng Lu, Stanislaw H. Zak, *An Introduction to Optimization, With Applications to Machine Learning*, 5th Edition, John Wiley & Sons, 2024. ISBN 978-1-119-87763-9.
|
|
153
|
+
>
|
|
154
|
+
> **Activation method**: Place `An Introduction to Optimization With Applications to Machine Learning.pdf` in the `math_book/` folder at the project root; the Agent can then automatically search the original text. The PDF is not distributed via npm/git (copyright reasons) and must be obtained separately.
|
|
155
|
+
|
|
156
|
+
Full-fidelity lookup = have the Agent directly search the local PDF
|
|
157
|
+
`math_book/An Introduction to Optimization With Applications to Machine Learning.pdf`, locating by actual chapter numbers:
|
|
158
|
+
|
|
159
|
+
- **Ch 8 Gradient Methods** (S8.3 convergence analysis, condition numbers) + **Ch 11 Quasi-Newton Methods** (S11.5 BFGS) -- Optimizers and feasible second-order methods.
|
|
160
|
+
- **Ch 22 Convex Optimization Problems** (S22.2 convex functions, S22.4 SDP/LMI) -- Convexity criteria and semidefinite programming.
|
|
161
|
+
- **Ch 23 Lagrangian Duality** (S23.5 strong duality, S23.6.3 Slater conditions) + **Ch 21 KKT Conditions** -- Duality / KKT / saddle points.
|
|
162
|
+
- **Ch 24 Algorithms for Constrained Optimization** (S24.3 projected gradient, S24.5 augmented Lagrangian, S24.6 penalty methods) -- Constrained training algorithms.
|
|
163
|
+
- **Ch 27 Stochastic Gradient Descent Algorithms** (S27.1 SGD, S27.2 SVRG, S27.3 distributed and communication/computation) -- Large-scale training core.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
#
|
|
1
|
+
# 最优化(含 ML 应用)/ Optimization with ML
|
|
2
2
|
|
|
3
3
|
> **An Introduction to Optimization, With Applications to Machine Learning** (5th Edition, 2024)
|
|
4
4
|
> Edwin K. P. Chong, Wu-Sheng Lu, Stanisław H. Żak — John Wiley & Sons(Hardback ISBN 9781119877639)
|
|
@@ -132,11 +132,10 @@
|
|
|
132
132
|
|
|
133
133
|
## 该调用哪个思想透镜
|
|
134
134
|
|
|
135
|
-
- **
|
|
136
|
-
- **algorithmic
|
|
137
|
-
- **
|
|
138
|
-
- **
|
|
139
|
-
- **probability-statistics**:SGD 随机梯度、SVRG 方差缩减、交叉验证 / 正则化。
|
|
135
|
+
- **variational**(主):目标-约束-最优性-收敛的完整框架,本书是其核心出处;把现实任务(分类 / 回归 / 约束)翻译成可解的优化问题。
|
|
136
|
+
- **algorithmic**:迭代算法的收敛性、复杂度、步长 / 停机判据。
|
|
137
|
+
- **duality**:对偶、核技巧、变量替换、SVD/PCA——"等价转换简化问题"。
|
|
138
|
+
- **probabilistic**:SGD 随机梯度、SVRG 方差缩减、交叉验证 / 正则化。
|
|
140
139
|
|
|
141
140
|
## 反模式
|
|
142
141
|
|
|
@@ -150,7 +149,7 @@
|
|
|
150
149
|
|
|
151
150
|
## 深挖入口
|
|
152
151
|
|
|
153
|
-
>
|
|
152
|
+
> **书目信息**:Edwin K. P. Chong, Wu-Sheng Lu, Stanisław H. Żak, *An Introduction to Optimization, With Applications to Machine Learning*, 5th Edition, John Wiley & Sons, 2024. ISBN 978-1-119-87763-9.
|
|
154
153
|
>
|
|
155
154
|
> **启用方式**:将 `An Introduction to Optimization With Applications to Machine Learning.pdf` 放入项目根目录的 `math_book/` 文件夹,Agent 即可自动搜索原文。PDF 不随 npm/git 分发(版权原因),需自行获取。
|
|
156
155
|
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# Smooth Manifolds
|
|
2
|
+
|
|
3
|
+
> **Book**: John M. Lee, *Introduction to Smooth Manifolds*, 2nd Edition. Graduate Texts in Mathematics 218, Springer, 2013. ISBN 978-1-4419-9981-8 / DOI 10.1007/978-1-4419-9982-5. MSC 53-01 / 58-01 / 57-01.
|
|
4
|
+
> **Positioning**: Equipping objects that "locally look like Euclidean space but can bend globally" (manifolds) with calculus (tangent spaces, vector fields, differential forms, flows, Lie derivatives) -- the mathematical mother of **manifold optimization, latent-space geometry, and differentiable structures**.
|
|
5
|
+
|
|
6
|
+
## Overview
|
|
7
|
+
|
|
8
|
+
Smooth manifold = a space that can be locally linearized by coordinate charts, with charts glued together by smooth transition maps. The main thread of the book: **first transplant Euclidean calculus onto curved spaces, then study geometric and topological invariants on them**. For AI, the most valuable part is the "differentiable machinery" in the first half -- tangent/cotangent spaces, vector fields, flows, Riemannian metrics.
|
|
9
|
+
|
|
10
|
+
Actual chapter map (2nd ed., chapter numbers match the book):
|
|
11
|
+
|
|
12
|
+
- **Ch 1-2 Smooth Manifolds / Smooth Maps**: topological manifolds, smooth structures (atlases), smooth maps, partitions of unity. -> The language of local linearization + global patching.
|
|
13
|
+
- **Ch 3 Tangent Vectors**: tangent space T_pM, differential / pushforward df_p, tangent bundle TM. -> **The core of local linearization**, the geometric prototype of backpropagation.
|
|
14
|
+
- **Ch 4-5 Submersions, Immersions, Embeddings / Submanifolds**: constant rank theorem, embeddings, regular level sets -> submanifolds. -> Constraint sets = submanifolds.
|
|
15
|
+
- **Ch 6 Sard's Theorem**: critical values have measure zero, Whitney embedding theorem (n-dimensional manifolds embed in R^{2n}). -> Embedding dimensions / manifold hypothesis.
|
|
16
|
+
- **Ch 7 Lie Groups**: both group and manifold (SO(n), U(n), GL(n), Stiefel...), Lie algebra = tangent space at the identity. -> Orthogonal/unitary weight constraints, equivariance.
|
|
17
|
+
- **Ch 8-9 Vector Fields / Integral Curves and Flows**: vector fields, integral curves, flows (one-parameter diffeomorphism groups), Lie derivatives and Lie brackets [X,Y]. -> **The mother structure of Neural ODE / diffusion / continuous normalizing flows**.
|
|
18
|
+
- **Ch 10-12 Vector Bundles / Cotangent Bundle / Tensors**: bundles, covector fields (1-forms), pullbacks, tensors. -> The true nature of gradients is covectors.
|
|
19
|
+
- **Ch 13 Riemannian Metrics**: inner product at each point, length/distance/volume, tangent-cotangent isomorphism (musical sharp/flat, raising/lowering indices). -> **The metric source for natural gradients / Riemannian optimization**.
|
|
20
|
+
- **Ch 14-16 Differential Forms / Orientations / Integration**: k-forms, wedge products, exterior derivative d (d^2=0), orientations, volume forms, integration on manifolds and change of variables. -> The log-det-Jacobian in normalizing flows = pullback of volume forms.
|
|
21
|
+
- **Ch 17-18 De Rham Cohomology / de Rham Theorem**: closed forms modulo exact forms = topological invariants read from differential data. -> Global obstructions / cohomological regularization.
|
|
22
|
+
- **Ch 19-22 Distributions & Foliations / Exponential Map / Quotient Manifolds / Symplectic Manifolds**: integrable distributions (Frobenius), exponential map (retraction prototype), quotient manifolds (Grassmannian etc.), symplectic forms and Hamiltonian flows. -> Retractions, quotient-space constraints, symplectic integrators / HMC.
|
|
23
|
+
|
|
24
|
+
**Author's stated boundary (preface)**: the book stops at "building the tools," **deliberately omitting** connections, geodesics, curvature, fiber bundles, and Hodge theory -- these are covered in Lee's sequel *Riemannian Manifolds*. So if a problem truly requires the deep geometry of curvature/parallel transport, this book only provides the entry point via metrics and the exponential map; one must continue with a Riemannian geometry text.
|
|
25
|
+
|
|
26
|
+
## Core Structures Transferable to AI/Infra
|
|
27
|
+
|
|
28
|
+
- **Tangent space = local linearization of parameter/latent space**. `df_p: T_pM -> T_{f(p)}N` is the Jacobian / pushforward; backpropagation = pushforward along composed maps (the geometric version of the chain rule). All first-order methods live in the tangent space.
|
|
29
|
+
- **Gradients are covectors, not vectors**. Autodiff produces 1-forms (elements of the cotangent space); to obtain a descent direction (tangent vector) one must use the **metric to raise indices** (sharp). Euclidean metric -> ordinary gradient; Fisher metric -> natural gradient. **This is the manifold-level root cause of natural gradients / mirror descent**.
|
|
30
|
+
- **Constraint sets = submanifolds**. Regular level set theorem: when g is a submersion, the solution set `g(x)=c` is a smooth submanifold; constrained optimization = unconstrained optimization on a submanifold.
|
|
31
|
+
- **Lie groups = differentiable symmetry groups**. SO(n)/U(n)/Stiefel/Grassmannian are all manifolds; their Lie algebras (e.g., skew-symmetric matrices so(n)) are linear spaces, mapped back to the group via `exp` -> **reparameterize "constrained weights" as "unconstrained Lie algebra + exp"**.
|
|
32
|
+
- **Flows = time-parameterized families of diffeomorphisms**. Learning a vector field + integrating along it = Neural ODE / continuous normalizing flows / diffusion sampling. Flow invertibility and volume preservation directly correspond to model properties.
|
|
33
|
+
- **Riemannian metric = designable/learnable "local geometry"**. It determines distances, angles, volumes, and orthogonality relations; changing the metric changes optimization trajectories and sampling measures.
|
|
34
|
+
- **Differential forms + volume forms = the language of change of variables**. The `log|det J|` term in normalizing flows is precisely the pullback of a volume form under a map; choosing the right structure (triangular/coupling Jacobians) makes it cheap.
|
|
35
|
+
|
|
36
|
+
## Problem Types Suited for Activation
|
|
37
|
+
|
|
38
|
+
- Parameters should satisfy **geometric constraints**: orthogonality, unit norm, unit determinant, SPD, low-rank manifolds, hyperbolic/spherical latent spaces.
|
|
39
|
+
- Optimization is more natural on **curved spaces**: subspace learning on Stiefel/Grassmannian, rotation/pose estimation, hyperspherical representations.
|
|
40
|
+
- Need **structure-preserving dynamics**: invertible generative models, volume-preserving flows, Hamiltonian systems, energy-conserving long-horizon simulations.
|
|
41
|
+
- **Latent space geometry**: interpolation, geodesics, metric learning, clustering/nearest-neighbor on manifolds.
|
|
42
|
+
- Need to upgrade from correlations to **topological invariants**: detecting "holes" in latent spaces, global obstructions, cohomology-based consistency regularization.
|
|
43
|
+
|
|
44
|
+
## Possible Algorithmic Inspirations
|
|
45
|
+
|
|
46
|
+
- **Riemannian/manifold optimizers**: transplanting Adam/SGD onto Stiefel, Grassmannian, SPD, hyperbolic spaces -- projecting gradients to the tangent space + retraction back to the manifold.
|
|
47
|
+
- **Orthogonal/Stiefel constrained weights**: using Cayley transforms or QR-retractions to maintain `W^T W = I`, mitigating gradient explosion/vanishing in RNNs/deep networks; or using so(n) Lie algebra + matrix-exp to reparameterize rotations.
|
|
48
|
+
- **Geodesic interpolation**: using closed-form geodesics for interpolation and mixing in spherical/hyperbolic/SPD latent spaces, replacing Euclidean linear interpolation.
|
|
49
|
+
- **Normalization on manifolds**: interpreting LayerNorm/feature normalization as projection onto spheres/unit manifolds; hyperspherical softmax and spectral normalization are instances of this.
|
|
50
|
+
- **Neural ODE / continuous normalizing flows / diffusion**: learning a vector field X_theta, solving via flows; structured Jacobians make `log-det` cheap.
|
|
51
|
+
- **Symplectic integrators / HMC**: using leapfrog-style symplectic, volume-preserving explicit updates for sampling and "optimization with momentum," stable over long horizons.
|
|
52
|
+
- **Equivariant networks**: using Lie group actions + quotient manifolds to bake symmetries into the architecture (geometric deep learning).
|
|
53
|
+
|
|
54
|
+
## GPU Friendliness Warning
|
|
55
|
+
|
|
56
|
+
> The sole authority for the acceptance gate: the **eight dimensions** in `../gpu-friendly-math.md`. The success or failure of manifold methods almost entirely hinges on one point: **whether the retraction / exponential map can be tensorized and expressed as GEMM, or must be solved iteratively.**
|
|
57
|
+
|
|
58
|
+
Dimension-by-dimension comparison:
|
|
59
|
+
|
|
60
|
+
- **D1-D2 Tensorization / GEMM**: tangent-space operations (pushforward/pullback, Jacobian-vector products, projecting gradients to the tangent space) are **naturally batched GEMM** [v] -- backpropagation itself is pushforward, which is extremely GPU-friendly. **However**, retractions/exp typically require QR, eigendecomposition, matrix exponentials, or small-matrix inverses: QR/eig are **not clean GEMM**, they are decompositions with serial dependencies (cuSOLVER batched small matrices are acceptable, but large matrices are O(n^3) with poor parallelism) -> **reformable** rather than natively friendly.
|
|
61
|
+
- **D3 Complexity**: geodesic distances, parallel transport, general `log|det J|` are all O(n^3) or worse. **Fix**: restrict to manifolds with closed-form geodesics (spheres/hyperbolic/SO(3)); use triangular/coupling layers in normalizing flows so that log-det reduces to a diagonal sum (O(n)).
|
|
62
|
+
- **D5 Low-precision**: [~] **biggest pitfall**. Matrix `exp / log / sqrt`, eigendecompositions, and the affine-invariant metric for SPD are **catastrophically unstable** under bf16/fp16, often silently requiring fp32/fp64. Manifold primitives frequently "appear to run but have numerically diverged long ago."
|
|
63
|
+
- **D6 Parallelism & communication**: the squaring chain in scaling-and-squaring, ODE integration steps, Householder/QR all have **serial recurrences**, making cross-SM/device overlap difficult. Positive counter-example: explicit symplectic integrators (leapfrog) are highly parallel [v].
|
|
64
|
+
- **D4/D7/D8 Memory / Sparsity / Fusion**: if Lie algebra/rotation parameterizations are restricted to **small matrices or block-diagonal** structures (e.g., per-head rotations, SO(3) Rodrigues closed form), they can be fused into kernels and use Tensor Cores; large dense manifold operators require materializing large intermediate tensors.
|
|
65
|
+
|
|
66
|
+
**Conclusions and reform strategies (echoing the toolbox in gpu-friendly-math.md)**:
|
|
67
|
+
|
|
68
|
+
1. **Prefer manifolds with closed-form retractions** (spheres, Stiefel-QR, SO(3), hyperbolic).
|
|
69
|
+
2. **Soften whenever possible**: replace hard constraints with pure-GEMM regularization terms (e.g., `lambda ||W^T W - I||^2` instead of strict orthogonal manifold) -- for most training this suffices.
|
|
70
|
+
3. **Small matrices / blocking**: restrict exp/Cayley/QR to small blocks or per-head, batched as batched GEMM.
|
|
71
|
+
4. **Structured Jacobians**: insist on triangular/coupling structures in normalizing flows, avoiding general LU for computing determinants.
|
|
72
|
+
5. **Precision guardrails**: for all matrix exp/log/eig, enforce fp32 accumulation and numerical stabilization (log-sum-exp style).
|
|
73
|
+
|
|
74
|
+
## Which Design Lens to Invoke
|
|
75
|
+
|
|
76
|
+
- **variational (primary)**: constrained optimization, Riemannian/manifold optimization, retraction selection.
|
|
77
|
+
- **symmetry**: Lie groups, equivariance, quotient manifolds, invariants under group actions.
|
|
78
|
+
- **duality**: coordinate chart transitions, pushforward/pullback, change of variables in normalizing flows, diffeomorphisms.
|
|
79
|
+
- **topological**: de Rham cohomology, global obstructions, "holes" and connectivity of latent spaces.
|
|
80
|
+
- **categorical**: extracting the "locally linear + smoothly patched" manifold skeleton from high-dimensional messy ambient data (the manifold hypothesis).
|
|
81
|
+
|
|
82
|
+
## Anti-patterns
|
|
83
|
+
|
|
84
|
+
- **Confusing ML "tensors" (arrays) with mathematical tensors (multilinear, with covariant/contravariant transformation laws)**, mistakenly assuming coordinate-independent invariance comes for free.
|
|
85
|
+
- **Putting exp / geodesics / matrix-log into bf16 hot training loops**: both slow (serial decomposition) and silently divergent. Ask first: "is there a closed-form retraction / can it be softened?"
|
|
86
|
+
- **Not softening when you should**: using strict manifold constraints for marginal gains while paying the throughput and stability cost of QR/eig; for many tasks, an orthogonality regularization term suffices.
|
|
87
|
+
- **Confusing gradients (covectors) with descent directions (vectors)**: forgetting the metric and treating raw autodiff output directly as natural gradients.
|
|
88
|
+
- **The illusion of a single global coordinate chart**: covering an entire manifold with one global parameterization inevitably introduces singularities (e.g., Euler angle gimbal lock); manifolds inherently require atlases / redundant parameterizations.
|
|
89
|
+
- **Manifold hypothesis overuse**: deploying the full Riemannian machinery when the parameter space is already flat Euclidean -- pure over-engineering (violates simplicity-first).
|
|
90
|
+
|
|
91
|
+
## Deep Dive Entry
|
|
92
|
+
|
|
93
|
+
> **Bibliographic info**: John M. Lee, *Introduction to Smooth Manifolds*, 2nd Edition, Graduate Texts in Mathematics 218, Springer, 2013. ISBN 978-1-4419-9981-8.
|
|
94
|
+
>
|
|
95
|
+
> **Activation method**: Place `Introduction to Smooth Manifolds.pdf` in the `math_book/` folder at the project root; the Agent can then automatically search the original text. The PDF is not distributed via npm/git (copyright reasons); obtain it independently.
|
|
96
|
+
|
|
97
|
+
> **Full-fidelity lookup**: when original definitions/theorems/proofs are needed, have the Agent **automatically search the local PDF** `math_book/Introduction to Smooth Manifolds.pdf` (locate by chapter number / keywords, do not rely on memory to restate). The following are actual chapter numbers (2nd ed.):
|
|
98
|
+
|
|
99
|
+
- **Ch 3 Tangent Vectors** -- tangent spaces, differentials/pushforwards, tangent bundles: local linearization and the geometric prototype of backpropagation.
|
|
100
|
+
- **Ch 11 The Cotangent Bundle** -- covector fields (1-forms), `df` as a covector, pullbacks: the true nature of gradients = covectors.
|
|
101
|
+
- **Ch 13 Riemannian Metrics** -- metrics, tangent-cotangent isomorphism (sharp/flat), distances: the root of natural gradients / Riemannian optimization.
|
|
102
|
+
- **Ch 9 Integral Curves and Flows** -- flows, integral curves, Lie derivatives/Lie brackets: Neural ODE / diffusion / structure-preserving dynamics.
|
|
103
|
+
- **Ch 20 The Exponential Map** -- the exponential map: the retraction prototype, and also the main bottleneck for GPU feasibility.
|
|
104
|
+
|
|
105
|
+
(Extensions: Ch 7 Lie Groups -> orthogonal/unitary constraints and equivariance; Ch 14 Differential Forms -> volume forms and log-det-Jacobian; Ch 22 Symplectic Manifolds -> symplectic integrators / HMC.)
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
#
|
|
1
|
+
# 光滑流形 / Smooth Manifolds
|
|
2
2
|
|
|
3
3
|
> **书目**:John M. Lee, *Introduction to Smooth Manifolds*, 2nd Edition. Graduate Texts in Mathematics 218, Springer, 2013. ISBN 978-1-4419-9981-8 / DOI 10.1007/978-1-4419-9982-5. MSC 53-01 / 58-01 / 57-01.
|
|
4
4
|
> **定位**:把"局部像欧氏空间、整体可弯曲"的对象(manifold)配上微积分(切空间、向量场、微分形式、流、李导数),是 **流形优化、隐空间几何、可微结构** 的数学母体。
|
|
@@ -16,7 +16,7 @@
|
|
|
16
16
|
- **Ch 7 Lie Groups**:既是群又是流形(SO(n), U(n), GL(n), Stiefel…),李代数 = 单位元处切空间。→ 正交/酉权重约束、等变。
|
|
17
17
|
- **Ch 8–9 Vector Fields / Integral Curves and Flows**:向量场、积分曲线、流(flow,单参数微分同胚群)、李导数与李括号 [X,Y]。→ **Neural ODE / 扩散 / 连续归一化流** 的母结构。
|
|
18
18
|
- **Ch 10–12 Vector Bundles / Cotangent Bundle / Tensors**:丛、余向量场(1-form)、拉回(pullback)、张量。→ 梯度的真身是余向量。
|
|
19
|
-
- **Ch 13 Riemannian Metrics**:每点内积、长度/距离/体积、切-余切同构(musical
|
|
19
|
+
- **Ch 13 Riemannian Metrics**:每点内积、长度/距离/体积、切-余切同构(musical (sharp)/(flat),升降指标)。→ **自然梯度 / 黎曼优化的度量来源**。
|
|
20
20
|
- **Ch 14–16 Differential Forms / Orientations / Integration**:k-形式、楔积、外微分 d(d²=0)、定向、体积形式、流形上积分与变量替换。→ 归一化流的 log-det-Jacobian = 体积形式拉回。
|
|
21
21
|
- **Ch 17–18 De Rham Cohomology / de Rham Theorem**:闭形式模去恰当形式 = 从微分数据读出的拓扑不变量。→ 全局障碍 / 上同调正则。
|
|
22
22
|
- **Ch 19–22 Distributions & Foliations / Exponential Map / Quotient Manifolds / Symplectic Manifolds**:可积分布(Frobenius)、指数映射(retraction 原型)、商流形(Grassmann 等)、辛形式与 Hamilton 流。→ retraction、商空间约束、辛积分器 / HMC。
|
|
@@ -26,7 +26,7 @@
|
|
|
26
26
|
## 可迁移到 AI/Infra 的核心结构
|
|
27
27
|
|
|
28
28
|
- **切空间 = 参数/隐空间的局部线性化(local linearization)**。`df_p: T_pM → T_{f(p)}N` 就是 Jacobian / pushforward;反向传播 = 沿复合映射做 pushforward(链式法则的几何版)。一切一阶方法都活在切空间里。
|
|
29
|
-
- **梯度是余向量(covector),不是向量**。autodiff 给出的是 1-form(余切空间元素);要变成可下降的方向(切向量)必须用
|
|
29
|
+
- **梯度是余向量(covector),不是向量**。autodiff 给出的是 1-form(余切空间元素);要变成可下降的方向(切向量)必须用 **度量升指标**((sharp))。欧氏度量 → 普通梯度;Fisher 度量 → 自然梯度(natural gradient)。**这是自然梯度 / 镜像下降的流形根因**。
|
|
30
30
|
- **约束集 = 子流形(submanifold)**。正则水平集定理:当 g 是 submersion 时 `g(x)=c` 的解集是光滑子流形;约束优化 = 在子流形上做无约束优化。
|
|
31
31
|
- **李群 = 可微的对称群**。SO(n)/U(n)/Stiefel/Grassmann 都是流形;其李代数(如反对称矩阵 so(n))是线性空间,用 `exp` 映射回群 → **把"约束权重"重参数化为"无约束李代数 + exp"**。
|
|
32
32
|
- **流(flow)= 时间参数化的微分同胚族**。学一个向量场 + 沿它积分 = Neural ODE / 连续归一化流 / 扩散采样。流的可逆性、保体积性直接对应模型性质。
|
|
@@ -57,10 +57,10 @@
|
|
|
57
57
|
|
|
58
58
|
逐维对照:
|
|
59
59
|
|
|
60
|
-
- **维度 1–2 张量化 / GEMM**:切空间运算(pushforward/pullback、Jacobian-向量积、把梯度投影到切空间)**天然是 batched GEMM**
|
|
60
|
+
- **维度 1–2 张量化 / GEMM**:切空间运算(pushforward/pullback、Jacobian-向量积、把梯度投影到切空间)**天然是 batched GEMM** [v]——反向传播本就是 pushforward,这部分对 GPU 极友好。**但** retraction/exp 多半要 QR、特征分解、矩阵指数或小矩阵求逆:QR/eig **不是干净的 GEMM**,是带串行依赖的分解(cuSOLVER 批量小矩阵尚可,大矩阵 O(n³) 且并行差)→ **可改造** 而非天然友好。
|
|
61
61
|
- **维度 3 复杂度**:测地线距离、平行移动、一般 `log|det J|` 都是 O(n³) 起。**改造**:限定有闭式测地线的流形(球面/双曲/SO(3));归一化流用三角/耦合层让 log-det 退化成对角和(O(n))。
|
|
62
|
-
- **维度 5
|
|
63
|
-
- **维度 6 并行与通信**:scaling-and-squaring 的平方链、ODE 积分步、Householder/QR 都有 **串行递推**,难跨 SM/设备 overlap。反例向好:显式辛积分器(leapfrog)高并行
|
|
62
|
+
- **维度 5 低精度**:[~] **最大坑**。矩阵 `exp / log / sqrt`、特征分解、SPD 的仿射不变度量在 bf16/fp16 下 **灾难性不稳定**,常静默地需要 fp32/fp64。流形原语经常"表面能跑、数值早已发散"。
|
|
63
|
+
- **维度 6 并行与通信**:scaling-and-squaring 的平方链、ODE 积分步、Householder/QR 都有 **串行递推**,难跨 SM/设备 overlap。反例向好:显式辛积分器(leapfrog)高并行 [v]。
|
|
64
64
|
- **维度 4/7/8 显存 / 稀疏 / 融合**:李代数/旋转参数化若限制在 **小矩阵或块对角**(如逐头旋转、SO(3) 的 Rodrigues 闭式),可融进 kernel、走 Tensor Core;大稠密流形算子则要物化大中间张量。
|
|
65
65
|
|
|
66
66
|
**结论与改造手法(呼应 gpu-friendly-math.md 工具箱)**:
|
|
@@ -73,11 +73,11 @@
|
|
|
73
73
|
|
|
74
74
|
## 该调用哪个思想透镜
|
|
75
75
|
|
|
76
|
-
- **
|
|
77
|
-
- **symmetry
|
|
76
|
+
- **variational(变分透镜)**:主透镜——约束下寻最优、黎曼/流形优化、retraction 选型。
|
|
77
|
+
- **symmetry(对称与不变性)**:李群、等变、商流形、群作用下的不变量。
|
|
78
78
|
- **duality(对偶透镜)**:坐标卡变换、pushforward/pullback、归一化流的变量替换、微分同胚。
|
|
79
|
-
- **topological
|
|
80
|
-
- **
|
|
79
|
+
- **topological(拓扑透镜)**:de Rham 上同调、全局障碍、隐空间的"洞"与连通性。
|
|
80
|
+
- **categorical(范畴化透镜)**:从高维杂乱的环境数据中抽出"局部线性 + 光滑拼接"的流形骨架(流形假设)。
|
|
81
81
|
|
|
82
82
|
## 反模式
|
|
83
83
|
|
|
@@ -90,7 +90,7 @@
|
|
|
90
90
|
|
|
91
91
|
## 深挖入口
|
|
92
92
|
|
|
93
|
-
>
|
|
93
|
+
> **书目信息**:John M. Lee, *Introduction to Smooth Manifolds*, 2nd Edition, Graduate Texts in Mathematics 218, Springer, 2013. ISBN 978-1-4419-9981-8.
|
|
94
94
|
>
|
|
95
95
|
> **启用方式**:将 `Introduction to Smooth Manifolds.pdf` 放入项目根目录的 `math_book/` 文件夹,Agent 即可自动搜索原文。PDF 不随 npm/git 分发(版权原因),需自行获取。
|
|
96
96
|
|
|
@@ -98,7 +98,7 @@
|
|
|
98
98
|
|
|
99
99
|
- **Ch 3 Tangent Vectors** — 切空间、微分/pushforward、切丛:局部线性化与反传的几何原型。
|
|
100
100
|
- **Ch 11 The Cotangent Bundle** — 余向量场(1-form)、`df` 作为余向量、pullback:梯度真身 = 余向量。
|
|
101
|
-
- **Ch 13 Riemannian Metrics** —
|
|
101
|
+
- **Ch 13 Riemannian Metrics** — 度量、切-余切同构((sharp)/(flat))、距离:自然梯度 / 黎曼优化的根。
|
|
102
102
|
- **Ch 9 Integral Curves and Flows** — 流、积分曲线、李导数/李括号:Neural ODE / 扩散 / 保结构动力学。
|
|
103
103
|
- **Ch 20 The Exponential Map** — 指数映射:retraction 原型,也是 GPU 可行性的主瓶颈。
|
|
104
104
|
|
|
@@ -5,6 +5,19 @@
|
|
|
5
5
|
>
|
|
6
6
|
> This file is the single source of truth for the "GPU-feasibility" acceptance gate. The activator, the 15 thinking lenses, the book references, and the math-critic all point here.
|
|
7
7
|
|
|
8
|
+
## Eight-Dimension Abbreviations
|
|
9
|
+
|
|
10
|
+
| Abbrev. | Full Name |
|
|
11
|
+
|---------|-----------|
|
|
12
|
+
| D1 | Tensorization |
|
|
13
|
+
| D2 | GEMM-mappability |
|
|
14
|
+
| D3 | Complexity |
|
|
15
|
+
| D4 | Memory & KV-Cache |
|
|
16
|
+
| D5 | Low-precision stability |
|
|
17
|
+
| D6 | Parallelism & communication |
|
|
18
|
+
| D7 | Sparse structure |
|
|
19
|
+
| D8 | Operator fusion |
|
|
20
|
+
|
|
8
21
|
## Core Proposition
|
|
9
22
|
|
|
10
23
|
**Mathematical beauty ≠ computability.** For a structure to truly enter training and inference on modern GPU clusters, it must simultaneously satisfy two requirements:
|
|
@@ -18,7 +31,7 @@ Many "beautiful on paper" modern mathematical structures cannot run at high perf
|
|
|
18
31
|
|
|
19
32
|
For any candidate structure (operator, attention variant, routing mechanism, regularization term, compression scheme…) rate each dimension as `Friendly / Retrofittable / Unfriendly` and provide adaptation recommendations.
|
|
20
33
|
|
|
21
|
-
| # | Dimension | Key Question | Friendly
|
|
34
|
+
| # | Dimension | Key Question | Friendly [v] | Unfriendly [x] |
|
|
22
35
|
|---|-----------|-------------|------------|--------------|
|
|
23
36
|
| 1 | **Tensorization** | Can it be expressed as dense tensor operations, avoiding element-wise irregular control flow? | Batched tensor algebra | Scalar loops, data-dependent branches |
|
|
24
37
|
| 2 | **GEMM-mappability** | Can it be reduced to matrix multiplication / batched GEMM / convolution to fully utilize Tensor Cores? | Expressible as a GEMM chain | Irregular computations that cannot be expressed as matrix operations |
|
|
@@ -57,7 +70,7 @@ Drawn from the auto-research directions cited in `agentic-workflow.md`, demonstr
|
|
|
57
70
|
|
|
58
71
|
| Component | Mathematical Source | GPU Friendliness |
|
|
59
72
|
|-----------|-------------------|-----------------|
|
|
60
|
-
| Tropical Gating | Tropical semiring, piecewise-linear | Replaces Top-K: element-wise max-plus — dim 1
|
|
73
|
+
| Tropical Gating | Tropical semiring, piecewise-linear | Replaces Top-K: element-wise max-plus — dim 1 [v] Tensorization / dim 2 [x] Not a Tensor Core GEMM (runs on CUDA cores) / dim 3 [v] Per-token gating only, sub-quadratic (min-plus matmul is APSP-hard, not sub-quadratic); sub-differentiable, kinks require LogSumExp smoothing (smoothing recovers standard softmax) |
|
|
61
74
|
| Cellular Sheaf Diffusion | Algebraic geometry / topology (sheaves, restriction maps) | Each edge is a low-rank linear transform = small GEMM (dim 2/4) |
|
|
62
75
|
| Čech Cohomology Regularization | Algebraic topology (first cohomology $H^1$) | Local, inexpensive; serves as an algebraic criterion for hallucination (dim 3/8) |
|
|
63
76
|
| Low-Rank Basis KV Compression (Plücker/Grassmannian perspective) | Projective geometry | Store the basis rather than Plücker coordinates (the latter expands when low-rank); block-summary candidate — compression ratio / error / throughput must be benchmarked (dim 4) |
|
|
@@ -7,6 +7,19 @@
|
|
|
7
7
|
>
|
|
8
8
|
> This file is the single source of truth for the "GPU-feasibility" acceptance gate. The activator, the 15 thinking lenses, the book references, and the math-critic all point here.
|
|
9
9
|
|
|
10
|
+
## 八维缩写速查
|
|
11
|
+
|
|
12
|
+
| 缩写 | 全称 |
|
|
13
|
+
|------|------|
|
|
14
|
+
| D1 | 张量化 |
|
|
15
|
+
| D2 | GEMM 可映射 |
|
|
16
|
+
| D3 | 复杂度 |
|
|
17
|
+
| D4 | 显存与 KV-Cache |
|
|
18
|
+
| D5 | 低精度稳定 |
|
|
19
|
+
| D6 | 并行与通信 |
|
|
20
|
+
| D7 | 稀疏结构 |
|
|
21
|
+
| D8 | 算子融合 |
|
|
22
|
+
|
|
10
23
|
## 核心命题 / Core Proposition
|
|
11
24
|
|
|
12
25
|
**数学美 ≠ 可算。** 一个结构要真正进入现代 GPU 集群的训练与推理,必须同时满足两件事:
|
|
@@ -20,7 +33,7 @@
|
|
|
20
33
|
|
|
21
34
|
对任何候选结构(算子、注意力变体、路由、正则项、压缩方案……)逐项评 `友好 / 可改造 / 不友好`,并给出改造建议。
|
|
22
35
|
|
|
23
|
-
| # | 维度 | 关键问题 | 友好
|
|
36
|
+
| # | 维度 | 关键问题 | 友好 [v] | 不友好 [x] |
|
|
24
37
|
|---|------|---------|--------|----------|
|
|
25
38
|
| 1 | **张量化 / Tensorization** | 能否表达为稠密张量运算,避免逐元素不规则控制流? | 批量张量代数 | 标量循环、数据相关分支 |
|
|
26
39
|
| 2 | **GEMM 可映射 / GEMM mappability** | 能否落到 矩阵乘 / batched GEMM / 卷积,吃满 Tensor Core? | 可写成 GEMM 链 | 无法矩阵化的不规则运算 |
|
|
@@ -59,7 +72,7 @@
|
|
|
59
72
|
|
|
60
73
|
| 组件 | 数学来源 | GPU 友好性 |
|
|
61
74
|
|------|---------|-----------|
|
|
62
|
-
| 热带门控 Tropical Gating | 热带半环分段线性 | 替代 Top-K:逐元素 max-plus——dim 1
|
|
75
|
+
| 热带门控 Tropical Gating | 热带半环分段线性 | 替代 Top-K:逐元素 max-plus——dim 1 [v] 张量化 / dim 2 [x] 非 Tensor Core GEMM(落 CUDA core)/ dim 3 [v] 仅逐 token 门控亚二次(min-plus matmul 则 APSP-hard 非亚二次);次可微,折点需 LogSumExp 软化(软化退回标准 softmax)|
|
|
63
76
|
| 胞腔层扩散 Cellular Sheaf Diffusion | 代数几何/拓扑(层、限制映射)| 每边低秩线性变换 = 小 GEMM(维度 2/4)|
|
|
64
77
|
| Čech 上同调正则 | 代数拓扑(一阶上同调 H¹)| 局部、廉价;作幻觉的代数判据(维度 3/8)|
|
|
65
78
|
| 低秩基底 KV 压缩(Plücker/Grassmannian 视角) | 射影几何 | 存基底而非 Plücker 坐标(后者低秩时反扩张);块摘要候选,压缩率/误差/吞吐需实测(维度 4)|
|
|
@@ -6,7 +6,33 @@ description: |
|
|
|
6
6
|
|
|
7
7
|
|
|
8
8
|
> **File routing**: When this document references any file (lenses, knowledge-base, design-patterns, references, agents), always load the `.en.md` variant if it exists. For example, `../../lenses/symmetry.md` → load `lenses/symmetry.en.md`.
|
|
9
|
-
|
|
9
|
+
>
|
|
10
|
+
> ## Language Routing & Mixed-Input Rules
|
|
11
|
+
>
|
|
12
|
+
> Language routing only determines "which language version to read" and "what language to respond in." It does NOT affect whether the math system triggers or the A/B/C/D/E scenario classification.
|
|
13
|
+
>
|
|
14
|
+
> ### Decision Rules
|
|
15
|
+
>
|
|
16
|
+
> 1. **Judge the natural-language frame first**
|
|
17
|
+
> - If the user's sentence structure, verbs, and mood particles are primarily Chinese, treat as Chinese even if English technical terms are interspersed.
|
|
18
|
+
> - E.g., "帮我 design 一个 attention" / "这个 loss 有没有理论问题" → Chinese.
|
|
19
|
+
>
|
|
20
|
+
> 2. **English technical terms do not count as English primary language**
|
|
21
|
+
> - attention, loss, routing, embedding, manifold, operator, kernel, KV-cache, transformer, MoE, etc. are domain terms and do not trigger a switch to English.
|
|
22
|
+
>
|
|
23
|
+
> 3. **Code, paths, and formulas are excluded from language detection**
|
|
24
|
+
> - File paths, function names, variable names, LaTeX formulas, and CLI arguments do not count toward language ratio.
|
|
25
|
+
>
|
|
26
|
+
> 4. **When primary language is unclear, follow the user's last clear language**
|
|
27
|
+
> - If the CN/EN ratio is close and indeterminate, use the user's most recent unambiguous natural language.
|
|
28
|
+
> - If there is no prior context, default to Chinese.
|
|
29
|
+
>
|
|
30
|
+
> 5. **Output language matches primary language**
|
|
31
|
+
> - Chinese primary → read `SKILL.md`, respond in Chinese, retaining necessary English terms.
|
|
32
|
+
> - English primary → read `SKILL.en.md`, respond in English.
|
|
33
|
+
> - If the user explicitly requests "in English" / "in Chinese," follow the explicit request.
|
|
34
|
+
|
|
35
|
+
# Math Research OS
|
|
10
36
|
|
|
11
37
|
> "The thinking system does not hand out theorems, the knowledge system does not indulge in loose inspiration, and the design layer does not fake profundity."
|
|
12
38
|
|
|
@@ -5,9 +5,32 @@ description: |
|
|
|
5
5
|
English: Mathematical research OS — auto-diagnoses user intent, routes to thinking lenses, math knowledge base, or design translation layer. Triggers on architecture/operator design, theoretical analysis, math-to-AI transfer. Does NOT trigger for pure engineering tasks.
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
>
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
> ## 语言路由与混合输入规则
|
|
9
|
+
>
|
|
10
|
+
> 语言路由只决定"读取哪个语言版本的说明"和"最终用什么语言回答",不参与数学系统是否触发、也不参与 A/B/C/D/E 场景判断。
|
|
11
|
+
>
|
|
12
|
+
> ### 判定规则
|
|
13
|
+
>
|
|
14
|
+
> 1. **先判断自然语言主框架**
|
|
15
|
+
> - 如果用户的请求句式、动词、语气词主要是中文,即使夹杂英文技术词,也按中文处理。
|
|
16
|
+
> - 例如:"帮我 design 一个 attention""这个 loss 有没有理论问题""能不能用 manifold 做 routing"均按中文处理。
|
|
17
|
+
>
|
|
18
|
+
> 2. **英文技术词不计入英文主语言**
|
|
19
|
+
> - attention、loss、routing、embedding、manifold、operator、kernel、KV-cache、transformer、MoE 等 AI/数学/工程术语视为领域术语,不作为切换到英文的依据。
|
|
20
|
+
>
|
|
21
|
+
> 3. **代码、路径、公式不参与语言判定**
|
|
22
|
+
> - 文件路径、函数名、变量名、LaTeX 公式、命令行参数不计入语言比例。
|
|
23
|
+
>
|
|
24
|
+
> 4. **主语言不明显时,沿用用户上一轮主要语言**
|
|
25
|
+
> - 若中英文比例接近且无法判断,以用户最近一次明确使用的自然语言为准。
|
|
26
|
+
> - 若没有上下文,默认中文。
|
|
27
|
+
>
|
|
28
|
+
> 5. **输出语言与主语言一致**
|
|
29
|
+
> - 中文主语言 → 读取中文 `SKILL.md`,用中文回答,保留必要英文术语。
|
|
30
|
+
> - 英文主语言 → 读取 `SKILL.en.md`,用英文回答。
|
|
31
|
+
> - 用户明确要求"用英文/用中文"时,以用户显式要求为准。
|
|
32
|
+
|
|
33
|
+
# 数学研究操作系统 / Math Research OS
|
|
11
34
|
|
|
12
35
|
> "思想系统不负责给定理,知识系统不负责乱启发,设计层不负责装深刻。"
|
|
13
36
|
|