math-skill 2.1.0 → 3.0.0

package/knowledge-base/optimization/convex-optimization.en.md

@@ -0,0 +1,50 @@
+# Convex Optimization
+
+## Minimal Definition
+
+Optimization problems that minimize a convex function over a convex set. Convexity guarantees that a local optimum is also a global optimum, and the first-order condition ($\nabla f = 0$) is both necessary and sufficient. Convex problems are the only class in optimization theory that guarantees global optimality with efficient solvability.
+
+## Core Formulas
+
+- Convex set: $\forall x, y \in C, \theta \in [0,1]: \theta x + (1-\theta)y \in C$
+- Convex function: $f(\theta x + (1-\theta)y) \leq \theta f(x) + (1-\theta)f(y)$
+- First-order condition: $f(y) \geq f(x) + \nabla f(x)^T(y-x)$ (tangent line lies below the function)
+- Second-order condition: $\nabla^2 f(x) \succeq 0$ (Hessian is positive semidefinite)
+- Standard convex problem form: $\min f(x)$ s.t. $g_i(x) \leq 0$ ($g_i$ convex), $Ax = b$
+- Convergence rate (strongly convex + smooth): $\|x_k - x^*\|^2 \leq (1 - \mu/L)^k \|x_0 - x^*\|^2$, condition number $\kappa = L/\mu$
+- Semidefinite programming (SDP): $\min \langle C, X \rangle$ s.t. $\langle A_i, X \rangle = b_i, X \succeq 0$
+
+## Applicable Problems
+
+- Linear/logistic regression: cross-entropy + linear model is a convex problem with a unique global optimum
+- Weight decay / regularization: $\|w\|_2^2$ and $\|w\|_1$ are both convex regularizers
+- SVM: hinge loss + quadratic regularization = convex problem
+- PCA: the maximum eigenvalue problem for the covariance matrix = a special case of SDP
+- Gram matrix optimization in kernel methods: SDP constraint $K \succeq 0$
+
+## AI Design Translation
+
+- **Convexity diagnosis of loss functions**: Cross-entropy is convex with respect to logits (softmax + NLL), and MSE is convex with respect to linear outputs. However, once composed with nonlinear layers (ReLU, attention), the overall problem becomes non-convex. Maintaining convexity from the last layer to the loss provides a convergence guarantee when designing losses.
+- **Learning rate scheduling and convex optimization convergence rates**: Under strong convexity + smoothness, SGD converges at rate $O(1/T)$; without strong convexity, $O(1/\sqrt{T})$. The condition number $\kappa = L/\mu$ determines convergence speed: the role of BatchNorm / LayerNorm is to reduce $\kappa$ (making the Hessian more isotropic), equivalent to preconditioning. Implemented as standard normalization layers.
+- **Convex relaxation**: Relaxing a non-convex problem into a convex one. Examples: $\ell_0$ sparsity $\to \ell_1$ (LASSO); matrix rank minimization $\to$ nuclear norm minimization; integer programming $\to$ LP relaxation. Implemented by substituting the regularizer or constraint.
+- **Projection onto convex sets**: $\text{proj}_C(x) = \arg\min_{y \in C} \|y - x\|^2$. $\ell_2$-ball projection = $x / \max(1, \|x\|_2/R)$ (elementwise + norm); $\ell_1$-ball projection = soft-thresholding + sort ($O(n\log n)$); box constraints = clamp (elementwise). All are GPU-friendly operations.
+- **Mirror descent**: In non-Euclidean geometries, replaces Euclidean distance with Bregman divergence in gradient updates. For $\ell_1$ constraints (sparse weights), uses exponential gradient $x_{k+1} \propto x_k \exp(-\eta \nabla f)$, implemented as elementwise exp + normalize.
+
+## Engineering Feasibility
+
+- **Primary operations**: Gradient computation = backpropagation (matmul chains); projection = elementwise + norm; convex function evaluation = forward pass. Overall isomorphic to the standard training loop.
+- **GPU friendliness**: Extremely high. First-order methods for convex optimization (gradient descent, projected gradient, mirror descent) map entirely to GPU operators. Second-order methods (Newton, interior point) are viable at moderate scale ($d < 10000$) via cuSOLVER.
+- **Complexity**: Gradient descent per step $O(d)$ (gradient computation $O(\text{model FLOPs})$); projection $O(d)$ to $O(d\log d)$; interior-point methods per step $O(d^3)$.
+- **Low precision**: Strongly convex problems are stable under bf16 (gradients are Lipschitz); when the condition number is large, normalization/preconditioning is needed, otherwise low precision amplifies ill-conditioning.
+
+## Risks and Failure Conditions
+
+- **False convexity**: A seemingly convex loss becomes non-convex after composition with nonlinearities (e.g., $f(W_2 \sigma(W_1 x))$ is non-convex in $W_1, W_2$). Convexity from the last layer to the loss does not imply global convexity.
+- **Condition number degradation**: As $\kappa = L/\mu \to \infty$ ($\mu \to 0$, weakly convex or flat directions), the convergence rate degrades to $O(1/\sqrt{T})$. Solution: add $\ell_2$ regularization to make the problem strongly convex ($\mu \geq \lambda$), or use normalization layers to improve $\kappa$.
+- **Poor scalability of SDP solvers**: Interior-point SDP solvers have complexity $O(n^6)$ ($n$ being the matrix dimension), becoming infeasible beyond $n = 500$. Large-scale settings require first-order methods (ADMM) or approximate convex relaxations.
+- **Convex relaxation gap**: $\ell_1$ relaxation does not necessarily recover the $\ell_0$ sparse solution (requires RIP conditions); nuclear norm relaxation does not necessarily yield the lowest-rank solution. Relaxation quality depends on problem structure.
+
+## Further References
+
+- Distilled notes: references/books/optimization-ml.md (Ch 22 Convex Optimization, Section 22.2 Convex Functions, Section 22.3 Convex Problems, Section 22.4 SDP/LMI)
+- Original text: Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 22 (Convex Optimization Problems Section 22.1-22.4)

package/knowledge-base/optimization/convex-optimization.md

@@ -0,0 +1,50 @@
+# 凸优化 (Convex Optimization)
+
+## 最小定义
+
+在凸集上最小化凸函数的优化问题。凸性保证局部最优即全局最优，且一阶条件（$\nabla f = 0$）是充分必要的。凸问题是优化理论中唯一能保证全局最优且可高效求解的类别。
+
+## 核心公式
+
+- 凸集：$\forall x, y \in C, \theta \in [0,1]: \theta x + (1-\theta)y \in C$
+- 凸函数：$f(\theta x + (1-\theta)y) \leq \theta f(x) + (1-\theta)f(y)$
+- 一阶条件：$f(y) \geq f(x) + \nabla f(x)^T(y-x)$（切线在下方的上方）
+- 二阶条件：$\nabla^2 f(x) \succeq 0$（Hessian 半正定）
+- 标准凸问题形式：$\min f(x)$ s.t. $g_i(x) \leq 0$（$g_i$ 凸）, $Ax = b$
+- 收敛速率（强凸 + 光滑）：$\|x_k - x^*\|^2 \leq (1 - \mu/L)^k \|x_0 - x^*\|^2$，条件数 $\kappa = L/\mu$
+- 半定规划 (SDP)：$\min \langle C, X \rangle$ s.t. $\langle A_i, X \rangle = b_i, X \succeq 0$
+
+## 适用问题
+
+- 线性/逻辑回归：交叉熵 + 线性模型是凸问题，有唯一全局最优
+- 权重衰减 / 正则化：$\|w\|_2^2$、$\|w\|_1$ 都是凸正则项
+- SVM：hinge loss + 二次正则 = 凸问题
+- PCA：协方差矩阵的最大特征值问题 = SDP 的特例
+- 核方法的 Gram 矩阵优化：SDP 约束 $K \succeq 0$
+
+## AI 设计翻译
+
+- **Loss 函数的凸性诊断**：交叉熵对 logits 是凸的（softmax + NLL），MSE 对线性输出是凸的。但一旦过非线性层（ReLU、attention），整体变为非凸。设计 loss 时保持最后一层到 loss 的凸性是收敛保障。
+- **学习率调度与凸优化收敛率**：强凸 + 光滑下 SGD 收敛率 $O(1/T)$，非强凸 $O(1/\sqrt{T})$。条件数 $\kappa = L/\mu$ 决定收敛速度：BatchNorm / LayerNorm 的作用是减小 $\kappa$（使 Hessian 更圆），等价于预条件。实现为标准归一化层。
+- **凸松弛 (Convex Relaxation)**：将非凸问题松弛为凸问题求解。例：$\ell_0$ 稀疏 $\to \ell_1$（LASSO）；矩阵秩最小化 $\to$ 核范数最小化；整数规划 $\to$ LP 松弛。实现为替换正则项或约束。
+- **投影到凸集 (Convex Projection)**：$\text{proj}_C(x) = \arg\min_{y \in C} \|y - x\|^2$。$\ell_2$-ball 投影 = $x / \max(1, \|x\|_2/R)$（elementwise + norm）；$\ell_1$-ball 投影 = soft-thresholding + sort（$O(n\log n)$）；box 约束 = clamp（elementwise）。全是 GPU 友好操作。
+- **镜像下降 (Mirror Descent)**：在非欧几何下用 Bregman 散度代替欧氏距离做梯度更新。对 $\ell_1$ 约束（稀疏权重），用指数梯度 $x_{k+1} \propto x_k \exp(-\eta \nabla f)$，实现为 elementwise exp + normalize。
+
+## 工程可行性
+
+- **主要操作**：梯度计算 = 反向传播（matmul 链）；投影 = elementwise + norm；凸函数评估 = 前向传播。整体与标准训练循环同构。
+- **GPU 友好度**：极高。凸优化的一阶方法（梯度下降、投影梯度、镜像下降）完全映射到 GPU 算子。二阶方法（Newton、内点法）在中等规模（$d < 10000$）可用 cuSOLVER。
+- **复杂度**：梯度下降每步 $O(d)$（梯度计算 $O(\text{model FLOPs})$）；投影 $O(d)$ 到 $O(d\log d)$；内点法每步 $O(d^3)$。
+- **低精度**：强凸问题在 bf16 下稳定（梯度是 Lipschitz 的）；条件数大时需归一化/预条件，否则低精度放大病态。
+
+## 风险与失效条件
+
+- **假凸性**：看似凸的 loss 在复合非线性后变成非凸（如 $f(W_2 \sigma(W_1 x))$ 对 $W_1, W_2$ 非凸）。仅最后一层到 loss 是凸的不代表全局凸。
+- **条件数退化**：$\kappa = L/\mu \to \infty$ 时（$\mu \to 0$，弱凸或平坦方向），收敛率退化为 $O(1/\sqrt{T})$。解决：加 $\ell_2$ 正则使问题强凸（$\mu \geq \lambda$），或用归一化层改善 $\kappa$。
+- **SDP 求解器扩展性差**：内点法 SDP 求解器复杂度 $O(n^6)$（$n$ 为矩阵维度），超过 $n = 500$ 基本不可行。大规模场景需一阶方法（ADMM）或凸松弛后近似。
+- **凸松弛的间隙**：$\ell_1$ 松弛不一定恢复 $\ell_0$ 稀疏解（需 RIP 条件）；核范数松弛不一定给出最低秩解。松弛质量依赖问题结构。
+
+## 深入参考
+
+- 蒸馏稿：references/books/optimization-ml.md（Ch 22 Convex Optimization、§22.2 凸函数、§22.3 凸问题、§22.4 SDP/LMI）
+- 原书：Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 22 (Convex Optimization Problems §22.1-22.4)

package/knowledge-base/optimization/lagrangian-duality.en.md

@@ -0,0 +1,50 @@
+# Lagrangian Duality
+
+## Minimal Definition
+
+Transforms a constrained optimization problem (primal) into a maximization problem over dual variables (dual). The dual function $g(\lambda, \nu) = \inf_x L(x, \lambda, \nu)$ provides a lower bound on the primal optimal value (weak duality); when the duality gap is zero (strong duality), the dual optimal value equals the primal optimal value.
+
+## Core Formulas
+
+- Primal problem: $\min_x f(x) \quad \text{s.t.} \quad g_i(x) \leq 0, \ h_j(x) = 0$
+- Lagrangian: $L(x, \lambda, \nu) = f(x) + \sum_i \lambda_i g_i(x) + \sum_j \nu_j h_j(x)$
+- Dual function: $g(\lambda, \nu) = \inf_x L(x, \lambda, \nu)$ (pointwise infimum over $x$, naturally concave)
+- Weak duality: $d^* \leq p^*$ (dual optimum $\leq$ primal optimum, always holds)
+- Strong duality condition (Slater): convex problem + existence of a strictly feasible point $g_i(x_0) < 0 \implies d^* = p^*$
+- Complementary slackness: $\lambda_i^* g_i(x^*) = 0$ (at optimality, either the constraint is tight or the multiplier is zero)
+- Minimax equivalence: strong duality $\iff \min_x \max_{\lambda \geq 0} L = \max_{\lambda \geq 0} \min_x L$
+
+## Applicable Problems
+
+- GAN / adversarial training: $\min_G \max_D V(D,G)$ is a minimax game; under strong duality, the saddle point = Nash equilibrium
+- SVM dual: primal $O(d)$ dimensions converted to dual $O(n)$ dimensions + kernel Gram matrix; profitable when the number of samples $\ll$ dimensionality
+- Constraint decomposition: large-scale problems decomposed by constraints into subproblems that can be solved in parallel after dual decomposition
+- Dual interpretation of regularization parameters: $\lambda$-regularization $\iff$ a constrained primal problem (e.g., $\|w\|_2^2 \leq C$)
+- Resource allocation / federated learning: global constraints decomposed into Lagrangian relaxations of local subproblems
+
+## AI Design Translation
+
+- **Minimax framework for adversarial training**: $L(\theta, \phi) = \mathbb{E}[\log D_\phi(x)] + \mathbb{E}[\log(1 - D_\phi(G_\theta(z)))]$. $G$ and $D$ alternate between gradient ascent and descent; the core operations are forward + backward passes (matmul chains). Gradient penalty / spectral norm constrains the Lipschitz constant of $D$ to ensure the existence of a saddle point.
+- **SVM dual + kernel trick**: Primal $\min_w \frac{1}{2}\|w\|^2 + C\sum\xi_i$ converted to dual $\max_\alpha \sum\alpha_i - \frac{1}{2}\alpha^T(K \circ yy^T)\alpha$. The kernel Gram matrix $K_{ij} = k(x_i, x_j)$ is computed via matmul (linear kernel) or elementwise operations (RBF kernel). Dual variables $\alpha \in \mathbb{R}^n$ are solved using SMO or gradient projection.
+- **Augmented Lagrangian constrained training**: $\mathcal{L}_{\text{AL}} = f(x) + \sum \lambda_i g_i(x) + \frac{\rho}{2}\sum g_i(x)^2$. The inner loop optimizes $x$ via SGD; the outer loop updates $\lambda$ via gradient ascent: $\lambda \leftarrow [\lambda + \rho g(x)]_+$. The $\rho$ term ensures inner-loop convexity, enabling convergence even for non-convex original problems.
+- **Dual decomposition**: $\min \sum_k f_k(x_k)$ s.t. $\sum x_k \leq b$ decomposes into independent subproblems $\min_{x_k} f_k(x_k) + \lambda^T x_k$, with the master problem $\max_\lambda g(\lambda)$ solved via subgradient ascent. Naturally parallelizable, suitable for distributed training.
+- **Dual perspective on the information bottleneck**: $\min I(Z;X) - \beta I(Z;Y)$ can be formulated as constrained optimization, where $\beta$ is the dual variable. The variational IB relaxes this via the ELBO, reducing to standard VAE training (reparameterization trick + SGD).
+
+## Engineering Feasibility
+
+- **Primary operations**: Lagrangian evaluation = primal objective + weighted sum of constraint terms (elementwise + reduce); dual gradient ascent = standard gradient update; SMO (SVM) = coordinate-descent-style $2 \times 2$ subproblems.
+- **GPU friendliness**: High (dual function evaluation) to moderate (sequential solvers such as SMO). The inner-loop optimization of the augmented Lagrangian is entirely standard SGD (matmul chains), and dual variable updates are elementwise. SMO uses coordinate-wise updates, which are not GPU-friendly, but CPU-sufficient when $n$ is not large.
+- **Complexity**: Dual function evaluation = one forward pass $O(\text{model FLOPs})$; dual gradient ascent per step $O(n)$ (number of constraints); SMO $O(n^2 d)$.
+- **Low precision**: Dual variables $\lambda$ should be kept in fp32 (multipliers span a wide range, with overflow risk under low precision); model parameters can use bf16.
+
+## Risks and Failure Conditions
+
+- **Duality gap for non-convex problems**: Strong duality is only guaranteed for convex problems + Slater's condition. In non-convex neural network training, $d^* < p^*$ is common, and the dual solution does not yield a primal-feasible solution. Solution: augmented Lagrangian (eliminates the gap when $\rho$ is sufficiently large) or SQP.
+- **Dual variable oscillation**: Improper step sizes for gradient ascent on $\lambda$ can cause dual variable oscillation and primal infeasibility. Solution: use adaptive step sizes (Adam updates for $\lambda$) or an increasing $\rho$ schedule in the augmented Lagrangian.
+- **Numerical determination of complementary slackness**: $\lambda_i g_i(x) = 0$ can only be satisfied to $\sim 10^{-6}$ in floating point; strict complementary slackness is unattainable. This affects SVM support vector identification; a threshold must be set.
+- **Mode collapse in minimax training**: The non-convex-non-concave game in GAN $\min\max$ leads to mode collapse or training instability. Additional regularization such as gradient penalty (WGAN-GP) or spectral normalization is required.
+
+## Further References
+
+- Distilled notes: references/books/optimization-ml.md (Ch 23 Lagrangian Duality, Section 23.5 Strong Duality, Section 23.6.3 Slater's Condition)
+- Original text: Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 23 (Lagrangian Duality Section 23.1-23.6) + Chapter 17 (LP Duality)

package/knowledge-base/optimization/lagrangian-duality.md

@@ -0,0 +1,50 @@
+# Lagrange 对偶 (Lagrangian Duality)
+
+## 最小定义
+
+将约束优化问题（primal）转化为对偶变量的最大化问题（dual）。对偶函数 $g(\lambda, \nu) = \inf_x L(x, \lambda, \nu)$ 是原问题最优值的下界（弱对偶），当对偶间隙为零时（强对偶）对偶问题的最优值等于原问题最优值。
+
+## 核心公式
+
+- 原始问题：$\min_x f(x) \quad \text{s.t.} \quad g_i(x) \leq 0, \ h_j(x) = 0$
+- Lagrangian：$L(x, \lambda, \nu) = f(x) + \sum_i \lambda_i g_i(x) + \sum_j \nu_j h_j(x)$
+- 对偶函数：$g(\lambda, \nu) = \inf_x L(x, \lambda, \nu)$（对 $x$ 的逐点下确界，天然凹）
+- 弱对偶：$d^* \leq p^*$（对偶最优 $\leq$ 原问题最优，恒成立）
+- 强对偶条件（Slater）：凸问题 + 存在严格可行点 $g_i(x_0) < 0 \implies d^* = p^*$
+- 互补松弛：$\lambda_i^* g_i(x^*) = 0$（最优时，要么约束紧、要么乘子为零）
+- Minimax 等价：强对偶 $\iff \min_x \max_{\lambda \geq 0} L = \max_{\lambda \geq 0} \min_x L$
+
+## 适用问题
+
+- GAN / 对抗训练：$\min_G \max_D V(D,G)$ 即 minimax 博弈，强对偶时鞍点 = Nash 均衡
+- SVM 对偶：原始 $O(d)$ 维转对偶 $O(n)$ 维 + 核 Gram 矩阵，样本数 $\ll$ 维度时大赚
+- 约束分解：大规模问题按约束拆成子问题，对偶分解后各子问题可并行求解
+- 正则化参数的对偶解释：$\lambda$-正则化 $\iff$ 带约束的原问题（如 $\|w\|_2^2 \leq C$）
+- 资源分配 / 联邦学习：全局约束分解为本地子问题的 Lagrange 松弛
+
+## AI 设计翻译
+
+- **对抗训练的 minimax 框架**：$L(\theta, \phi) = \mathbb{E}[\log D_\phi(x)] + \mathbb{E}[\log(1 - D_\phi(G_\theta(z)))]$。$G$ 和 $D$ 交替梯度上升/下降，核心操作是前向 + 反向传播（matmul 链）。梯度惩罚 / spectral norm 约束 $D$ 的 Lipschitz 保证 minimax 有鞍点。
+- **SVM 对偶 + 核技巧**：原始 $\min_w \frac{1}{2}\|w\|^2 + C\sum\xi_i$ 转对偶 $\max_\alpha \sum\alpha_i - \frac{1}{2}\alpha^T(K \circ yy^T)\alpha$。核 Gram $K_{ij} = k(x_i, x_j)$ 是 matmul（线性核）或 elementwise（RBF 核）。对偶变量 $\alpha \in \mathbb{R}^n$，用 SMO 或梯度投影求解。
+- **增广 Lagrangian 约束训练**：$\mathcal{L}_{\text{AL}} = f(x) + \sum \lambda_i g_i(x) + \frac{\rho}{2}\sum g_i(x)^2$。内层对 $x$ 用 SGD 优化，外层对 $\lambda$ 梯度上升更新：$\lambda \leftarrow [\lambda + \rho g(x)]_+$。$\rho$ 项保证内层凸性，即使原问题非凸也收敛。
+- **对偶分解 (Dual Decomposition)**：$\min \sum_k f_k(x_k)$ s.t. $\sum x_k \leq b$ 分解为各子问题 $\min_{x_k} f_k(x_k) + \lambda^T x_k$ 独立求解，主问题 $\max_\lambda g(\lambda)$ 用次梯度上升。天然可并行，适合分布式训练。
+- **信息瓶颈的对偶视角**：$\min I(Z;X) - \beta I(Z;Y)$ 可写为带约束优化，$\beta$ 是对偶变量。Variational IB 用 ELBO 松弛后变成标准 VAE 训练（reparameterization trick + SGD）。
+
+## 工程可行性
+
+- **主要操作**：Lagrangian 评估 = 原目标 + 约束项的加权求和（elementwise + reduce）；对偶梯度上升 = 标准梯度更新；SMO（SVM）= 坐标下降式的 $2 \times 2$ 子问题。
+- **GPU 友好度**：高（对偶函数评估）到中等（SMO 等串行求解器）。增广 Lagrangian 的内层优化完全是标准 SGD（matmul 链），对偶变量更新是 elementwise。SMO 是坐标式更新，GPU 不友好，但 $n$ 不大时 CPU 即可。
+- **复杂度**：对偶函数评估 = 一次前向传播 $O(\text{model FLOPs})$；对偶梯度上升每步 $O(n)$（约束数）；SMO $O(n^2 d)$。
+- **低精度**：对偶变量 $\lambda$ 应保持在 fp32（乘子范围大，低精度溢出风险）；模型参数可用 bf16。
+
+## 风险与失效条件
+
+- **非凸问题的对偶间隙**：强对偶仅对凸问题 + Slater 条件保证。非凸神经网络训练中 $d^* < p^*$ 常见，对偶解不给出原始可行解。解决：增广 Lagrangian（$\rho$ 足够大时消除间隙）或 SQP。
+- **对偶变量振荡**：$\lambda$ 的梯度上升步长不当会导致对偶变量振荡、原始不可行。解决：使用自适应步长（Adam 更新 $\lambda$）或增广 Lagrangian 的 $\rho$ 递增策略。
+- **互补松弛的数值判定**：$\lambda_i g_i(x) = 0$ 在浮点下只能达到 $\sim 10^{-6}$，严格互补松弛不可得。影响 SVM 支持向量识别等，需设阈值。
+- **Minimax 训练的 mode collapse**：GAN 中 $\min\max$ 的非凸-非凹博弈导致模式坍缩或训练不稳定。需梯度惩罚（WGAN-GP）、谱归一化等额外正则化。
+
+## 深入参考
+
+- 蒸馏稿：references/books/optimization-ml.md（Ch 23 Lagrangian Duality、§23.5 强对偶、§23.6.3 Slater 条件）
+- 原书：Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 23 (Lagrangian Duality §23.1-23.6) + Chapter 17 (LP Duality)

package/knowledge-base/optimization/proximal-method.en.md

@@ -0,0 +1,52 @@
+# Proximal Methods
+
+## Minimal Definition
+
+For non-smooth or non-differentiable objective functions $f(x) = g(x) + h(x)$ ($g$ smooth, $h$ possibly non-differentiable but "simple"), uses the proximal operator $\text{prox}_{\eta h}(v) = \arg\min_x \{h(x) + \frac{1}{2\eta}\|x - v\|^2\}$ in place of the gradient of $h$. Proximal methods encapsulate the non-differentiable part as a closed-form subproblem.
+
+## Core Formulas
+
+- Proximal operator: $\text{prox}_{\eta h}(v) = \arg\min_x \left\{h(x) + \frac{1}{2\eta}\|x - v\|^2\right\}$
+- Proximal gradient descent (ISTA): $x_{k+1} = \text{prox}_{\eta h}(x_k - \eta \nabla g(x_k))$
+- Accelerated proximal gradient (FISTA): $y_k = x_k + \frac{k-1}{k+2}(x_k - x_{k-1})$, $x_{k+1} = \text{prox}_{\eta h}(y_k - \eta \nabla g(y_k))$, convergence rate $O(1/k^2)$ vs. ISTA's $O(1/k)$
+- Soft-thresholding ($\ell_1$ proximal): $\text{prox}_{\eta\|\cdot\|_1}(v)_i = \text{sign}(v_i)\max(|v_i| - \eta, 0)$
+- Projection (proximal of indicator function): $\text{prox}_{\eta \delta_\mathcal{C}}(v) = \text{proj}_\mathcal{C}(v)$
+- Nuclear norm proximal (singular value soft-thresholding): $\text{prox}_{\eta\|\cdot\|_*}(A) = U(\Sigma - \eta I)_+ V^H$
+- Moreau envelope: $h_\eta(v) = \min_x \{h(x) + \frac{1}{2\eta}\|x-v\|^2\}$ (smooth approximation of $h$)
+- ADMM splitting: $\min f(x) + g(z)$ s.t. $Ax + Bz = c$, alternating updates of $x, z, u$ (dual variable)
+
+## Applicable Problems
+
+- Sparse training / $\ell_1$ regularization: weight sparsification, feature selection
+- Low-rank matrix recovery: nuclear norm regularization (matrix completion, robust PCA)
+- Group sparsity / Group Lasso: $\sum_g \|w_g\|_2$ regularization (structured pruning)
+- Operator splitting for constrained optimization: ADMM decomposes complex constraints into simple subproblems
+- Quantization-aware training: modeling weight quantization as a proximal operator (round + straight-through gradient)
+
+## AI Design Translation
+
+- **Soft-thresholding for sparse training**: $\text{prox}_{\eta\lambda\|\cdot\|_1}(w) = \text{sign}(w) \odot \max(|w| - \eta\lambda, 0)$, implemented as `w.sign() * (w.abs() - eta * lam).clamp(min=0)`, pure elementwise, $O(d)$, zero additional memory. Applying soft-thresholding after each SGD update yields sparse weights.
+- **Singular value soft-thresholding for low-rank regularization**: $\text{prox}_{\eta\|\cdot\|_*}(W) = U(\Sigma - \eta)_+ V^H$. Requires SVD; for large matrices, approximate with randomized SVD: first perform randomized SVD to rank $r$, then apply elementwise soft-thresholding to $\Sigma$, and reconstruct. Core operations are matmul chains + elementwise.
+- **Group Lasso structured pruning**: $\text{prox}_{\eta\sum_g\|w_g\|_2}(w)_g = w_g \cdot \max(1 - \eta/\|w_g\|_2, 0)$. After grouping by channel/head, each group undergoes independent soft-thresholding (norm + elementwise scale), $O(d)$. Implemented as reshape + norm(dim) + clamp + mul.
+- **ADMM for distributed training**: $\min \sum_i f_i(x_i) + g(z)$ s.t. $x_i = z$. Each node independently updates $x_i$ (local SGD), the server updates $z = \text{prox}_{g/\rho}(\bar{x} + u)$ (aggregation + proximal), and $u$ is updated as the dual variable. More communication-efficient than all-reduce (only $x_i$ and $z$ need to be transmitted).
+- **Quantization proximal operator**: Models weight quantization as $\text{prox}(w) = \Delta \cdot \text{round}(w/\Delta)$, with backpropagation using the straight-through estimator (STE): $\partial \text{prox}/\partial w \approx 1$. Implemented as `w_q = (w / delta).round() * delta`; the forward pass is elementwise round + mul, the backward pass is identity.
+
+## Engineering Feasibility
+
+- **Primary operations**: Most proximal operators are elementwise (soft-thresholding, clamp, group norm) or matmul + small SVD (nuclear norm). Gradient steps = standard backpropagation.
+- **GPU friendliness**: Extremely high. $\ell_1$ proximal = elementwise; group lasso proximal = reshape + norm + scale = elementwise; nuclear norm proximal = matmul + small SVD. FISTA's momentum term is also elementwise. ADMM's communication pattern is compatible with data parallelism.
+- **Complexity**: ISTA/FISTA per step = one gradient computation + one proximal operator ($O(d)$ elementwise); nuclear norm proximal = $O(nd^2)$ (reduced to $O(ndk)$ via randomized SVD); ADMM per node = local SGD + $O(d)$ communication.
+- **Low precision**: Elementwise proximal operators are stable under bf16 (no delicate numerical operations). SVD-based proximals must be computed in fp32. FISTA's momentum accumulation may lose precision under bf16; storing $y_k$ in fp32 is recommended.
+
+## Risks and Failure Conditions
+
+- **SVD overhead of nuclear norm proximal**: Full SVD of large matrices at every step is $O(n^3)$, infeasible. Solution: (1) randomized SVD approximation; (2) factorize $\|W\|_* = \min_{W=UV^T} \frac{1}{2}(\|U\|_F^2 + \|V\|_F^2)$, converting to smooth optimization over $U, V$, eliminating the need for SVD.
+- **FISTA restart issues**: FISTA's $y_k$ sequence may oscillate (non-monotonically), causing actual convergence to be slower than theory predicts. Solution: adaptive restart (reset momentum $t_k = 1$ when $f(x_{k+1}) > f(x_k)$), implemented as a simple if-condition.
+- **Sensitivity of ADMM's $\rho$ parameter**: $\rho$ too large causes ill-conditioning of subproblems; $\rho$ too small causes slow dual convergence. Solution: adaptive $\rho$ (automatically adjusted based on the primal/dual residual ratio), updating every several steps as $\rho \leftarrow \rho \cdot \tau$ ($\tau > 1$ if primal residual $>$ dual residual).
+- **No closed-form proximal operator**: Not all $h(x)$ have closed-form prox. Complex regularizers (e.g., TV regularization, overlapping group lasso) require inner-loop iterative solvers. Solution: use Dykstra's algorithm to decompose into compositions of simple prox operators, or switch to ADMM splitting.
+- **Bias of the straight-through estimator**: The STE gradient for the quantization proximal is biased ($\partial \text{round}/\partial w = 0$ almost everywhere), and long-term training may diverge. Solution: compensate with a learnable scale factor, or use stochastic rounding (noisy quantization).
+
+## Further References
+
+- Distilled notes: references/books/optimization-ml.md (Ch 8 Gradient Methods Section 8.3 Convergence Analysis, Ch 11 Quasi-Newton Section 11.5 BFGS, Ch 24 Constrained Algorithms Section 24.5 Augmented Lagrangian)
+- Original text: Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 24 (Constrained Algorithms Section 24.5 Augmented Lagrangian) + Parikh & Boyd, *Proximal Algorithms*, Foundations and Trends in Optimization, 2014

package/knowledge-base/optimization/proximal-method.md

@@ -0,0 +1,52 @@
+# 近端方法 (Proximal Methods)
+
+## 最小定义
+
+对不可微或非光滑目标函数 $f(x) = g(x) + h(x)$（$g$ 光滑、$h$ 可能不可微但"简单"），用近端算子 $\text{prox}_{\eta h}(v) = \arg\min_x \{h(x) + \frac{1}{2\eta}\|x - v\|^2\}$ 代替对 $h$ 的梯度。近端方法将不可微部分封装为一个闭式子问题。
+
+## 核心公式
+
+- 近端算子：$\text{prox}_{\eta h}(v) = \arg\min_x \left\{h(x) + \frac{1}{2\eta}\|x - v\|^2\right\}$
+- 近端梯度下降 (ISTA)：$x_{k+1} = \text{prox}_{\eta h}(x_k - \eta \nabla g(x_k))$
+- 加速近端梯度 (FISTA)：$y_k = x_k + \frac{k-1}{k+2}(x_k - x_{k-1})$，$x_{k+1} = \text{prox}_{\eta h}(y_k - \eta \nabla g(y_k))$，收敛率 $O(1/k^2)$ vs. ISTA 的 $O(1/k)$
+- Soft-thresholding（$\ell_1$ 近端）：$\text{prox}_{\eta\|\cdot\|_1}(v)_i = \text{sign}(v_i)\max(|v_i| - \eta, 0)$
+- 投影（指示函数近端）：$\text{prox}_{\eta \delta_\mathcal{C}}(v) = \text{proj}_\mathcal{C}(v)$
+- 核范数近端（奇异值软阈值）：$\text{prox}_{\eta\|\cdot\|_*}(A) = U(\Sigma - \eta I)_+ V^H$
+- Moreau envelope：$h_\eta(v) = \min_x \{h(x) + \frac{1}{2\eta}\|x-v\|^2\}$（$h$ 的光滑近似）
+- ADMM 分裂：$\min f(x) + g(z)$ s.t. $Ax + Bz = c$，交替更新 $x, z, u$（对偶变量）
+
+## 适用问题
+
+- 稀疏训练 / $\ell_1$ 正则化：权重稀疏化、特征选择
+- 低秩矩阵恢复：核范数正则化（矩阵补全、鲁棒 PCA）
+- 分组稀疏 / Group Lasso：$\sum_g \|w_g\|_2$ 正则化（结构化剪枝）
+- 约束优化的算子分裂：ADMM 将复杂约束分解为简单子问题
+- 量化感知训练：将权重量化建模为近端算子（round + straight-through gradient）
+
+## AI 设计翻译
+
+- **Soft-thresholding 做稀疏训练**：$\text{prox}_{\eta\lambda\|\cdot\|_1}(w) = \text{sign}(w) \odot \max(|w| - \eta\lambda, 0)$，实现为 `w.sign() * (w.abs() - eta * lam).clamp(min=0)`，纯 elementwise，$O(d)$，零额外显存。每次 SGD 更新后做一次 soft-thresholding 即可得到稀疏权重。
+- **奇异值软阈值做低秩正则**：$\text{prox}_{\eta\|\cdot\|_*}(W) = U(\Sigma - \eta)_+ V^H$。需 SVD，大矩阵用随机化 SVD 近似：先做 randomized SVD 到 rank $r$，再对 $\Sigma$ 做 elementwise soft-threshold，重构。核心是 matmul 链 + elementwise。
+- **Group Lasso 结构化剪枝**：$\text{prox}_{\eta\sum_g\|w_g\|_2}(w)_g = w_g \cdot \max(1 - \eta/\|w_g\|_2, 0)$。按通道/头分组后，每组独立做 soft-thresholding（norm + elementwise scale），$O(d)$。实现为 reshape + norm(dim) + clamp + mul。
+- **ADMM 做分布式训练**：$\min \sum_i f_i(x_i) + g(z)$ s.t. $x_i = z$。各节点独立更新 $x_i$（本地 SGD），server 更新 $z = \text{prox}_{g/\rho}(\bar{x} + u)$（聚合 + 近端），$u$ 对偶变量更新。通信效率高于 all-reduce（只需传 $x_i$ 和 $z$）。
+- **量化近端算子**：将权重量化建模为 $\text{prox}(w) = \Delta \cdot \text{round}(w/\Delta)$，反向传播用 straight-through estimator（STE）：$\partial \text{prox}/\partial w \approx 1$。实现为 `w_q = (w / delta).round() * delta`，forward 是 elementwise round + mul，backward 是 identity。
+
+## 工程可行性
+
+- **主要操作**：近端算子多为 elementwise（soft-thresholding、clamp、group norm）或 matmul + 小 SVD（核范数）。梯度步 = 标准反向传播。
+- **GPU 友好度**：极高。$\ell_1$ 近端 = elementwise；group lasso 近端 = reshape + norm + scale = elementwise；核范数近端 = matmul + 小 SVD。FISTA 的动量项也是 elementwise。ADMM 的通信模式适配数据并行。
+- **复杂度**：ISTA/FISTA 每步 = 一次梯度计算 + 一次近端算子（$O(d)$ elementwise）；核范数近端 = $O(nd^2)$（随机化 SVD 降到 $O(ndk)$）；ADMM 每节点 = 本地 SGD + $O(d)$ 通信。
+- **低精度**：elementwise 近端算子在 bf16 下稳定（不涉及精细数值运算）。SVD 类近端需在 fp32 下计算。FISTA 的动量累积在 bf16 下可能丢精度，建议用 fp32 存储 $y_k$。
+
+## 风险与失效条件
+
+- **核范数近端的 SVD 开销**：大矩阵每步完整 SVD 是 $O(n^3)$，不可行。解决：(1) 随机化 SVD 近似；(2) 因子化 $\|W\|_* = \min_{W=UV^T} \frac{1}{2}(\|U\|_F^2 + \|V\|_F^2)$ 转为对 $U, V$ 的光滑优化，无需 SVD。
+- **FISTA 的重启问题**：FISTA 的 $y_k$ 序列可能振荡（非单调），导致实际收敛慢于理论。解决：adaptive restart（当 $f(x_{k+1}) > f(x_k)$ 时重置动量 $t_k = 1$），实现为简单的 if 判断。
+- **ADMM 的 $\rho$ 参数敏感**：$\rho$ 过大导致子问题病态，$\rho$ 过小导致对偶收敛慢。解决：adaptive $\rho$（根据 primal/dual residual 比值自动调整），每若干步 $\rho \leftarrow \rho \cdot \tau$（$\tau > 1$ 若 primal residual $>$ dual residual）。
+- **近端算子无闭式解**：并非所有 $h(x)$ 都有闭式 prox。复杂正则项（如 TV 正则、重叠 group lasso）需内层迭代求解。解决：用 Dykstra 算法分裂为多个简单 prox 的组合，或改用 ADMM 分裂。
+- **Straight-Through Estimator 的偏差**：量化 prox 的 STE 梯度有偏（$\partial \text{round}/\partial w = 0$ 几乎处处），长期训练可能发散。解决：用 learnable scale factor 补偿，或加噪声的量化（stochastic rounding）。
+
+## 深入参考
+
+- 蒸馏稿：references/books/optimization-ml.md（Ch 8 梯度法 §8.3 收敛分析、Ch 11 拟牛顿 §11.5 BFGS、Ch 24 约束算法 §24.5 增广 Lagrange）
+- 原书：Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 24 (Constrained Algorithms §24.5 Augmented Lagrangian) + Parikh & Boyd, *Proximal Algorithms*, Foundations and Trends in Optimization, 2014

package/knowledge-base/optimization/riemannian-optimization.en.md

@@ -0,0 +1,51 @@
+# Riemannian Optimization
+
+## Minimal Definition
+
+Optimization on smooth manifolds $\mathcal{M}$ (e.g., the orthogonal group $O(n)$, Stiefel manifold, Grassmann manifold, hyperbolic space). Core idea: project the Euclidean gradient onto the tangent space of the manifold, update along geodesics (or retractions), and ensure iterates remain on the manifold.
+
+## Core Formulas
+
+- Riemannian gradient: $\text{grad} f(x) = \text{proj}_{T_x\mathcal{M}}(\nabla f(x))$ (Euclidean gradient projected onto the tangent space)
+- Riemannian gradient descent: $x_{k+1} = R_{x_k}(-\alpha_k \cdot \text{grad} f(x_k))$, where $R$ is a retraction
+- Tangent space of the orthogonal group $O(n)$: $T_Q O(n) = \{Q\Omega : \Omega^T = -\Omega\}$ (left multiplication by skew-symmetric matrices)
+- Cayley retraction for the orthogonal group: $R_Q(\xi) = Q(I + \frac{1}{2}\Omega)^{-1}(I - \frac{1}{2}\Omega)$, $\xi = Q\Omega$
+- Polar decomposition retraction: $R_Q(\xi) = (Q + \xi)(I + \xi^T\xi)^{-1/2}$ (projection onto the nearest orthogonal matrix)
+- Newton-Schulz orthogonalization: $X_{k+1} = \frac{1}{2}X_k(3I - X_k^T X_k)$, converging to the nearest orthogonal matrix
+- Hyperbolic space (Poincaré ball): $\text{grad}_{\mathcal{H}} f = \frac{(1-\|x\|^2)^2}{4} \nabla f(x)$
+
+## Applicable Problems
+
+- Orthogonal weight constraints: $W^TW = I$ preserves eigenvalue moduli at 1, stabilizing RNN/SSM training
+- Muon optimizer: projects the gradient onto the nearest orthogonal matrix as the update direction (a cheap surrogate with "second-order flavor")
+- Low-rank subspace tracking: online PCA on the Grassmann manifold
+- Hyperbolic embeddings: Poincaré embeddings for hierarchical structures (trees, taxonomies)
+- Metric learning: distance metrics on the SPD matrix manifold (covariance matrix space)
+
+## AI Design Translation
+
+- **Muon optimizer (orthogonalized gradient updates)**: Projects the momentum matrix $M$ onto the nearest orthogonal matrix via Newton-Schulz iteration: $X_{k+1} = \frac{1}{2}X_k(3I - X_k^TX_k)$, converging in 5 steps. Update $W \leftarrow W - \alpha \cdot U$ ($U$ being the orthogonalized result). The core is a pure matmul chain, 2 matmul operations per step, fully tensor-core-friendly, stable under bf16.
+- **Manifold perspective on spectral normalization**: The constraint $\sigma_{\max}(W) = 1$ is equivalent to projection onto a tangent direction of the Stiefel manifold. Power iteration for direction estimation + normalization = an approximate Riemannian gradient projection. Implementation is identical to standard spectral norm.
+- **Orthogonal RNN**: Hidden state recurrence $h_t = \sigma(W h_{t-1} + U x_t)$, constraining $W \in O(n)$ to avoid vanishing/exploding gradients. During training, uses the Cayley parameterization $W = (I-A)(I+A)^{-1}$ ($A$ skew-symmetric), with backpropagation computing unconstrained gradients with respect to $A$. The core is matrix inversion $O(n^3)$ (acceptable for small layer dimensions).
+- **Poincaré embeddings (hyperbolic space)**: Embeds hierarchical data into the Poincaré ball $\mathcal{B}^n = \{x : \|x\| < 1\}$. Distance $d(x,y) = \text{arcosh}(1 + 2\|x-y\|^2 / ((1-\|x\|^2)(1-\|y\|^2)))$. The gradient is simply scaled by the metric factor $(1-\|x\|^2)^2/4$. Implemented as elementwise scaling, $O(d)$.
+- **Subspace learning on the Grassmann manifold**: Treats low-rank subspaces as points on the Grassmann manifold, updated online via Riemannian SGD. More suitable for streaming data than SVD. The projection $P = QQ^T$ update is implemented via QR decomposition, with the core being matmul + thin QR.
+
+## Engineering Feasibility
+
+- **Primary operations**: Riemannian gradient projection = matmul ($Q^T \nabla$ to obtain the tangent space component); retraction = matmul + small-matrix inversion / Newton-Schulz (pure matmul); hyperbolic metric = elementwise.
+- **GPU friendliness**: High (Newton-Schulz orthogonalization = pure matmul chain) to moderate (Cayley retraction requires $n \times n$ matrix inversion, where $n$ is the layer dimension; feasible via cuSOLVER when $n \leq 1024$). The metric scaling for hyperbolic embeddings is pure elementwise.
+- **Complexity**: Newton-Schulz per step $O(n^3)$ (where $n$ is the layer dimension, not total model parameters); Cayley $O(n^3)$; hyperbolic gradient $O(d)$; Grassmann QR $O(nd^2)$.
+- **Low precision**: Newton-Schulz is stable under bf16 (pure matmul iteration, no division or square roots); Cayley retraction's matrix inversion may fail under bf16 (fp32 required); the hyperbolic metric's denominator approaches zero as $\|x\| \to 1$, requiring clamping to prevent overflow.
+
+## Risks and Failure Conditions
+
+- **Inversion overhead of Cayley / matrix exponential maps**: $O(n^3)$ matrix inversion per step becomes a bottleneck when layer dimension $> 4096$. Solution: switch to Newton-Schulz orthogonalization (pure matmul) or approximate retractions via polar decomposition.
+- **Numerical overflow in hyperbolic space**: As $\|x\| \to 1$, $d(x,y) \to \infty$ and the metric factor $(1-\|x\|^2)^{-2} \to \infty$, causing gradient explosion. Solution: clamp $\|x\| \leq 1 - \epsilon$ ($\epsilon \sim 10^{-5}$), or use the Lorentz model (a numerically more stable hyperbolic parameterization).
+- **Retraction vs. exponential map**: Retraction is a first-order approximation of the exponential map, with reduced accuracy at large step sizes. For learning-rate-sensitive optimization problems, the true exponential map may be needed (at higher cost).
+- **Unboundedness of non-compact manifolds**: SPD manifolds / hyperbolic space are non-compact, and optimization paths may diverge to infinity. Regularization or trust-region constraints are needed.
+- **Conflict between orthogonal constraints and BatchNorm**: The affine transformation in BatchNorm breaks orthogonality. Either disable BN scale/shift after orthogonally constrained layers, or switch to GroupNorm.
+
+## Further References
+
+- Distilled notes: references/books/matrix-analysis.md (Section 7.3 Polar Decomposition, Newton-Schulz Iteration, Section 2.6 SVD and Orthogonal Factors)
+- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 7 Section 7.3 (Polar Decomposition) + Absil, Mahony, Sepulchre, *Optimization Algorithms on Matrix Manifolds*, Princeton University Press, 2008

package/knowledge-base/optimization/riemannian-optimization.md

@@ -0,0 +1,51 @@
+# 黎曼优化 (Riemannian Optimization)
+
+## 最小定义
+
+在光滑流形 $\mathcal{M}$（如正交群 $O(n)$、Stiefel 流形、Grassmann 流形、双曲空间）上做优化。核心思想：将欧氏空间的梯度投影到流形的切空间，沿测地线（或收缩映射）更新，保证迭代始终在流形上。
+
+## 核心公式
+
+- 黎曼梯度：$\text{grad} f(x) = \text{proj}_{T_x\mathcal{M}}(\nabla f(x))$（欧氏梯度投影到切空间）
+- 黎曼梯度下降：$x_{k+1} = R_{x_k}(-\alpha_k \cdot \text{grad} f(x_k))$，$R$ 为收缩映射（retraction）
+- 正交群 $O(n)$ 的切空间：$T_Q O(n) = \{Q\Omega : \Omega^T = -\Omega\}$（反对称矩阵左乘）
+- 正交群收缩映射（Cayley）：$R_Q(\xi) = Q(I + \frac{1}{2}\Omega)^{-1}(I - \frac{1}{2}\Omega)$，$\xi = Q\Omega$
+- 极分解收缩：$R_Q(\xi) = (Q + \xi)(I + \xi^T\xi)^{-1/2}$（投影到最近正交矩阵）
+- Newton-Schulz 正交化：$X_{k+1} = \frac{1}{2}X_k(3I - X_k^T X_k)$，收敛到最近正交矩阵
+- 双曲空间（Poincaré ball）：$\text{grad}_{\mathcal{H}} f = \frac{(1-\|x\|^2)^2}{4} \nabla f(x)$
+
+## 适用问题
+
+- 正交权重约束：$W^TW = I$ 保持特征值模长为 1，稳定 RNN/SSM 训练
+- Muon 优化器：将梯度投影到最近正交矩阵作为更新方向（"二阶味"的廉价替代）
+- 低秩子空间追踪：Grassmann 流形上的在线 PCA
+- 双曲嵌入：层次结构（树、taxonomy）的 Poincaré 嵌入
+- 度量学习：SPD 矩阵流形（协方差矩阵空间）上的距离度量
+
+## AI 设计翻译
+
+- **Muon 优化器 (正交化梯度更新)**：将动量矩阵 $M$ 通过 Newton-Schulz 迭代投影到最近正交矩阵：$X_{k+1} = \frac{1}{2}X_k(3I - X_k^TX_k)$，5 步收敛。更新 $W \leftarrow W - \alpha \cdot U$（$U$ 为正交化结果）。核心是纯 matmul 链，每步 2 次 matmul，完全 tensor core 友好，bf16 稳定。
+- **谱归一化的流形视角**：约束 $\sigma_{\max}(W) = 1$ 等价于在 Stiefel 流形的某个切方向上做投影。Power iteration 估计方向 + 归一化 = 一种近似的黎曼梯度投影。实现同标准 spectral norm。
+- **正交 RNN (orthogonal RNN)**：隐藏状态递推 $h_t = \sigma(W h_{t-1} + U x_t)$，约束 $W \in O(n)$ 避免梯度消失/爆炸。训练时用 Cayley 参数化 $W = (I-A)(I+A)^{-1}$（$A$ 反对称），反向传播走 $A$ 的无约束梯度。核心是矩阵求逆 $O(n^3)$（层维度小，可接受）。
+- **Poincaré 嵌入 (双曲空间)**：将层次数据嵌入 Poincaré ball $\mathcal{B}^n = \{x : \|x\| < 1\}$。距离 $d(x,y) = \text{arcosh}(1 + 2\|x-y\|^2 / ((1-\|x\|^2)(1-\|y\|^2)))$。梯度乘以度量因子 $(1-\|x\|^2)^2/4$ 即可。实现为 elementwise 缩放，$O(d)$。
+- **Grassmann 流形上的子空间学习**：将低秩子空间视为 Grassmann 流形上的点，用黎曼 SGD 在线更新。比 SVD 更适合 streaming 数据。投影 $P = QQ^T$ 的更新通过 QR 分解实现，核心是 matmul + thin QR。
+
+## 工程可行性
+
+- **主要操作**：黎曼梯度投影 = matmul（$Q^T \nabla$ 得切空间分量）；收缩映射 = matmul + 小矩阵求逆 / Newton-Schulz（纯 matmul）；双曲度量 = elementwise。
+- **GPU 友好度**：高（Newton-Schulz 正交化 = 纯 matmul 链）到中等（Cayley 映射需 $n \times n$ 矩阵求逆，$n$ 为层维度，$n \leq 1024$ 时 cuSOLVER 可行）。双曲嵌入的度量缩放是纯 elementwise。
+- **复杂度**：Newton-Schulz 每步 $O(n^3)$（但 $n$ 为层维度，非模型总参数量）；Cayley $O(n^3)$；双曲梯度 $O(d)$；Grassmann QR $O(nd^2)$。
+- **低精度**：Newton-Schulz 在 bf16 下稳定（纯 matmul 迭代，不涉及除法/开方）；Cayley 映射的求逆在 bf16 下可能失败（需 fp32）；双曲度量在 $\|x\| \to 1$ 时分母趋零，需 clamp 防溢出。
+
+## 风险与失效条件
+
+- **Cayley / 矩阵指数映射的求逆开销**：每步 $O(n^3)$ 矩阵求逆，层维度 $> 4096$ 时成为瓶颈。解决：改用 Newton-Schulz 正交化（纯 matmul）或 polar decomposition 的近似收缩。
+- **双曲空间的数值溢出**：$\|x\| \to 1$ 时 $d(x,y) \to \infty$，度量因子 $(1-\|x\|^2)^{-2} \to \infty$，梯度爆炸。解决：clamp $\|x\| \leq 1 - \epsilon$（$\epsilon \sim 10^{-5}$），或用 Lorentz 模型（数值更稳定的双曲参数化）。
+- **收缩映射 vs. 指数映射**：收缩映射（retraction）是指数映射的一阶近似，大步长时精度下降。对学习率敏感的优化问题，可能需要真正的指数映射（更贵）。
+- **非紧凑流形的无界性**：SPD 流形 / 双曲空间非紧，优化路径可能跑到无穷远。需加正则化或信赖域约束。
+- **正交约束与 BatchNorm 冲突**：BatchNorm 的仿射变换破坏正交性。需在正交约束层后禁用 BN 的 scale/shift，或改用 GroupNorm。
+
+## 深入参考
+
+- 蒸馏稿：references/books/matrix-analysis.md（§7.3 极分解、Newton-Schulz 迭代、§2.6 SVD 与正交因子）
+- 原书：Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 7 §7.3 (Polar Decomposition) + Absil, Mahony, Sepulchre, *Optimization Algorithms on Matrix Manifolds*, Princeton University Press, 2008