math-skill 2.1.0 → 3.0.0

package/knowledge-base/matrix-analysis/low-rank-approximation.md

@@ -0,0 +1,49 @@
+# 低秩逼近 (Low-Rank Approximation)
+
+## 最小定义
+
+给定矩阵 $A \in \mathbb{R}^{m \times n}$，寻找秩不超过 $k$ 的矩阵 $B$ 使得 $\|A - B\|$ 最小。Eckart-Young-Mirsky 定理保证截断 SVD 给出 Frobenius 范数和谱范数下的唯一最优解：$B_k = \sum_{i=1}^k \sigma_i u_i v_i^H$。
+
+## 核心公式
+
+- SVD：$A = U\Sigma V^H = \sum_{i=1}^r \sigma_i u_i v_i^H$，$\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r > 0$
+- 截断 SVD（最优秩-$k$ 逼近）：$A_k = U_k \Sigma_k V_k^H$
+- Eckart-Young 误差：$\|A - A_k\|_F = \sqrt{\sum_{i=k+1}^r \sigma_i^2}$，$\|A - A_k\|_2 = \sigma_{k+1}$
+- 随机化 SVD：$A \approx Q(Q^HA)$，$Q$ 为 $A\Omega$（$\Omega$ 随机高斯）的 QR 分解之 $Q$ 因子
+- 有效秩：$r_{\text{eff}}(A) = \|A\|_F^2 / \|A\|_2^2 = \sum \sigma_i^2 / \sigma_1^2$
+- 核范数（秩的凸松弛）：$\|A\|_* = \sum \sigma_i$，是谱范数的对偶
+
+## 适用问题
+
+- LoRA 权重压缩：$W \approx W_0 + BA$，$B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times d}$，$r \ll d$
+- KV-Cache 压缩：将 Key/Value 缓存投影到低维子空间，显存 $O(n) \to O(k)$
+- PCA / 白化：数据协方差的前 $k$ 个主成分即截断 SVD
+- 梯度压缩：大模型梯度矩阵的有效秩通常远低于名义秩，可安全截断
+- 推荐系统 / 矩阵补全：低秩因子分解 $R \approx UV^H$
+
+## AI 设计翻译
+
+- **LoRA (Low-Rank Adaptation)**：冻结 $W_0$，训练 $\Delta W = BA$（$r \ll d$），推理时合并 $W = W_0 + BA$。前向传播 = 两次 matmul（$x \to Ax \to BAx$），训练参数量从 $O(d^2)$ 降到 $O(dr)$。用 `torch.mm(B, torch.mm(A, x))` 或合并为单次 matmul。
+- **随机化 SVD 算子**：对大矩阵 $A \in \mathbb{R}^{m \times n}$，先采样 $Y = A\Omega$（$\Omega \in \mathbb{R}^{n \times (k+p)}$ 随机高斯），QR 分解 $Y = QR$，再算 $B = Q^HA$（小矩阵 $O(k \times n)$），对 $B$ 做 SVD。总复杂度 $O(mnk)$ 而非 $O(mn^2)$，核心操作全是 matmul。
+- **KV-Cache 低秩化**：维护 $K_k = K P_k$（$P_k$ 为投影到前 $k$ 主成分），每新到 token 做增量 PCA 或 streaming SVD 更新。Attention 计算 $\text{softmax}(Q K_k^H / \sqrt{d}) V_k$，三次 matmul，序列维度从 $L$ 降到 $k$。
+- **核范数正则化**：$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \|W\|_*$ 促进低秩解。但核范数计算需完整 SVD（$O(n^3)$），替代方案：(1) 用截断 SVD 近似；(2) 因子化 $\|W\|_* = \min_{W=UV^H} \frac{1}{2}(\|U\|_F^2 + \|V\|_F^2)$ 转为对 $U, V$ 的 Frobenius 正则。
+- **梯度低秩压缩 (分布式训练)**：将梯度 $G$ 截断为 $G_k$（top-$k$ SVD）后 all-reduce 通信量从 $O(d)$ 降到 $O(kd)$。用随机化 SVD 在每卡本地算，再合并。
+
+## 工程可行性
+
+- **主要操作**：matmul + 小矩阵 SVD。LoRA 前向 = 两次 matmul；随机化 SVD = 三次 matmul + 一次小 QR；截断 SVD = 完整 SVD 的 $O(k/n)$ 倍（用 Lanczos）。
+- **GPU 友好度**：极高。LoRA 前向/反向全是 tensor core matmul；随机化 SVD 的主要开销也是 matmul。小矩阵 SVD 有 cuSOLVER 的 batched 实现。
+- **复杂度**：LoRA 前向 $O(dk)$ per sample vs. $O(d^2)$ 全秩；随机化 SVD $O(mnk)$；完整 SVD $O(\min(m^2n, mn^2))$。
+- **显存**：LoRA 存储 $O(dr)$ vs. $O(d^2)$；KV-Cache 低秩化 $O(Lk)$ vs. $O(Ld)$。
+
+## 风险与失效条件
+
+- **秩选择错误**：$k$ 过小导致信息丢失（$\sigma_{k+1}$ 不可忽略），$k$ 过大失去压缩意义。解决：监控奇异值衰减曲线，选 $\sum_{i>k}\sigma_i^2 / \sum\sigma_i^2 < \epsilon$ 的拐点。
+- **随机化 SVD 精度不足**：oversampling $p$ 太小（通常 $p = 5 \sim 10$）或 power iteration 次数不足时，低阶奇异值估计偏差大。解决：增加 $q = 1 \sim 2$ 步 power iteration $Y = (AA^H)^q A\Omega$，但增加 matmul 次数。
+- **LoRA 不适用于所有层**：Attention 的 Q/K/V 通常低秩有效，但 FFN 层和 embedding 的有效秩可能接近满秩，强行 LoRA 会损失精度。需逐层诊断有效秩。
+- **核范数 proximal 的 SVD 开销**：soft-thresholding $\text{prox}_{\lambda\|\cdot\|_*}(A) = U(\Sigma - \lambda I)_+ V^H$ 需要 SVD，大矩阵每步算不起。解决：用因子化替代或随机化近似。
+
+## 深入参考
+
+- 蒸馏稿：references/books/matrix-analysis.md（§2.6 SVD、§7.4 极分解与 SVD、核范数-谱范数对偶 §5.5）
+- 原书：Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 2 §2.6 (SVD) + Chapter 7 §7.3-7.4 (Polar Decomposition & SVD)

package/knowledge-base/matrix-analysis/matrix-perturbation.en.md

@@ -0,0 +1,49 @@
+# Matrix Perturbation Theory
+
+## Minimal Definition
+
+Studies how the eigenvalues, singular values, and eigenspaces of a matrix $A$ change under a small perturbation $E$. Core results: eigenvalues of Hermitian matrices are Lipschitz continuous with respect to perturbation (Weyl bound), while the stability of eigenvectors/eigenspaces is governed by the eigengap.
+
+## Core Formulas
+
+- **Weyl eigenvalue perturbation bound** (Hermitian): $|\lambda_i(A+E) - \lambda_i(A)| \leq \|E\|_2$
+- **Bauer-Fike theorem** (diagonalizable matrices): $\min_j |\lambda_i(A+E) - \lambda_j(A)| \leq \kappa(V) \|E\|_2$, where $V$ is the eigenvector matrix
+- **Davis-Kahan $\sin\Theta$ theorem**: $\|\sin\Theta(\hat{U}, U)\|_2 \leq \frac{\|E\|_2}{\delta}$, where $\delta$ is the gap between the subspace and the rest of the spectrum
+- **Singular value perturbation (Mirsky)**: $|\sigma_i(A+E) - \sigma_i(A)| \leq \|E\|_2$
+- **Geršgorin discs**: $\lambda_i(A) \in \bigcup_j \{z : |z - a_{jj}| \leq \sum_{k \neq j} |a_{jk}|\}$
+- **Condition number and relative error**: $\frac{|\delta x|}{|x|} \leq \kappa(A) \frac{\|\delta A\|}{\|A\|}$
+
+## Applicable Problems
+
+- Spectral drift analysis under quantization/low-precision training: how much do singular values of weight matrices shift under bf16/fp8?
+- Error bounds for model pruning/distillation: how large is the spectral change after removing $k$ parameters?
+- LoRA approximation error: how does the perturbation $\|W - W_0 - BA\|_2$ affect downstream outputs?
+- Training stability certificates: when gradient noise $\|E\|_2 \leq \epsilon$, eigenvalue drift is bounded by $\epsilon$
+- Numerical diagnostics: cheaply estimating spectral location via Geršgorin discs without running full EVD
+
+## AI Design Translation
+
+- **Geršgorin cheap spectral radius monitoring**: Every $N$ steps in the training loop, compute $\rho_{\text{est}} = \max_j (|a_{jj}| + \sum_{k\neq j}|a_{jk}|)$ as an upper bound on the spectral radius, requiring only $O(n^2)$ row-wise absolute-value summation. Implemented as `torch.sum(torch.abs(A), dim=1)`, an elementwise + reduce operation, extremely cheap, and embeddable in the training loop as a stability gate.
+- **Spectral drift bound for quantization error**: $W_{\text{quant}} = W + E$, $\|E\|_2 \leq \epsilon$; by the Weyl theorem, $\sigma_i$ shifts by $\leq \epsilon$. For an $L$-layer network, output perturbation $\leq \prod_i (\sigma_1(W_i) + \epsilon) - \prod_i \sigma_1(W_i)$. This guides quantization precision selection: if $\sigma_{\min}(W)$ is close to $\epsilon$, that layer requires higher precision.
+- **Davis-Kahan subspace stability**: The reliability of subspaces in PCA/LoRA is determined by the eigengap $\delta = \lambda_k - \lambda_{k+1}$. Larger $\delta$ yields a more stable truncated subspace; as $\delta \to 0$, the subspace becomes extremely sensitive to noise. This serves as a diagnostic tool for selecting the LoRA rank $r$: choose $r$ that maximizes $\delta_r$.
+- **Perturbation modeling for pruning**: Unstructured pruning = sparse perturbation $E$, $\|E\|_2 \leq \|E\|_F = \sqrt{\sum e_{ij}^2}$. The Weyl bound provides an upper bound on spectral drift, guiding the pruning ratio: maintain $\|E\|_2 / \sigma_1(W) < \tau$ (e.g., $\tau = 0.05$).
+- **Spectral regularization as a robustness certificate**: $\mathcal{L}_{\text{robust}} = \mathcal{L}_{\text{task}} + \lambda \max(0, \|E\|_2 - \epsilon)^2$, constraining spectral drift under perturbation. Combined with power iteration to estimate $\|E\|_2$, implemented as an additional loss term.
+
+## Engineering Feasibility
+
+- **Primary operations**: Geršgorin = elementwise abs + row-sum ($O(n^2)$); Weyl bound only requires $\|E\|_2$ (power iteration $O(n^2)$/step); Davis-Kahan requires eigengap (partial EVD, $O(n^2 k)$).
+- **GPU friendliness**: High. Geršgorin is pure elementwise + reduce; $\|E\|_2$ estimation is a matvec chain; eigengap uses the Lanczos algorithm (matmul + tridiagonal EVD). All operations support batching.
+- **Complexity**: Geršgorin $O(n^2)$; single power iteration $O(n^2)$; Lanczos $k$ steps $O(kn^2)$; full EVD $O(n^3)$ (should be avoided).
+- **Low precision**: The Weyl bound is itself Lipschitz; the estimation of $\|E\|_2$ under low precision has $\sim \sqrt{n} \cdot \text{eps}$ floating-point noise, which is typically negligible.
+
+## Risks and Failure Conditions
+
+- **Bauer-Fike amplification for non-normal matrices**: $\kappa(V)$ can be extremely large (high condition number of the eigenvector matrix for non-normal matrices), the Weyl bound no longer applies, and perturbations are amplified by a factor of $\kappa(V)$. Solution: use SVD singular values instead (the Mirsky bound does not depend on normality), or employ pseudospectral analysis.
+- **Overly loose Geršgorin bounds**: The union of discs may be much larger than the actual spectral range (especially for sparse matrices), yielding overly conservative bounds. Solution: apply a diagonal similarity transformation $D^{-1}AD$ to shrink the discs (Osborne balancing), or incorporate sparse structure corrections.
+- **Davis-Kahan gap assumption**: As $\delta \to 0$, the bound degenerates to $\infty$ (subspace becomes unidentifiable), and the low-rank approximation itself is no longer unique. The eigengap must be checked first to confirm that the subspace is well-defined.
+- **Weyl bound not directly applicable to non-Hermitian components**: The perturbation bound for $A + A^H$ does not directly yield eigenvalue perturbation bounds for $A$ itself. For non-symmetric matrices, one must revert to Bauer-Fike or pseudospectral analysis.
+
+## Further References
+
+- Distilled notes: references/books/matrix-analysis.md (Section 4.3 Eigenvalue Inequalities Weyl/Interlacing, Section 6.1-6.3 Geršgorin Discs and Perturbation Theorems, Section 5.8 Condition Numbers)
+- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 4 Section 4.3 (Eigenvalue Inequalities) + Chapter 6 (Location and Perturbation of Eigenvalues Section 6.1-6.3)

package/knowledge-base/matrix-analysis/matrix-perturbation.md

@@ -0,0 +1,49 @@
+# 矩阵扰动 (Matrix Perturbation Theory)
+
+## 最小定义
+
+研究矩阵 $A$ 在受到小扰动 $E$ 后，其特征值、奇异值、特征子空间如何变化。核心结论：Hermitian 矩阵的特征值对扰动是 Lipschitz 连续的（Weyl 界），而特征向量/子空间的稳定性由特征值间隙（eigengap）决定。
+
+## 核心公式
+
+- **Weyl 特征值扰动界**（Hermitian）：$|\lambda_i(A+E) - \lambda_i(A)| \leq \|E\|_2$
+- **Bauer-Fike 定理**（可对角化矩阵）：$\min_j |\lambda_i(A+E) - \lambda_j(A)| \leq \kappa(V) \|E\|_2$，$V$ 为特征向量矩阵
+- **Davis-Kahan $\sin\Theta$ 定理**：$\|\sin\Theta(\hat{U}, U)\|_2 \leq \frac{\|E\|_2}{\delta}$，$\delta$ 为子空间与其余谱的间隙
+- **奇异值扰动 (Mirsky)**：$|\sigma_i(A+E) - \sigma_i(A)| \leq \|E\|_2$
+- **Ger\v{s}gorin 圆盘**：$\lambda_i(A) \in \bigcup_j \{z : |z - a_{jj}| \leq \sum_{k \neq j} |a_{jk}|\}$
+- **条件数与相对误差**：$\frac{|\delta x|}{|x|} \leq \kappa(A) \frac{\|\delta A\|}{\|A\|}$
+
+## 适用问题
+
+- 量化/低精度训练的谱漂移分析：bf16/fp8 下权重矩阵的奇异值偏移多少？
+- 模型剪枝/蒸馏的误差界：删掉 $k$ 个参数后谱变化有多大？
+- LoRA 近似误差：$\|W - W_0 - BA\|_2$ 的扰动如何影响下游输出？
+- 训练稳定性证书：梯度噪声 $\|E\|_2 \leq \epsilon$ 时特征值漂移不超过 $\epsilon$
+- 数值诊断：不跑完整 EVD，用 Ger\v{s}gorin 圆盘廉价估计谱位置
+
+## AI 设计翻译
+
+- **Ger\v{s}gorin 廉价谱半径监控**：训练循环中每 $N$ 步计算 $\rho_{\text{est}} = \max_j (|a_{jj}| + \sum_{k\neq j}|a_{jk}|)$ 作为谱半径上界，仅 $O(n^2)$ 逐行求绝对值和。实现为 `torch.sum(torch.abs(A), dim=1)`，是 elementwise + reduce，极廉价，可嵌入训练 loop 做稳定性 gate。
+- **量化误差的谱漂移界**：$W_{\text{quant}} = W + E$，$\|E\|_2 \leq \epsilon$，由 Weyl 定理 $\sigma_i$ 偏移 $\leq \epsilon$。对 $L$ 层网络，输出扰动 $\leq \prod_i (\sigma_1(W_i) + \epsilon) - \prod_i \sigma_1(W_i)$。指导量化精度选择：若 $\sigma_{\min}(W)$ 接近 $\epsilon$，则该层需更高精度。
+- **Davis-Kahan 子空间稳定性**：PCA/LoRA 中子空间的可靠性由 eigengap $\delta = \lambda_k - \lambda_{k+1}$ 决定。$\delta$ 越大，截断子空间越稳定；$\delta \to 0$ 时子空间对噪声极度敏感。可作为选择 LoRA rank $r$ 的诊断工具：选 $\delta_r$ 最大的 $r$。
+- **剪枝的扰动建模**：非结构化剪枝 = 稀疏扰动 $E$，$\|E\|_2 \leq \|E\|_F = \sqrt{\sum e_{ij}^2}$。Weyl 界给出谱漂移上界，指导剪枝比例：保持 $\|E\|_2 / \sigma_1(W) < \tau$（如 $\tau = 0.05$）。
+- **谱正则化作为鲁棒性证书**：$\mathcal{L}_{\text{robust}} = \mathcal{L}_{\text{task}} + \lambda \max(0, \|E\|_2 - \epsilon)^2$，约束扰动下的谱漂移。结合 power iteration 估计 $\|E\|_2$，实现为附加 loss 项。
+
+## 工程可行性
+
+- **主要操作**：Ger\v{s}gorin = elementwise abs + row-sum（$O(n^2)$）；Weyl 界只需 $\|E\|_2$（power iteration $O(n^2)$/step）；Davis-Kahan 需 eigengap（需部分 EVD，$O(n^2 k)$）。
+- **GPU 友好度**：高。Ger\v{s}gorin 是纯 elementwise + reduce；$\|E\|_2$ 估计是 matvec 链；eigengap 用 Lanczos 算法（matmul + tridiag EVD）。所有操作均可 batch。
+- **复杂度**：Ger\v{s}gorin $O(n^2)$；单次 power iteration $O(n^2)$；Lanczos $k$ 步 $O(kn^2)$；完整 EVD $O(n^3)$（应避免）。
+- **低精度**：Weyl 界本身是 Lipschitz 的，低精度下扰动 $\|E\|_2$ 的估计有 $\sim \sqrt{n} \cdot \text{eps}$ 的浮点噪声，通常可忽略。
+
+## 风险与失效条件
+
+- **非正规矩阵的 Bauer-Fike 放大**：$\kappa(V)$ 可能极大（非正规矩阵的特征向量矩阵条件数高），Weyl 界不再适用，扰动被放大 $\kappa(V)$ 倍。解决：改用 SVD 奇异值（Mirsky 界不依赖正规性），或用 pseudospectra 分析。
+- **Ger\v{s}gorin 过松**：圆盘并集可能远大于实际谱范围（尤其对稀疏矩阵），给出过于保守的界。解决：先做对角相似变换 $D^{-1}AD$ 压缩圆盘（Osborne 平衡），或结合稀疏结构修正。
+- **Davis-Kahan 的间隙假设**：$\delta \to 0$ 时界退化为 $\infty$（子空间不可辨识），此时低秩逼近本身不唯一。需先检测 eigengap，确认子空间良定义。
+- **Weyl 界对非 Hermitian 部分不直接适用**：$A + A^H$ 的扰动界不直接给出 $A$ 本身的特征值扰动。对非对称矩阵需回到 Bauer-Fike 或 pseudospectra。
+
+## 深入参考
+
+- 蒸馏稿：references/books/matrix-analysis.md（§4.3 特征值不等式 Weyl/交错、§6.1-6.3 Ger\v{s}gorin 圆盘与扰动定理、§5.8 条件数）
+- 原书：Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 4 §4.3 (Eigenvalue Inequalities) + Chapter 6 (Location and Perturbation of Eigenvalues §6.1-6.3)

package/knowledge-base/matrix-analysis/positive-semidefinite.en.md

@@ -0,0 +1,50 @@
+# Positive Semidefinite Matrices
+
+## Minimal Definition
+
+A Hermitian matrix $A$ is positive semidefinite (PSD) if $x^HAx \geq 0$ for all nonzero vectors $x$, denoted $A \succeq 0$. Equivalent conditions: all eigenvalues $\geq 0$; there exists $B$ such that $A = B^HB$ (Gram representation); all leading principal minors $\geq 0$. Positive definite (PD) requires strict $> 0$, denoted $A \succ 0$.
+
+## Core Formulas
+
+- PSD equivalence: $A \succeq 0 \iff \lambda_i(A) \geq 0 \ \forall i \iff A = B^HB$
+- PD equivalence: $A \succ 0 \iff \lambda_i(A) > 0 \ \forall i \iff$ all leading principal minors $> 0$ (Sylvester's criterion)
+- Cholesky decomposition: $A \succ 0 \implies A = LL^H$, $L$ lower triangular
+- Loewner partial order: $A \succeq B \iff A - B \succeq 0$
+- Schur product theorem: $A \succeq 0, B \succeq 0 \implies A \circ B \succeq 0$ (Hadamard product preserves PSD)
+- Simultaneous diagonalization: $A, B \succ 0 \implies \exists C$ such that $C^HAC = I, C^HBC = \Lambda$
+- Polar decomposition: $A = UP$, $P = (A^HA)^{1/2} \succeq 0$
+
+## Applicable Problems
+
+- Kernel methods: The Gram matrix $K_{ij} = k(x_i, x_j)$ must be PSD to guarantee the existence of an RKHS
+- Covariance matrices: $\Sigma = \mathbb{E}[xx^H] \succeq 0$; PCA/whitening relies on positive definiteness
+- Second-order optimization preconditioning: The PSD structure of the Hessian/Fisher information matrix guarantees descent directions
+- Semidefinite programming (SDP): optimizing a linear objective subject to $X \succeq 0$ constraints
+- Attention matrix analysis: Gram structure of the row-stochastic matrix output by softmax
+
+## AI Design Translation
+
+- **PSD kernel engineering (learnable kernels)**: Use the Schur product theorem to combine multiple PSD kernels: $K = K_1 \circ K_2 \circ \cdots$ (Hadamard product), guaranteeing the result remains PSD. Implemented as elementwise tensor multiplication `K = K1 * K2`, $O(n^2)$ elementwise, extremely GPU-friendly. Can parameterize $K_\theta(x,y) = \exp(-\|f_\theta(x)-f_\theta(y)\|^2)$ to guarantee PSD.
+- **Covariance whitening**: $\hat{x} = \Sigma^{-1/2}x$, where $\Sigma^{-1/2}$ is approximated via Newton-Schulz iteration (pure matmul). Newton-Schulz: $X_{k+1} = \frac{1}{2}X_k(3I - AX_k)$, two matmul operations per step, converging in 5-6 steps. BatchNorm can be viewed as an approximation to diagonal whitening.
+- **Cholesky preconditioner**: For a PSD Hessian $H$, use the $H = LL^H$ decomposition and solve $L^{-1}L^{-H}g$ instead of $H^{-1}g$. cuSOLVER provides batched Cholesky `potrf`. In K-FAC, the inversion of each Kronecker factor proceeds via Cholesky.
+- **Nearest PSD approximation (Higham)**: Given a symmetric matrix $A$, find the nearest PSD matrix $A_+ = \arg\min_{X \succeq 0} \|A - X\|_F$. Solution: EVD $A = U\Lambda U^H$, clamp negative values in $\Lambda$ to zero, $A_+ = U\Lambda_+ U^H$. Used to correct loss of positive definiteness in covariance matrices due to floating-point errors.
+- **Jitter / diagonal loading**: $A_{\text{stable}} = A + \epsilon I$ ($\epsilon \sim 10^{-6}$), ensuring numerical positive definiteness. Standard practice in Gaussian processes, kernel methods, and Cholesky decomposition. Implemented as `A + eps * torch.eye(n)`, a zero-cost operation.
+
+## Engineering Feasibility
+
+- **Primary operations**: Cholesky decomposition $O(n^3/3)$ (cuSOLVER batched available); Gram matrix construction $O(n^2d)$ (matmul); Newton-Schulz iteration $O(n^3)$/step (pure matmul); Hadamard product $O(n^2)$ (elementwise).
+- **GPU friendliness**: High. Gram matrix = matmul; Hadamard product = elementwise; Newton-Schulz = pure matmul chain; Cholesky has cuSOLVER batched versions for parallel execution across multiple groups.
+- **Complexity**: Gram construction $O(n^2d)$; Cholesky $O(n^3/3)$; Newton-Schulz 5 steps $O(5n^3)$; jitter $O(n)$.
+- **Low precision**: Cholesky may fail under bf16 (diagonal entries becoming negative); jitter must be added or the decomposition performed in fp32. Newton-Schulz is stable under bf16 (since it involves pure matmul without division).
+
+## Risks and Failure Conditions
+
+- **Loss of positive definiteness in floating point**: Covariance/Gram matrices may lose PSD property under bf16 ($\lambda_{\min} < 0$), causing Cholesky to fail outright. Solution: add jitter $\epsilon I$, compute the decomposition in fp32, or use Newton-Schulz (which does not involve square roots or division).
+- **Near-singularity**: As $\lambda_{\min} \to 0$, the condition number $\kappa \to \infty$, and the entries of $A^{-1}$ blow up in magnitude. Solution: truncate small eigenvalues (spectral cutoff) or apply ridge regularization $A + \lambda I$.
+- **Misuse of the Schur product theorem**: $A \circ B \succeq 0$ requires **both** $A$ and $B$ to be PSD; if either is not PSD, the result is not guaranteed. Each factor must be verified for PSD property in learnable kernel design.
+- **Non-differentiable SDP solvers**: Interior-point SDP solvers (e.g., SCS, MOSEK) cannot be integrated into the gradient graph and do not support end-to-end training. Solution: use a differentiable PSD projection layer (EVD + truncation + reconstruction) as a replacement.
+
+## Further References
+
+- Distilled notes: references/books/matrix-analysis.md (Ch 7 Positive Definite and Semidefinite Matrices, Section 7.5 Schur Product Theorem, Section 7.7 Loewner Partial Order)
+- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 7 (Positive Definite and Semidefinite Matrices Section 7.1-7.8)

package/knowledge-base/matrix-analysis/positive-semidefinite.md

@@ -0,0 +1,50 @@
+# 半正定矩阵 (Positive Semidefinite Matrices)
+
+## 最小定义
+
+Hermitian 矩阵 $A$ 若对所有非零向量 $x$ 满足 $x^HAx \geq 0$，则称半正定（PSD），记 $A \succeq 0$。等价条件：所有特征值 $\geq 0$；存在 $B$ 使 $A = B^HB$（Gram 表示）；所有顺序主子式 $\geq 0$。正定（PD）要求严格 $> 0$，记 $A \succ 0$。
+
+## 核心公式
+
+- PSD 等价条件：$A \succeq 0 \iff \lambda_i(A) \geq 0 \ \forall i \iff A = B^HB$
+- 正定等价：$A \succ 0 \iff \lambda_i(A) > 0 \ \forall i \iff$ 所有顺序主子式 $> 0$（Sylvester 准则）
+- Cholesky 分解：$A \succ 0 \implies A = LL^H$，$L$ 下三角
+- Loewner 偏序：$A \succeq B \iff A - B \succeq 0$
+- Schur 积定理：$A \succeq 0, B \succeq 0 \implies A \circ B \succeq 0$（Hadamard 积保 PSD）
+- 同时对角化：$A, B \succ 0 \implies \exists C$ 使 $C^HAC = I, C^HBC = \Lambda$
+- 极分解：$A = UP$，$P = (A^HA)^{1/2} \succeq 0$
+
+## 适用问题
+
+- 核方法：Gram 矩阵 $K_{ij} = k(x_i, x_j)$ 必须 PSD 才能保证 RKHS 存在
+- 协方差矩阵：$\Sigma = \mathbb{E}[xx^H] \succeq 0$，PCA/白化依赖其正定性
+- 二阶优化预条件：Hessian/Fisher 信息矩阵的 PSD 结构保证下降方向
+- 半定规划 (SDP)：约束 $X \succeq 0$ 下的线性目标优化
+- Attention 矩阵分析：softmax 输出的行随机矩阵的 Gram 结构
+
+## AI 设计翻译
+
+- **PSD 核工程 (可学习核)**：用 Schur 积定理组合多个 PSD 核：$K = K_1 \circ K_2 \circ \cdots$（Hadamard 积），保证结果始终 PSD。实现为 elementwise 张量乘 `K = K1 * K2`，$O(n^2)$ elementwise，极 GPU 友好。可参数化 $K_\theta(x,y) = \exp(-\|f_\theta(x)-f_\theta(y)\|^2)$ 保证 PSD。
+- **协方差白化 (Whitening)**：$\hat{x} = \Sigma^{-1/2}x$，其中 $\Sigma^{-1/2}$ 通过 Newton-Schulz 迭代近似（纯 matmul）。Newton-Schulz：$X_{k+1} = \frac{1}{2}X_k(3I - AX_k)$，每步两次 matmul，5-6 步收敛。BatchNorm 可视为对角白化的近似。
+- **Cholesky 预条件子**：对 PSD Hessian $H$，用 $H = LL^H$ 分解后解 $L^{-1}L^{-H}g$ 代替 $H^{-1}g$。cuSOLVER 有 batched Cholesky `potrf`。K-FAC 中每个 Kronecker 因子的求逆即走 Cholesky。
+- **最近 PSD 近似 (Higham)**：给定对称矩阵 $A$，求最近 PSD 矩阵 $A_+ = \arg\min_{X \succeq 0} \|A - X\|_F$。解法：EVD $A = U\Lambda U^H$，将 $\Lambda$ 中负值截零，$A_+ = U\Lambda_+ U^H$。用于修正浮点误差导致的协方差矩阵失去正定性。
+- **Jitter / 对角加载**：$A_{\text{stable}} = A + \epsilon I$（$\epsilon \sim 10^{-6}$），保证数值正定性。高斯过程、核方法、Cholesky 分解中标准做法。实现为 `A + eps * torch.eye(n)`，零成本操作。
+
+## 工程可行性
+
+- **主要操作**：Cholesky 分解 $O(n^3/3)$（有 cuSOLVER batched）；Gram 矩阵构造 $O(n^2d)$（matmul）；Newton-Schulz 迭代 $O(n^3)$/step（纯 matmul）；Hadamard 积 $O(n^2)$（elementwise）。
+- **GPU 友好度**：高。Gram 矩阵 = matmul；Hadamard 积 = elementwise；Newton-Schulz = 纯 matmul 链；Cholesky 有 cuSOLVER batched 版本，可并行多组。
+- **复杂度**：构造 Gram $O(n^2d)$；Cholesky $O(n^3/3)$；Newton-Schulz 5 步 $O(5n^3)$；Jitter $O(n)$。
+- **低精度**：bf16 下 Cholesky 可能失败（对角元变负），必须加 jitter 或用 fp32 做分解。Newton-Schulz 在 bf16 下稳定（因为纯 matmul 不涉除法）。
+
+## 风险与失效条件
+
+- **浮点丢正定性**：协方差/Gram 矩阵在 bf16 下可能失去 PSD 性（$\lambda_{\min} < 0$），Cholesky 直接崩溃。解决：加 jitter $\epsilon I$，或用 fp32 计算分解，或用 Newton-Schulz（不涉及开方/除法）。
+- **近奇异性**：$\lambda_{\min} \to 0$ 时条件数 $\kappa \to \infty$，$A^{-1}$ 的元素量级爆炸。解决：截断小特征值（spectral cutoff）或用岭正则 $A + \lambda I$。
+- **Schur 积定理的滥用**：$A \circ B \succeq 0$ 要求 $A, B$ **都** PSD；若其中一个非 PSD，结果不保证。可学习核设计中需逐因子验证 PSD 性。
+- **SDP 求解器不可微**：半定规划的内点法求解器（如 SCS, MOSEK）不可嵌入梯度图，不能直接做端到端训练。解决：用可微的 PSD 投影层（EVD + 截断 + 重构）替代。
+
+## 深入参考
+
+- 蒸馏稿：references/books/matrix-analysis.md（Ch 7 Positive Definite and Semidefinite Matrices、§7.5 Schur 积定理、§7.7 Loewner 偏序）
+- 原书：Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 7 (Positive Definite and Semidefinite Matrices §7.1-7.8)

package/knowledge-base/matrix-analysis/projection.en.md

@@ -0,0 +1,47 @@
+# Orthogonal Projection
+
+## Minimal Definition
+
+Maps a vector $v$ onto a subspace $\mathcal{S}$ such that the residual $v - Pv$ is orthogonal to $\mathcal{S}$. The projection matrix $P$ satisfies $P^2 = P$ and $P = P^H$ (idempotent + Hermitian), and is the unique linear operator that determines the nearest point.
+
+## Core Formulas
+
+- Projection matrix: $P = A(A^HA)^{-1}A^H$, where the columns of $A$ span the subspace $\mathcal{S}$
+- If the columns of $A$ are orthonormal ($A^HA = I$), then $P = AA^H$
+- Nearest distance after projection: $\|v - Pv\|^2 = \|v\|^2 - \|A^Hv\|^2$
+- Courant-Fischer variational characterization: $\lambda_k = \max_{\dim(S)=k} \min_{x \in S, \|x\|=1} x^HAx$
+- Orthogonal complement projection: $P^\perp = I - P$
+
+## Applicable Problems
+
+- Least squares regression: $\hat{x} = \arg\min \|Ax - b\|^2$ is equivalent to projecting $b$ onto $\text{Col}(A)$
+- PCA dimensionality reduction: projecting data onto the subspace spanned by the top $k$ principal components
+- Feasible directions in constrained optimization: projecting the gradient onto the constraint tangent space (projected gradient method)
+- Orthogonal decomposition in residual networks: decomposing a signal into an explained component + residual
+
+## AI Design Translation
+
+- **Low-rank bottleneck analysis for linear layers**: The truncation $W = U_k \Sigma_k V_k^H$ is a projection onto a rank-$k$ subspace; implemented via `torch.mm(U_k, torch.mm(Sigma_k, V_k.t()))` as three matmul steps, reducing memory from $O(mn)$ to $O(k(m+n))$
+- **Subspace projection in attention heads**: The Q/K/V matrices essentially project inputs into different subspaces for similarity computation; multi-head = parallel projection subspaces, directly implementable as batched matmul
+- **Projection head (contrastive learning)**: The projection head in SimCLR/MoCo = a multi-layer MLP followed by $L_2$-normalize, equivalent to projecting onto the unit sphere; implemented as `F.normalize(self.mlp(x), dim=-1)`, an elementwise norm operation
+- **Gradient projection / Orthogonal Gradient Descent (OGD)**: In continual learning, projecting the new-task gradient onto the orthogonal complement of the old-task gradient space to avoid catastrophic forgetting; requires maintaining a basis matrix $G$ and computing $(I - G(G^TG)^{-1}G^T)\nabla$, with the core being two matmul operations
+- **Orthogonal residuals in ResNet**: The residual connection $x + F(x)$ can be interpreted as a projection onto the identity subspace + orthogonal correction; spectral normalization constrains the Lipschitz constant of $F$ to stabilize the projection
+
+## Engineering Feasibility
+
+- **Primary operations**: Matrix multiplication (matmul) + matrix inversion/Cholesky. When the subspace dimension $k$ is much smaller than $n$, $(A^HA)^{-1}$ only requires inverting a $k \times k$ matrix, at negligible cost.
+- **GPU friendliness**: High. $P = AA^H$ (orthonormal columns case) is two matmul operations, perfectly mapped to tensor cores. The Cholesky decomposition for the non-orthogonal case has cuSOLVER batched implementations.
+- **Complexity**: Projection operation $O(nk)$ per vector (orthonormal basis case), basis construction $O(nk^2)$ (Gram-Schmidt) or $O(n^2k)$ (QR decomposition).
+- **Low precision**: Under bf16, $A^HA$ may lose positive definiteness; a jitter $\epsilon I$ ($\epsilon \sim 10^{-6}$) must be added.
+
+## Risks and Failure Conditions
+
+- **Ill-conditioned basis matrix**: When the condition number $\kappa(A) \gg 1$, $(A^HA)^{-1}$ suffers catastrophic cancellation under low precision. Solution: first perform QR decomposition to obtain an orthonormal basis $Q$, then use $P = QQ^H$, avoiding explicit inversion.
+- **Excessively high subspace dimension**: When $k$ approaches $n$, the projection degenerates to the identity mapping, wasting computation. A randomized SVD should first be performed to confirm the effective rank.
+- **Non-stationary subspace**: In online/continual learning, the projection basis drifts with the data; the basis matrix must be periodically updated (incremental QR), otherwise the projection directions become stale.
+- **Non-orthogonal projection**: Oblique projections where $P^2 = P$ but $P \neq P^H$ are numerically unstable in floating point and should be avoided; if they must be used, monitor $\|P - P^H\|_F$.
+
+## Further References
+
+- Distilled notes: references/books/matrix-analysis.md (Section 2.1 QR Decomposition, Section 4.2 Courant-Fischer Variational Characterization, Section 2.6 SVD)
+- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 2 (Unitary Similarity) + Chapter 4 (Hermitian Matrices Section 4.2 Variational Characterizations)

package/knowledge-base/matrix-analysis/projection.md

@@ -0,0 +1,47 @@
+# 正交投影 (Orthogonal Projection)
+
+## 最小定义
+
+将向量 $v$ 映射到子空间 $\mathcal{S}$ 上，使得残差 $v - Pv$ 与 $\mathcal{S}$ 正交。投影矩阵 $P$ 满足 $P^2 = P$ 且 $P = P^H$（幂等 + Hermitian），是唯一确定最近点的线性算子。
+
+## 核心公式
+
+- 投影矩阵：$P = A(A^HA)^{-1}A^H$，其中 $A$ 的列张成子空间 $\mathcal{S}$
+- 若 $A$ 列正交归一（$A^HA = I$），则 $P = AA^H$
+- 投影后最近距离：$\|v - Pv\|^2 = \|v\|^2 - \|A^Hv\|^2$
+- Courant-Fischer 变分刻画：$\lambda_k = \max_{\dim(S)=k} \min_{x \in S, \|x\|=1} x^HAx$
+- 正交补投影：$P^\perp = I - P$
+
+## 适用问题
+
+- 最小二乘回归：$\hat{x} = \arg\min \|Ax - b\|^2$ 即求 $b$ 在 $\text{Col}(A)$ 上的投影
+- PCA 降维：数据投影到前 $k$ 个主成分张成的子空间
+- 约束优化中的可行方向：将梯度投影到约束切空间（投影梯度法）
+- 残差网络的正交分解：将信号分解为已解释部分 + 残差
+
+## AI 设计翻译
+
+- **Linear 层的低秩瓶颈分析**：$W = U_k \Sigma_k V_k^H$ 的截断即投影到秩-$k$ 子空间；用 `torch.mm(U_k, torch.mm(Sigma_k, V_k.t()))` 三步 matmul 实现，显存从 $O(mn)$ 降到 $O(k(m+n))$
+- **Attention 头的子空间投影**：Q/K/V 矩阵本质是把输入投影到不同子空间做相似度计算；多头 = 多个投影子空间的并行，可直接 batched matmul
+- **Projection Head (对比学习)**：SimCLR/MoCo 的 projection head = 多层 MLP 后接 $L_2$-normalize，等价于投影到单位球面；实现为 `F.normalize(self.mlp(x), dim=-1)`，是 elementwise norm 操作
+- **梯度投影 / 正交梯度下降 (OGD)**：在持续学习中，将新任务梯度投影到旧任务梯度空间的正交补，避免灾难性遗忘；需维护基矩阵 $G$ 并计算 $(I - G(G^TG)^{-1}G^T)\nabla$，核心是两次 matmul
+- **ResNet 的正交残差**：残差连接 $x + F(x)$ 可理解为在恒等子空间上的投影 + 正交修正；谱归一化约束 $F$ 的 Lipschitz 使投影稳定
+
+## 工程可行性
+
+- **主要操作**：矩阵乘法 (matmul) + 矩阵求逆/Cholesky。当子空间维度 $k$ 远小于 $n$ 时，$(A^HA)^{-1}$ 只需对 $k \times k$ 矩阵求逆，代价可忽略。
+- **GPU 友好度**：高。$P = AA^H$（列正交情形）是两次 matmul，完美映射 tensor core。非正交情形的 Cholesky 分解有 cuSOLVER batched 实现。
+- **复杂度**：投影操作 $O(nk)$ per vector（正交基情形），构造基 $O(nk^2)$（Gram-Schmidt）或 $O(n^2k)$（QR 分解）。
+- **低精度**：bf16 下 $A^HA$ 可能丢正定性，需加 jitter $\epsilon I$（$\epsilon \sim 10^{-6}$）。
+
+## 风险与失效条件
+
+- **基矩阵病态**：当 $A$ 条件数 $\kappa(A) \gg 1$ 时，$(A^HA)^{-1}$ 在低精度下灾难性抵消。解决：先 QR 分解得正交基 $Q$，再用 $P = QQ^H$，避免显式求逆。
+- **子空间维度过高**：$k$ 接近 $n$ 时投影退化为恒等映射，计算浪费。需先做随机 SVD 确认有效秩。
+- **非平稳子空间**：在线学习/持续学习中投影基随数据漂移，需定期更新基矩阵（增量 QR），否则投影方向过期。
+- **非正交投影**：$P^2 = P$ 但 $P \neq P^H$ 的斜投影在浮点下不稳定，应避免；若必须用，需监控 $\|P - P^H\|_F$。
+
+## 深入参考
+
+- 蒸馏稿：references/books/matrix-analysis.md（§2.1 QR 分解、§4.2 Courant-Fischer 变分刻画、§2.6 SVD）
+- 原书：Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 2 (Unitary Similarity) + Chapter 4 (Hermitian Matrices §4.2 Variational Characterizations)

package/knowledge-base/matrix-analysis/spectral-decomposition.en.md

@@ -0,0 +1,48 @@
+# Spectral Decomposition
+
+## Minimal Definition
+
+Decomposes a matrix into a linear combination of eigenvalue-eigenvector pairs. For a Hermitian matrix $A$, there exists a unitary matrix $U$ such that $A = U \Lambda U^H$, where $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$ contains real eigenvalues. For general matrices, the Schur decomposition $A = QTQ^H$ ($T$ upper triangular) serves as a numerically reliable alternative.
+
+## Core Formulas
+
+- Hermitian spectral decomposition: $A = \sum_{i=1}^{n} \lambda_i u_i u_i^H = U\Lambda U^H$
+- Spectral mapping: $f(A) = U f(\Lambda) U^H$ (matrix exponential, logarithm, powers, etc.)
+- Spectral radius: $\rho(A) = \max_i |\lambda_i|$
+- Normal matrix criterion: $A^HA = AA^H \iff A$ is unitarily diagonalizable
+- Schur decomposition (general matrices): $A = QTQ^H$, $T$ upper triangular, diagonal entries = eigenvalues
+- Trace-eigenvalue relations: $\text{tr}(A) = \sum \lambda_i$, $\det(A) = \prod \lambda_i$
+
+## Applicable Problems
+
+- Hessian spectral analysis: determining loss surface curvature (positive definite = local minimum, negative eigenvalues present = saddle point)
+- Gradient covariance spectrum: diagnosing training dynamics; the spectral radius determines stability
+- Spectral normalization: constraining $\sigma_{\max}(W) \leq 1$ to stabilize GAN/diffusion model training
+- State space model (SSM) stability: spectral radius of the discretization matrix $< 1$ guarantees non-divergence
+- Graph neural networks: Laplacian spectral decomposition = Fourier basis on graphs
+
+## AI Design Translation
+
+- **Power iteration for spectral radius / largest singular value estimation**: $u_{k+1} = Au_k / \|Au_k\|$, each step requiring only one matvec + norm, $O(n^2)$. Standard approach in spectral normalization (SN-GAN); built into PyTorch as `torch.nn.utils.spectral_norm`. Note: single-vector iteration has low parallelism; block iteration is needed for parallelization.
+- **Hessian-free optimization (HVP + CG)**: Without materializing the Hessian, computes $Hv$ via autodiff (one forward + one backward pass), then feeds it to CG to solve $Hd = -g$. The core operations are two backward passes (matvec), fully GPU-friendly.
+- **Kronecker-factored approximate curvature (K-FAC)**: Approximates the Hessian as $H \approx A \otimes B$ (Kronecker product), where each factor is a small matrix (on the order of the layer dimension), reducing inversion to small GEMM operations. Each layer is independent, naturally parallelizable.
+- **Spectral regularization loss**: $\mathcal{L}_{\text{spec}} = \max(0, \rho(A) - 1)^2$ or $\mathcal{L}_{\text{spec}} = \|\sigma_{\max}(W) - 1\|^2$, estimated via power iteration and added to the total loss. Implemented as an additional scalar loss term without affecting the main computational graph structure.
+- **Graph Fourier transform**: The eigendecomposition of the graph Laplacian $L = D - A$, $L = U\Lambda U^H$, provides the graph frequency domain basis. Spectral convolution in GCN = $U g(\Lambda) U^H x$, three matmul operations. For large-scale graphs, Chebyshev polynomial approximation avoids explicit decomposition.
+
+## Engineering Feasibility
+
+- **Primary operations**: Full EVD is $O(n^3)$, but full decomposition is rarely needed in AI. Power iteration is $O(n^2)$/step matvec; K-FAC factors are $O(d^2)$ small-matrix inversions.
+- **GPU friendliness**: Medium to high (method-dependent). Power iteration / HVP = matvec = friendly; full EVD is infeasible for $n > 1000$. cuSOLVER provides `syevd` (symmetric EVD) and `gesvd` (SVD), but the $O(n^3)$ cost limits scalability.
+- **Low precision**: Hermitian matrix EVD is relatively stable under bf16 (eigenvalues are Lipschitz continuous, Weyl bound). Eigenvalues of non-normal matrices may be severely distorted under low precision; SVD should be used instead.
+
+## Risks and Failure Conditions
+
+- **Non-normal matrix trap**: When $A^HA \neq AA^H$, eigenvalues do not predict transient behavior (pseudospectra may be much larger than the spectral radius), and using eigenvalues to assess stability can be seriously misleading. Solution: use SVD to examine singular values instead.
+- **Numerical sensitivity of repeated eigenvalues**: When algebraic multiplicity > geometric multiplicity (defective matrices), eigenvectors are extremely sensitive to perturbation, and Jordan blocks are uncomputable in floating point. Solution: use the Schur decomposition instead.
+- **Slow power iteration convergence**: When $\lambda_1 / \lambda_2 \approx 1$, convergence is extremely slow (requiring $O(1/(1-\lambda_2/\lambda_1))$ steps). Solution: block iteration or Lanczos acceleration.
+- **Numerical precision of HVP**: Floating-point errors in $Hv$ accumulate during CG iteration, potentially preventing CG convergence. Periodic re-orthogonalization or restart is needed.
+
+## Further References
+
+- Distilled notes: references/books/matrix-analysis.md (Ch 1 Eigenvalues and Similarity, Section 2.4-2.5 Schur Triangularization and Normal Matrices, Section 4.2 Courant-Fischer)
+- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 1 (Eigenvalues, Eigenvectors, Similarity) + Chapter 2 (Unitary Similarity Section 2.4-2.5)

package/knowledge-base/matrix-analysis/spectral-decomposition.md

@@ -0,0 +1,48 @@
+# 谱分解 (Spectral Decomposition)
+
+## 最小定义
+
+将矩阵分解为特征值-特征向量对的线性组合。对 Hermitian 矩阵 $A$，存在酉矩阵 $U$ 使得 $A = U \Lambda U^H$，其中 $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$ 为实特征值。对一般矩阵，用 Schur 分解 $A = QTQ^H$（$T$ 为上三角）作为数值可靠的替代。
+
+## 核心公式
+
+- Hermitian 谱分解：$A = \sum_{i=1}^{n} \lambda_i u_i u_i^H = U\Lambda U^H$
+- 谱映射：$f(A) = U f(\Lambda) U^H$（矩阵指数、对数、幂次等）
+- 谱半径：$\rho(A) = \max_i |\lambda_i|$
+- 正规矩阵判据：$A^HA = AA^H \iff A$ 可酉对角化
+- Schur 分解（一般矩阵）：$A = QTQ^H$，$T$ 上三角，对角元 = 特征值
+- 迹-特征值关系：$\text{tr}(A) = \sum \lambda_i$，$\det(A) = \prod \lambda_i$
+
+## 适用问题
+
+- Hessian 谱分析：判断损失面曲率（正定 = 局部极小、有负特征值 = 鞍点）
+- 梯度协方差谱：诊断训练动力学，谱半径决定稳定性
+- 谱归一化：约束 $\sigma_{\max}(W) \leq 1$ 稳定 GAN/扩散模型训练
+- 状态空间模型 (SSM) 稳定性：离散化矩阵谱半径 $< 1$ 保证不发散
+- 图神经网络：Laplacian 谱分解 = 图上的 Fourier 基
+
+## AI 设计翻译
+
+- **Power Iteration 估谱半径/最大奇异值**：$u_{k+1} = Au_k / \|Au_k\|$，每步仅一次 matvec + norm，$O(n^2)$。谱归一化中标准做法（SN-GAN），PyTorch 内置 `torch.nn.utils.spectral_norm`。注意：单向量迭代串行度低，需 block iteration 并行化。
+- **Hessian-Free 优化 (HVP + CG)**：不物化 Hessian，用 autodiff 算 $Hv$（一次前向 + 一次反向），再喂给 CG 求解 $Hd = -g$。核心操作是两次反向传播（matvec），完全 GPU 友好。
+- **Kronecker-Factored 近似曲率 (K-FAC)**：将 Hessian 近似为 $H \approx A \otimes B$（Kronecker 积），每个因子是小矩阵（层维度量级），求逆退化为小 GEMM。每层独立，天然可并行。
+- **谱正则化 Loss**：$\mathcal{L}_{\text{spec}} = \max(0, \rho(A) - 1)^2$ 或 $\mathcal{L}_{\text{spec}} = \|\sigma_{\max}(W) - 1\|^2$，通过 power iteration 估计后加入总 loss。实现为附加标量 loss，不影响主计算图结构。
+- **Graph Fourier Transform**：图 Laplacian $L = D - A$ 的特征分解 $L = U\Lambda U^H$ 给出图频域基。GCN 的谱卷积 = $U g(\Lambda) U^H x$，三次 matmul。大规模图用 Chebyshev 多项式近似避免显式分解。
+
+## 工程可行性
+
+- **主要操作**：完整 EVD 是 $O(n^3)$，但 AI 中极少需要完整分解。Power iteration 是 $O(n^2)$/step 的 matvec；K-FAC 因子是 $O(d^2)$ 的小矩阵求逆。
+- **GPU 友好度**：中-高（取决于方法）。Power iteration / HVP = matvec = 友好；完整 EVD 在 $n > 1000$ 时不可行。cuSOLVER 有 `syevd`（对称 EVD）和 `gesvd`（SVD），但 $O(n^3)$ 限制规模。
+- **低精度**：Hermitian 矩阵 EVD 在 bf16 下相对稳定（特征值是 Lipschitz 连续的，Weyl 界）。非正规矩阵的特征值在低精度下可能严重失真，需改用 SVD。
+
+## 风险与失效条件
+
+- **非正规矩阵陷阱**：$A^HA \neq AA^H$ 时特征值不预测瞬态行为（pseudospectra 可能远大于谱半径），用特征值判断稳定性会严重误判。解决：改用 SVD 看奇异值。
+- **重特征值的数值敏感性**：代数重数 > 几何重数时（亏损矩阵），特征向量对扰动极度敏感，Jordan 块在浮点下不可算。解决：用 Schur 分解代替。
+- **Power iteration 收敛慢**：当 $\lambda_1 / \lambda_2 \approx 1$ 时收敛极慢（需 $O(1/(1-\lambda_2/\lambda_1))$ 步）。解决：block iteration 或 Lanczos 加速。
+- **HVP 的数值精度**：$Hv$ 的浮点误差在 CG 迭代中累积，可能导致 CG 不收敛。需定期重正交化或加重启动。
+
+## 深入参考
+
+- 蒸馏稿：references/books/matrix-analysis.md（Ch 1 特征值与相似、§2.4-2.5 Schur 三角化与正规矩阵、§4.2 Courant-Fischer）
+- 原书：Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 1 (Eigenvalues, Eigenvectors, Similarity) + Chapter 2 (Unitary Similarity §2.4-2.5)

package/knowledge-base/optimization/constrained-optimization.en.md

@@ -0,0 +1,53 @@
+# Constrained Optimization
+
+## Minimal Definition
+
+Minimizes an objective $f(x)$ over a constraint set $\mathcal{C} = \{x : g_i(x) \leq 0, h_j(x) = 0\}$. The optimal solution satisfies the KKT conditions (first-order necessary conditions): stationarity of the gradient (in the Lagrangian sense), primal feasibility, dual feasibility, and complementary slackness. Constrained optimization systematically incorporates "hard restrictions" into the optimization framework.
+
+## Core Formulas
+
+- KKT conditions (inequality constraints):
+  - $\nabla_x L = \nabla f + \sum \lambda_i \nabla g_i + \sum \nu_j \nabla h_j = 0$ (stationarity)
+  - $g_i(x) \leq 0, \ h_j(x) = 0$ (primal feasibility)
+  - $\lambda_i \geq 0$ (dual feasibility)
+  - $\lambda_i g_i(x) = 0$ (complementary slackness)
+- Projected gradient method: $x_{k+1} = \text{proj}_{\mathcal{C}}(x_k - \alpha \nabla f(x_k))$
+- Penalty function method: $\min_x f(x) + \frac{\rho}{2}\sum [\max(0, g_i(x))]^2 + \frac{\rho}{2}\sum h_j(x)^2$
+- Augmented Lagrangian: $\mathcal{L}_\rho(x,\lambda) = f(x) + \sum \lambda_i g_i(x) + \frac{\rho}{2}\sum g_i(x)^2$
+- Armijo line search (constrained version): $\alpha$ satisfies $f(\text{proj}_\mathcal{C}(x - \alpha \nabla f)) \leq f(x) - \sigma \alpha \|\nabla f\|^2$
+
+## Applicable Problems
+
+- Weight norm constraints: $\|w\|_2 \leq R$ (weight clipping / norm ball)
+- Spectral norm constraints: $\sigma_{\max}(W) \leq 1$ (spectral normalization, stabilizing GAN/diffusion models)
+- Safety constraints / RLHF: $\mathbb{E}[r_{\text{safety}}] \geq \tau$ (KL-constrained policy optimization)
+- Fairness constraints: $|P(\hat{y}|A=0) - P(\hat{y}|A=1)| \leq \epsilon$
+- Resource constraints: performance optimization under model size / FLOPs / inference latency budgets
+
+## AI Design Translation
+
+- **Weight clipping (WGAN)**: $W \leftarrow \text{clamp}(W, -c, c)$ is projection onto an $\ell_\infty$-box constraint. Implemented as `torch.clamp(W, -c, c)`, an elementwise operation with zero additional computation. Simple but coarse, less refined than spectral normalization.
+- **Spectral normalization**: Constrains $\sigma_{\max}(W) \leq 1$, i.e., projects onto the operator norm ball. Uses power iteration to estimate $u \leftarrow W^T W u / \|W^T W u\|$, then normalizes $W \leftarrow W / \sigma_{\max}$. Two matvec + norm operations per step; built into PyTorch as `torch.nn.utils.spectral_norm`.
+- **Projected gradient for $\ell_2$-ball constraints**: $\text{proj}(w) = w \cdot \min(1, R/\|w\|_2)$, implemented as `w * min(1, R / w.norm())`, one norm + elementwise operation, $O(d)$. Used in trust regions and adversarial robustness $\epsilon$-ball constraints.
+- **Augmented Lagrangian for RLHF/PPO**: $\mathcal{L} = -\mathbb{E}[r] + \lambda(\text{KL}(\pi\|\pi_{\text{ref}}) - \epsilon) + \frac{\rho}{2}(\text{KL} - \epsilon)^2$. The inner loop optimizes $\pi$ via PPO; the outer loop updates $\lambda \leftarrow \lambda + \rho(\text{KL} - \epsilon)$. KL computation involves softmax + elementwise log-ratio, GPU-friendly.
+- **Penalty method for sparsity/low-rank constraints**: $\mathcal{L}_{\text{penalty}} = \mathcal{L}_{\text{task}} + \rho \sum_i \max(0, \|w_i\|_1 - \tau)^2$ constrains per-layer sparsity to not exceed $\tau$. The penalty term is elementwise + reduce, differentiable and GPU-friendly. $\rho$ increasing schedule: $\rho \leftarrow \beta \rho$ ($\beta > 1$), doubling every several steps.
+
+## Engineering Feasibility
+
+- **Primary operations**: Projection = elementwise + norm ($O(d)$); penalty term = elementwise + reduce ($O(d)$); KKT gradient = standard backpropagation; power iteration = matvec ($O(d^2)$ or $O(nd)$).
+- **GPU friendliness**: High. Projection steps in projected gradient methods are mostly cheap elementwise operations (norm-ball, box, $\ell_1$-ball); penalty terms / augmented Lagrangian only add elementwise computation; power iteration for spectral normalization is matvec.
+- **Complexity**: Projection $O(d)$ (norm-ball / box) to $O(d \log d)$ ($\ell_1$-ball); penalty evaluation $O(d)$; spectral normalization $O(nd)$ per iteration; interior-point methods per step $O(d^3)$ (avoid in the inner training loop).
+- **Low precision**: Projection operations are stable under bf16 (norm and clamp do not involve delicate numerical operations); the penalty coefficient $\rho$ must be range-controlled to avoid overflow (use fp32 when $\rho > 10^6$).
+
+## Risks and Failure Conditions
+
+- **Non-differentiability of projection**: $\text{proj}_\mathcal{C}(x)$ is non-differentiable at constraint boundaries (e.g., gradient discontinuity when $\|x\|_2 = R$), affecting backpropagation. Solution: smooth via the Moreau envelope, or replace hard projection with penalty methods / augmented Lagrangian.
+- **Ill-conditioning of penalty methods**: As $\rho \to \infty$, the Hessian condition number $\kappa \sim \rho$, slowing gradient descent convergence. Solution: augmented Lagrangian ($\rho$ need not go to infinity to exactly satisfy constraints), or dual ascent methods.
+- **Expensive projection onto general polyhedra**: Projection onto $\mathcal{C} = \{x : Ax \leq b\}$ requires solving a QP with complexity $O(d^3)$, infeasible at every step. Solution: switch to penalty methods or augmented Lagrangian to avoid explicit projection.
+- **KKT is only a necessary condition**: For non-convex problems, KKT points are not necessarily local optima (they may be saddle points). Second-order sufficient conditions (positive definiteness of the Lagrangian Hessian on the constraint tangent space) are needed for verification.
+- **Numerical precision of complementary slackness**: $\lambda_i g_i(x) = 0$ can only be satisfied to $\sim \epsilon$ in floating point, making active set identification difficult. The augmented Lagrangian naturally avoids this issue (it does not rely on exact complementary slackness).
+
+## Further References
+
+- Distilled notes: references/books/optimization-ml.md (Ch 20-21 Equality/Inequality Constraints and KKT, Ch 24 Constrained Algorithms Section 24.3 Projected Gradient, Section 24.5 Augmented Lagrangian, Section 24.6 Penalty Methods)
+- Original text: Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 20-21 (Equality & Inequality Constraints) + Chapter 24 (Algorithms for Constrained Optimization Section 24.2-24.6)

package/knowledge-base/optimization/constrained-optimization.md

@@ -0,0 +1,53 @@
+# 约束优化 (Constrained Optimization)
+
+## 最小定义
+
+在约束集 $\mathcal{C} = \{x : g_i(x) \leq 0, h_j(x) = 0\}$ 上最小化目标 $f(x)$。最优解满足 KKT 条件（一阶必要条件）：梯度为零（在 Lagrangian 意义下）、原始可行、对偶可行、互补松弛。约束优化将"硬限制"系统性地纳入优化框架。
+
+## 核心公式
+
+- KKT 条件（不等式约束）：
+  - $\nabla_x L = \nabla f + \sum \lambda_i \nabla g_i + \sum \nu_j \nabla h_j = 0$（驻点）
+  - $g_i(x) \leq 0, \ h_j(x) = 0$（原始可行）
+  - $\lambda_i \geq 0$（对偶可行）
+  - $\lambda_i g_i(x) = 0$（互补松弛）
+- 投影梯度法：$x_{k+1} = \text{proj}_{\mathcal{C}}(x_k - \alpha \nabla f(x_k))$
+- 惩罚函数法：$\min_x f(x) + \frac{\rho}{2}\sum [\max(0, g_i(x))]^2 + \frac{\rho}{2}\sum h_j(x)^2$
+- 增广 Lagrangian：$\mathcal{L}_\rho(x,\lambda) = f(x) + \sum \lambda_i g_i(x) + \frac{\rho}{2}\sum g_i(x)^2$
+- Armijo 线搜索（约束版）：$\alpha$ 满足 $f(\text{proj}_\mathcal{C}(x - \alpha \nabla f)) \leq f(x) - \sigma \alpha \|\nabla f\|^2$
+
+## 适用问题
+
+- 权重范数约束：$\|w\|_2 \leq R$（weight clipping / norm ball）
+- 谱范数约束：$\sigma_{\max}(W) \leq 1$（谱归一化，稳定 GAN/扩散模型）
+- 安全约束 / RLHF：$\mathbb{E}[r_{\text{safety}}] \geq \tau$（KL 约束策略优化）
+- 公平性约束：$|P(\hat{y}|A=0) - P(\hat{y}|A=1)| \leq \epsilon$
+- 资源约束：模型大小 / FLOPs / 推理延迟预算下的性能优化
+
+## AI 设计翻译
+
+- **Weight Clipping (WGAN)**：$W \leftarrow \text{clamp}(W, -c, c)$ 即投影到 $\ell_\infty$-box 约束。实现为 `torch.clamp(W, -c, c)`，elementwise 操作，零额外计算。简单但粗糙，不如谱归一化精细。
+- **谱归一化 (Spectral Normalization)**：约束 $\sigma_{\max}(W) \leq 1$，即投影到算子范数球。用 power iteration 估 $u \leftarrow W^T W u / \|W^T W u\|$，归一化 $W \leftarrow W / \sigma_{\max}$。每步 2 次 matvec + norm，PyTorch 内置 `torch.nn.utils.spectral_norm`。
+- **投影梯度做 $\ell_2$-ball 约束**：$\text{proj}(w) = w \cdot \min(1, R/\|w\|_2)$，实现为 `w * min(1, R / w.norm())`，一次 norm + elementwise，$O(d)$。用于 trust region、对抗鲁棒的 $\epsilon$-ball 约束。
+- **增广 Lagrangian 做 RLHF/PPO**：$\mathcal{L} = -\mathbb{E}[r] + \lambda(\text{KL}(\pi\|\pi_{\text{ref}}) - \epsilon) + \frac{\rho}{2}(\text{KL} - \epsilon)^2$。内层对 $\pi$ 用 PPO 优化，外层 $\lambda \leftarrow \lambda + \rho(\text{KL} - \epsilon)$。KL 计算是 softmax + elementwise log-ratio，GPU 友好。
+- **惩罚法做稀疏/低秩约束**：$\mathcal{L}_{\text{penalty}} = \mathcal{L}_{\text{task}} + \rho \sum_i \max(0, \|w_i\|_1 - \tau)^2$ 约束每层稀疏度不超 $\tau$。惩罚项是 elementwise + reduce，可微且 GPU 友好。$\rho$ 递增策略：$\rho \leftarrow \beta \rho$（$\beta > 1$），每若干步加倍。
+
+## 工程可行性
+
+- **主要操作**：投影 = elementwise + norm（$O(d)$）；惩罚项 = elementwise + reduce（$O(d)$）；KKT 梯度 = 标准反向传播；power iteration = matvec（$O(d^2)$ 或 $O(nd)$）。
+- **GPU 友好度**：高。投影梯度法的投影步骤多为廉价 elementwise（norm-ball、box、$\ell_1$-ball）；惩罚项 / 增广 Lagrangian 仅增加 elementwise 计算；谱归一化的 power iteration 是 matvec。
+- **复杂度**：投影 $O(d)$（norm-ball / box）到 $O(d \log d)$（$\ell_1$-ball）；惩罚评估 $O(d)$；谱归一化 $O(nd)$ per iteration；内点法每步 $O(d^3)$（避免在训练内环使用）。
+- **低精度**：投影操作在 bf16 下稳定（norm 和 clamp 不涉及精细数值）；惩罚项的 $\rho$ 需控制范围避免溢出（$\rho > 10^6$ 时用 fp32）。
+
+## 风险与失效条件
+
+- **投影的不可微性**：$\text{proj}_\mathcal{C}(x)$ 在约束边界处不可微（如 $\|x\|_2 = R$ 时梯度不连续），影响反向传播。解决：用 Moreau envelope 平滑化，或惩罚法 / 增广 Lagrangian 替代硬投影。
+- **惩罚法的病态**：$\rho \to \infty$ 时 Hessian 条件数 $\kappa \sim \rho$，梯度下降收敛变慢。解决：增广 Lagrangian（$\rho$ 不需趋于无穷即可精确满足约束），或对偶上升法。
+- **一般多面体投影昂贵**：$\mathcal{C} = \{x : Ax \leq b\}$ 的投影需解 QP，复杂度 $O(d^3)$，不能每步做。解决：改用惩罚法或增广 Lagrangian，避免显式投影。
+- **KKT 只是必要条件**：非凸问题下 KKT 点不一定是局部最优（可能是鞍点）。需配合二阶充分条件（Lagrangian Hessian 在约束切空间上正定）判断。
+- **互补松弛的数值精度**：$\lambda_i g_i(x) = 0$ 在浮点下只能达到 $\sim \epsilon$，active set 识别困难。增广 Lagrangian 天然避免此问题（不依赖精确互补松弛）。
+
+## 深入参考
+
+- 蒸馏稿：references/books/optimization-ml.md（Ch 20-21 等式/不等式约束与 KKT、Ch 24 约束算法 §24.3 投影梯度 §24.5 增广 Lagrange §24.6 惩罚法）
+- 原书：Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 20-21 (Equality & Inequality Constraints) + Chapter 24 (Algorithms for Constrained Optimization §24.2-24.6)