math-skill 2.0.1 → 3.0.0

package/knowledge-base/matrix-analysis/projection.en.md

@@ -0,0 +1,47 @@
+# Orthogonal Projection
+
+## Minimal Definition
+
+Maps a vector $v$ onto a subspace $\mathcal{S}$ such that the residual $v - Pv$ is orthogonal to $\mathcal{S}$. The projection matrix $P$ satisfies $P^2 = P$ and $P = P^H$ (idempotent + Hermitian), and is the unique linear operator that determines the nearest point.
+
+## Core Formulas
+
+- Projection matrix: $P = A(A^HA)^{-1}A^H$, where the columns of $A$ span the subspace $\mathcal{S}$
+- If the columns of $A$ are orthonormal ($A^HA = I$), then $P = AA^H$
+- Nearest distance after projection: $\|v - Pv\|^2 = \|v\|^2 - \|A^Hv\|^2$
+- Courant-Fischer variational characterization: $\lambda_k = \max_{\dim(S)=k} \min_{x \in S, \|x\|=1} x^HAx$
+- Orthogonal complement projection: $P^\perp = I - P$
+
+## Applicable Problems
+
+- Least squares regression: $\hat{x} = \arg\min \|Ax - b\|^2$ is equivalent to projecting $b$ onto $\text{Col}(A)$
+- PCA dimensionality reduction: projecting data onto the subspace spanned by the top $k$ principal components
+- Feasible directions in constrained optimization: projecting the gradient onto the constraint tangent space (projected gradient method)
+- Orthogonal decomposition in residual networks: decomposing a signal into an explained component + residual
+
+## AI Design Translation
+
+- **Low-rank bottleneck analysis for linear layers**: The truncation $W = U_k \Sigma_k V_k^H$ is a projection onto a rank-$k$ subspace; implemented via `torch.mm(U_k, torch.mm(Sigma_k, V_k.t()))` as three matmul steps, reducing memory from $O(mn)$ to $O(k(m+n))$
+- **Subspace projection in attention heads**: The Q/K/V matrices essentially project inputs into different subspaces for similarity computation; multi-head = parallel projection subspaces, directly implementable as batched matmul
+- **Projection head (contrastive learning)**: The projection head in SimCLR/MoCo = a multi-layer MLP followed by $L_2$-normalize, equivalent to projecting onto the unit sphere; implemented as `F.normalize(self.mlp(x), dim=-1)`, an elementwise norm operation
+- **Gradient projection / Orthogonal Gradient Descent (OGD)**: In continual learning, projecting the new-task gradient onto the orthogonal complement of the old-task gradient space to avoid catastrophic forgetting; requires maintaining a basis matrix $G$ and computing $(I - G(G^TG)^{-1}G^T)\nabla$, with the core being two matmul operations
+- **Orthogonal residuals in ResNet**: The residual connection $x + F(x)$ can be interpreted as a projection onto the identity subspace + orthogonal correction; spectral normalization constrains the Lipschitz constant of $F$ to stabilize the projection
+
+## Engineering Feasibility
+
+- **Primary operations**: Matrix multiplication (matmul) + matrix inversion/Cholesky. When the subspace dimension $k$ is much smaller than $n$, $(A^HA)^{-1}$ only requires inverting a $k \times k$ matrix, at negligible cost.
+- **GPU friendliness**: High. $P = AA^H$ (orthonormal columns case) is two matmul operations, perfectly mapped to tensor cores. The Cholesky decomposition for the non-orthogonal case has cuSOLVER batched implementations.
+- **Complexity**: Projection operation $O(nk)$ per vector (orthonormal basis case), basis construction $O(nk^2)$ (Gram-Schmidt) or $O(n^2k)$ (QR decomposition).
+- **Low precision**: Under bf16, $A^HA$ may lose positive definiteness; a jitter $\epsilon I$ ($\epsilon \sim 10^{-6}$) must be added.
+
+## Risks and Failure Conditions
+
+- **Ill-conditioned basis matrix**: When the condition number $\kappa(A) \gg 1$, $(A^HA)^{-1}$ suffers catastrophic cancellation under low precision. Solution: first perform QR decomposition to obtain an orthonormal basis $Q$, then use $P = QQ^H$, avoiding explicit inversion.
+- **Excessively high subspace dimension**: When $k$ approaches $n$, the projection degenerates to the identity mapping, wasting computation. A randomized SVD should first be performed to confirm the effective rank.
+- **Non-stationary subspace**: In online/continual learning, the projection basis drifts with the data; the basis matrix must be periodically updated (incremental QR), otherwise the projection directions become stale.
+- **Non-orthogonal projection**: Oblique projections where $P^2 = P$ but $P \neq P^H$ are numerically unstable in floating point and should be avoided; if they must be used, monitor $\|P - P^H\|_F$.
+
+## Further References
+
+- Distilled notes: references/books/matrix-analysis.md (Section 2.1 QR Decomposition, Section 4.2 Courant-Fischer Variational Characterization, Section 2.6 SVD)
+- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 2 (Unitary Similarity) + Chapter 4 (Hermitian Matrices Section 4.2 Variational Characterizations)

package/knowledge-base/matrix-analysis/projection.md

@@ -0,0 +1,47 @@
+# 正交投影 (Orthogonal Projection)
+
+## 最小定义
+
+将向量 $v$ 映射到子空间 $\mathcal{S}$ 上，使得残差 $v - Pv$ 与 $\mathcal{S}$ 正交。投影矩阵 $P$ 满足 $P^2 = P$ 且 $P = P^H$（幂等 + Hermitian），是唯一确定最近点的线性算子。
+
+## 核心公式
+
+- 投影矩阵：$P = A(A^HA)^{-1}A^H$，其中 $A$ 的列张成子空间 $\mathcal{S}$
+- 若 $A$ 列正交归一（$A^HA = I$），则 $P = AA^H$
+- 投影后最近距离：$\|v - Pv\|^2 = \|v\|^2 - \|A^Hv\|^2$
+- Courant-Fischer 变分刻画：$\lambda_k = \max_{\dim(S)=k} \min_{x \in S, \|x\|=1} x^HAx$
+- 正交补投影：$P^\perp = I - P$
+
+## 适用问题
+
+- 最小二乘回归：$\hat{x} = \arg\min \|Ax - b\|^2$ 即求 $b$ 在 $\text{Col}(A)$ 上的投影
+- PCA 降维：数据投影到前 $k$ 个主成分张成的子空间
+- 约束优化中的可行方向：将梯度投影到约束切空间（投影梯度法）
+- 残差网络的正交分解：将信号分解为已解释部分 + 残差
+
+## AI 设计翻译
+
+- **Linear 层的低秩瓶颈分析**：$W = U_k \Sigma_k V_k^H$ 的截断即投影到秩-$k$ 子空间；用 `torch.mm(U_k, torch.mm(Sigma_k, V_k.t()))` 三步 matmul 实现，显存从 $O(mn)$ 降到 $O(k(m+n))$
+- **Attention 头的子空间投影**：Q/K/V 矩阵本质是把输入投影到不同子空间做相似度计算；多头 = 多个投影子空间的并行，可直接 batched matmul
+- **Projection Head (对比学习)**：SimCLR/MoCo 的 projection head = 多层 MLP 后接 $L_2$-normalize，等价于投影到单位球面；实现为 `F.normalize(self.mlp(x), dim=-1)`，是 elementwise norm 操作
+- **梯度投影 / 正交梯度下降 (OGD)**：在持续学习中，将新任务梯度投影到旧任务梯度空间的正交补，避免灾难性遗忘；需维护基矩阵 $G$ 并计算 $(I - G(G^TG)^{-1}G^T)\nabla$，核心是两次 matmul
+- **ResNet 的正交残差**：残差连接 $x + F(x)$ 可理解为在恒等子空间上的投影 + 正交修正；谱归一化约束 $F$ 的 Lipschitz 使投影稳定
+
+## 工程可行性
+
+- **主要操作**：矩阵乘法 (matmul) + 矩阵求逆/Cholesky。当子空间维度 $k$ 远小于 $n$ 时，$(A^HA)^{-1}$ 只需对 $k \times k$ 矩阵求逆，代价可忽略。
+- **GPU 友好度**：高。$P = AA^H$（列正交情形）是两次 matmul，完美映射 tensor core。非正交情形的 Cholesky 分解有 cuSOLVER batched 实现。
+- **复杂度**：投影操作 $O(nk)$ per vector（正交基情形），构造基 $O(nk^2)$（Gram-Schmidt）或 $O(n^2k)$（QR 分解）。
+- **低精度**：bf16 下 $A^HA$ 可能丢正定性，需加 jitter $\epsilon I$（$\epsilon \sim 10^{-6}$）。
+
+## 风险与失效条件
+
+- **基矩阵病态**：当 $A$ 条件数 $\kappa(A) \gg 1$ 时，$(A^HA)^{-1}$ 在低精度下灾难性抵消。解决：先 QR 分解得正交基 $Q$，再用 $P = QQ^H$，避免显式求逆。
+- **子空间维度过高**：$k$ 接近 $n$ 时投影退化为恒等映射，计算浪费。需先做随机 SVD 确认有效秩。
+- **非平稳子空间**：在线学习/持续学习中投影基随数据漂移，需定期更新基矩阵（增量 QR），否则投影方向过期。
+- **非正交投影**：$P^2 = P$ 但 $P \neq P^H$ 的斜投影在浮点下不稳定，应避免；若必须用，需监控 $\|P - P^H\|_F$。
+
+## 深入参考
+
+- 蒸馏稿：references/books/matrix-analysis.md（§2.1 QR 分解、§4.2 Courant-Fischer 变分刻画、§2.6 SVD）
+- 原书：Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 2 (Unitary Similarity) + Chapter 4 (Hermitian Matrices §4.2 Variational Characterizations)

package/knowledge-base/matrix-analysis/spectral-decomposition.en.md

@@ -0,0 +1,48 @@
+# Spectral Decomposition
+
+## Minimal Definition
+
+Decomposes a matrix into a linear combination of eigenvalue-eigenvector pairs. For a Hermitian matrix $A$, there exists a unitary matrix $U$ such that $A = U \Lambda U^H$, where $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$ contains real eigenvalues. For general matrices, the Schur decomposition $A = QTQ^H$ ($T$ upper triangular) serves as a numerically reliable alternative.
+
+## Core Formulas
+
+- Hermitian spectral decomposition: $A = \sum_{i=1}^{n} \lambda_i u_i u_i^H = U\Lambda U^H$
+- Spectral mapping: $f(A) = U f(\Lambda) U^H$ (matrix exponential, logarithm, powers, etc.)
+- Spectral radius: $\rho(A) = \max_i |\lambda_i|$
+- Normal matrix criterion: $A^HA = AA^H \iff A$ is unitarily diagonalizable
+- Schur decomposition (general matrices): $A = QTQ^H$, $T$ upper triangular, diagonal entries = eigenvalues
+- Trace-eigenvalue relations: $\text{tr}(A) = \sum \lambda_i$, $\det(A) = \prod \lambda_i$
+
+## Applicable Problems
+
+- Hessian spectral analysis: determining loss surface curvature (positive definite = local minimum, negative eigenvalues present = saddle point)
+- Gradient covariance spectrum: diagnosing training dynamics; the spectral radius determines stability
+- Spectral normalization: constraining $\sigma_{\max}(W) \leq 1$ to stabilize GAN/diffusion model training
+- State space model (SSM) stability: spectral radius of the discretization matrix $< 1$ guarantees non-divergence
+- Graph neural networks: Laplacian spectral decomposition = Fourier basis on graphs
+
+## AI Design Translation
+
+- **Power iteration for spectral radius / largest singular value estimation**: $u_{k+1} = Au_k / \|Au_k\|$, each step requiring only one matvec + norm, $O(n^2)$. Standard approach in spectral normalization (SN-GAN); built into PyTorch as `torch.nn.utils.spectral_norm`. Note: single-vector iteration has low parallelism; block iteration is needed for parallelization.
+- **Hessian-free optimization (HVP + CG)**: Without materializing the Hessian, computes $Hv$ via autodiff (one forward + one backward pass), then feeds it to CG to solve $Hd = -g$. The core operations are two backward passes (matvec), fully GPU-friendly.
+- **Kronecker-factored approximate curvature (K-FAC)**: Approximates the Hessian as $H \approx A \otimes B$ (Kronecker product), where each factor is a small matrix (on the order of the layer dimension), reducing inversion to small GEMM operations. Each layer is independent, naturally parallelizable.
+- **Spectral regularization loss**: $\mathcal{L}_{\text{spec}} = \max(0, \rho(A) - 1)^2$ or $\mathcal{L}_{\text{spec}} = \|\sigma_{\max}(W) - 1\|^2$, estimated via power iteration and added to the total loss. Implemented as an additional scalar loss term without affecting the main computational graph structure.
+- **Graph Fourier transform**: The eigendecomposition of the graph Laplacian $L = D - A$, $L = U\Lambda U^H$, provides the graph frequency domain basis. Spectral convolution in GCN = $U g(\Lambda) U^H x$, three matmul operations. For large-scale graphs, Chebyshev polynomial approximation avoids explicit decomposition.
+
+## Engineering Feasibility
+
+- **Primary operations**: Full EVD is $O(n^3)$, but full decomposition is rarely needed in AI. Power iteration is $O(n^2)$/step matvec; K-FAC factors are $O(d^2)$ small-matrix inversions.
+- **GPU friendliness**: Medium to high (method-dependent). Power iteration / HVP = matvec = friendly; full EVD is infeasible for $n > 1000$. cuSOLVER provides `syevd` (symmetric EVD) and `gesvd` (SVD), but the $O(n^3)$ cost limits scalability.
+- **Low precision**: Hermitian matrix EVD is relatively stable under bf16 (eigenvalues are Lipschitz continuous, Weyl bound). Eigenvalues of non-normal matrices may be severely distorted under low precision; SVD should be used instead.
+
+## Risks and Failure Conditions
+
+- **Non-normal matrix trap**: When $A^HA \neq AA^H$, eigenvalues do not predict transient behavior (pseudospectra may be much larger than the spectral radius), and using eigenvalues to assess stability can be seriously misleading. Solution: use SVD to examine singular values instead.
+- **Numerical sensitivity of repeated eigenvalues**: When algebraic multiplicity > geometric multiplicity (defective matrices), eigenvectors are extremely sensitive to perturbation, and Jordan blocks are uncomputable in floating point. Solution: use the Schur decomposition instead.
+- **Slow power iteration convergence**: When $\lambda_1 / \lambda_2 \approx 1$, convergence is extremely slow (requiring $O(1/(1-\lambda_2/\lambda_1))$ steps). Solution: block iteration or Lanczos acceleration.
+- **Numerical precision of HVP**: Floating-point errors in $Hv$ accumulate during CG iteration, potentially preventing CG convergence. Periodic re-orthogonalization or restart is needed.
+
+## Further References
+
+- Distilled notes: references/books/matrix-analysis.md (Ch 1 Eigenvalues and Similarity, Section 2.4-2.5 Schur Triangularization and Normal Matrices, Section 4.2 Courant-Fischer)
+- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 1 (Eigenvalues, Eigenvectors, Similarity) + Chapter 2 (Unitary Similarity Section 2.4-2.5)

package/knowledge-base/matrix-analysis/spectral-decomposition.md

@@ -0,0 +1,48 @@
+# 谱分解 (Spectral Decomposition)
+
+## 最小定义
+
+将矩阵分解为特征值-特征向量对的线性组合。对 Hermitian 矩阵 $A$，存在酉矩阵 $U$ 使得 $A = U \Lambda U^H$，其中 $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$ 为实特征值。对一般矩阵，用 Schur 分解 $A = QTQ^H$（$T$ 为上三角）作为数值可靠的替代。
+
+## 核心公式
+
+- Hermitian 谱分解：$A = \sum_{i=1}^{n} \lambda_i u_i u_i^H = U\Lambda U^H$
+- 谱映射：$f(A) = U f(\Lambda) U^H$（矩阵指数、对数、幂次等）
+- 谱半径：$\rho(A) = \max_i |\lambda_i|$
+- 正规矩阵判据：$A^HA = AA^H \iff A$ 可酉对角化
+- Schur 分解（一般矩阵）：$A = QTQ^H$，$T$ 上三角，对角元 = 特征值
+- 迹-特征值关系：$\text{tr}(A) = \sum \lambda_i$，$\det(A) = \prod \lambda_i$
+
+## 适用问题
+
+- Hessian 谱分析：判断损失面曲率（正定 = 局部极小、有负特征值 = 鞍点）
+- 梯度协方差谱：诊断训练动力学，谱半径决定稳定性
+- 谱归一化：约束 $\sigma_{\max}(W) \leq 1$ 稳定 GAN/扩散模型训练
+- 状态空间模型 (SSM) 稳定性：离散化矩阵谱半径 $< 1$ 保证不发散
+- 图神经网络：Laplacian 谱分解 = 图上的 Fourier 基
+
+## AI 设计翻译
+
+- **Power Iteration 估谱半径/最大奇异值**：$u_{k+1} = Au_k / \|Au_k\|$，每步仅一次 matvec + norm，$O(n^2)$。谱归一化中标准做法（SN-GAN），PyTorch 内置 `torch.nn.utils.spectral_norm`。注意：单向量迭代串行度低，需 block iteration 并行化。
+- **Hessian-Free 优化 (HVP + CG)**：不物化 Hessian，用 autodiff 算 $Hv$（一次前向 + 一次反向），再喂给 CG 求解 $Hd = -g$。核心操作是两次反向传播（matvec），完全 GPU 友好。
+- **Kronecker-Factored 近似曲率 (K-FAC)**：将 Hessian 近似为 $H \approx A \otimes B$（Kronecker 积），每个因子是小矩阵（层维度量级），求逆退化为小 GEMM。每层独立，天然可并行。
+- **谱正则化 Loss**：$\mathcal{L}_{\text{spec}} = \max(0, \rho(A) - 1)^2$ 或 $\mathcal{L}_{\text{spec}} = \|\sigma_{\max}(W) - 1\|^2$，通过 power iteration 估计后加入总 loss。实现为附加标量 loss，不影响主计算图结构。
+- **Graph Fourier Transform**：图 Laplacian $L = D - A$ 的特征分解 $L = U\Lambda U^H$ 给出图频域基。GCN 的谱卷积 = $U g(\Lambda) U^H x$，三次 matmul。大规模图用 Chebyshev 多项式近似避免显式分解。
+
+## 工程可行性
+
+- **主要操作**：完整 EVD 是 $O(n^3)$，但 AI 中极少需要完整分解。Power iteration 是 $O(n^2)$/step 的 matvec；K-FAC 因子是 $O(d^2)$ 的小矩阵求逆。
+- **GPU 友好度**：中-高（取决于方法）。Power iteration / HVP = matvec = 友好；完整 EVD 在 $n > 1000$ 时不可行。cuSOLVER 有 `syevd`（对称 EVD）和 `gesvd`（SVD），但 $O(n^3)$ 限制规模。
+- **低精度**：Hermitian 矩阵 EVD 在 bf16 下相对稳定（特征值是 Lipschitz 连续的，Weyl 界）。非正规矩阵的特征值在低精度下可能严重失真，需改用 SVD。
+
+## 风险与失效条件
+
+- **非正规矩阵陷阱**：$A^HA \neq AA^H$ 时特征值不预测瞬态行为（pseudospectra 可能远大于谱半径），用特征值判断稳定性会严重误判。解决：改用 SVD 看奇异值。
+- **重特征值的数值敏感性**：代数重数 > 几何重数时（亏损矩阵），特征向量对扰动极度敏感，Jordan 块在浮点下不可算。解决：用 Schur 分解代替。
+- **Power iteration 收敛慢**：当 $\lambda_1 / \lambda_2 \approx 1$ 时收敛极慢（需 $O(1/(1-\lambda_2/\lambda_1))$ 步）。解决：block iteration 或 Lanczos 加速。
+- **HVP 的数值精度**：$Hv$ 的浮点误差在 CG 迭代中累积，可能导致 CG 不收敛。需定期重正交化或加重启动。
+
+## 深入参考
+
+- 蒸馏稿：references/books/matrix-analysis.md（Ch 1 特征值与相似、§2.4-2.5 Schur 三角化与正规矩阵、§4.2 Courant-Fischer）
+- 原书：Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 1 (Eigenvalues, Eigenvectors, Similarity) + Chapter 2 (Unitary Similarity §2.4-2.5)

package/knowledge-base/optimization/constrained-optimization.en.md

@@ -0,0 +1,53 @@
+# Constrained Optimization
+
+## Minimal Definition
+
+Minimizes an objective $f(x)$ over a constraint set $\mathcal{C} = \{x : g_i(x) \leq 0, h_j(x) = 0\}$. The optimal solution satisfies the KKT conditions (first-order necessary conditions): stationarity of the gradient (in the Lagrangian sense), primal feasibility, dual feasibility, and complementary slackness. Constrained optimization systematically incorporates "hard restrictions" into the optimization framework.
+
+## Core Formulas
+
+- KKT conditions (inequality constraints):
+  - $\nabla_x L = \nabla f + \sum \lambda_i \nabla g_i + \sum \nu_j \nabla h_j = 0$ (stationarity)
+  - $g_i(x) \leq 0, \ h_j(x) = 0$ (primal feasibility)
+  - $\lambda_i \geq 0$ (dual feasibility)
+  - $\lambda_i g_i(x) = 0$ (complementary slackness)
+- Projected gradient method: $x_{k+1} = \text{proj}_{\mathcal{C}}(x_k - \alpha \nabla f(x_k))$
+- Penalty function method: $\min_x f(x) + \frac{\rho}{2}\sum [\max(0, g_i(x))]^2 + \frac{\rho}{2}\sum h_j(x)^2$
+- Augmented Lagrangian: $\mathcal{L}_\rho(x,\lambda) = f(x) + \sum \lambda_i g_i(x) + \frac{\rho}{2}\sum g_i(x)^2$
+- Armijo line search (constrained version): $\alpha$ satisfies $f(\text{proj}_\mathcal{C}(x - \alpha \nabla f)) \leq f(x) - \sigma \alpha \|\nabla f\|^2$
+
+## Applicable Problems
+
+- Weight norm constraints: $\|w\|_2 \leq R$ (weight clipping / norm ball)
+- Spectral norm constraints: $\sigma_{\max}(W) \leq 1$ (spectral normalization, stabilizing GAN/diffusion models)
+- Safety constraints / RLHF: $\mathbb{E}[r_{\text{safety}}] \geq \tau$ (KL-constrained policy optimization)
+- Fairness constraints: $|P(\hat{y}|A=0) - P(\hat{y}|A=1)| \leq \epsilon$
+- Resource constraints: performance optimization under model size / FLOPs / inference latency budgets
+
+## AI Design Translation
+
+- **Weight clipping (WGAN)**: $W \leftarrow \text{clamp}(W, -c, c)$ is projection onto an $\ell_\infty$-box constraint. Implemented as `torch.clamp(W, -c, c)`, an elementwise operation with zero additional computation. Simple but coarse, less refined than spectral normalization.
+- **Spectral normalization**: Constrains $\sigma_{\max}(W) \leq 1$, i.e., projects onto the operator norm ball. Uses power iteration to estimate $u \leftarrow W^T W u / \|W^T W u\|$, then normalizes $W \leftarrow W / \sigma_{\max}$. Two matvec + norm operations per step; built into PyTorch as `torch.nn.utils.spectral_norm`.
+- **Projected gradient for $\ell_2$-ball constraints**: $\text{proj}(w) = w \cdot \min(1, R/\|w\|_2)$, implemented as `w * min(1, R / w.norm())`, one norm + elementwise operation, $O(d)$. Used in trust regions and adversarial robustness $\epsilon$-ball constraints.
+- **Augmented Lagrangian for RLHF/PPO**: $\mathcal{L} = -\mathbb{E}[r] + \lambda(\text{KL}(\pi\|\pi_{\text{ref}}) - \epsilon) + \frac{\rho}{2}(\text{KL} - \epsilon)^2$. The inner loop optimizes $\pi$ via PPO; the outer loop updates $\lambda \leftarrow \lambda + \rho(\text{KL} - \epsilon)$. KL computation involves softmax + elementwise log-ratio, GPU-friendly.
+- **Penalty method for sparsity/low-rank constraints**: $\mathcal{L}_{\text{penalty}} = \mathcal{L}_{\text{task}} + \rho \sum_i \max(0, \|w_i\|_1 - \tau)^2$ constrains per-layer sparsity to not exceed $\tau$. The penalty term is elementwise + reduce, differentiable and GPU-friendly. $\rho$ increasing schedule: $\rho \leftarrow \beta \rho$ ($\beta > 1$), doubling every several steps.
+
+## Engineering Feasibility
+
+- **Primary operations**: Projection = elementwise + norm ($O(d)$); penalty term = elementwise + reduce ($O(d)$); KKT gradient = standard backpropagation; power iteration = matvec ($O(d^2)$ or $O(nd)$).
+- **GPU friendliness**: High. Projection steps in projected gradient methods are mostly cheap elementwise operations (norm-ball, box, $\ell_1$-ball); penalty terms / augmented Lagrangian only add elementwise computation; power iteration for spectral normalization is matvec.
+- **Complexity**: Projection $O(d)$ (norm-ball / box) to $O(d \log d)$ ($\ell_1$-ball); penalty evaluation $O(d)$; spectral normalization $O(nd)$ per iteration; interior-point methods per step $O(d^3)$ (avoid in the inner training loop).
+- **Low precision**: Projection operations are stable under bf16 (norm and clamp do not involve delicate numerical operations); the penalty coefficient $\rho$ must be range-controlled to avoid overflow (use fp32 when $\rho > 10^6$).
+
+## Risks and Failure Conditions
+
+- **Non-differentiability of projection**: $\text{proj}_\mathcal{C}(x)$ is non-differentiable at constraint boundaries (e.g., gradient discontinuity when $\|x\|_2 = R$), affecting backpropagation. Solution: smooth via the Moreau envelope, or replace hard projection with penalty methods / augmented Lagrangian.
+- **Ill-conditioning of penalty methods**: As $\rho \to \infty$, the Hessian condition number $\kappa \sim \rho$, slowing gradient descent convergence. Solution: augmented Lagrangian ($\rho$ need not go to infinity to exactly satisfy constraints), or dual ascent methods.
+- **Expensive projection onto general polyhedra**: Projection onto $\mathcal{C} = \{x : Ax \leq b\}$ requires solving a QP with complexity $O(d^3)$, infeasible at every step. Solution: switch to penalty methods or augmented Lagrangian to avoid explicit projection.
+- **KKT is only a necessary condition**: For non-convex problems, KKT points are not necessarily local optima (they may be saddle points). Second-order sufficient conditions (positive definiteness of the Lagrangian Hessian on the constraint tangent space) are needed for verification.
+- **Numerical precision of complementary slackness**: $\lambda_i g_i(x) = 0$ can only be satisfied to $\sim \epsilon$ in floating point, making active set identification difficult. The augmented Lagrangian naturally avoids this issue (it does not rely on exact complementary slackness).
+
+## Further References
+
+- Distilled notes: references/books/optimization-ml.md (Ch 20-21 Equality/Inequality Constraints and KKT, Ch 24 Constrained Algorithms Section 24.3 Projected Gradient, Section 24.5 Augmented Lagrangian, Section 24.6 Penalty Methods)
+- Original text: Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 20-21 (Equality & Inequality Constraints) + Chapter 24 (Algorithms for Constrained Optimization Section 24.2-24.6)

package/knowledge-base/optimization/constrained-optimization.md

@@ -0,0 +1,53 @@
+# 约束优化 (Constrained Optimization)
+
+## 最小定义
+
+在约束集 $\mathcal{C} = \{x : g_i(x) \leq 0, h_j(x) = 0\}$ 上最小化目标 $f(x)$。最优解满足 KKT 条件（一阶必要条件）：梯度为零（在 Lagrangian 意义下）、原始可行、对偶可行、互补松弛。约束优化将"硬限制"系统性地纳入优化框架。
+
+## 核心公式
+
+- KKT 条件（不等式约束）：
+  - $\nabla_x L = \nabla f + \sum \lambda_i \nabla g_i + \sum \nu_j \nabla h_j = 0$（驻点）
+  - $g_i(x) \leq 0, \ h_j(x) = 0$（原始可行）
+  - $\lambda_i \geq 0$（对偶可行）
+  - $\lambda_i g_i(x) = 0$（互补松弛）
+- 投影梯度法：$x_{k+1} = \text{proj}_{\mathcal{C}}(x_k - \alpha \nabla f(x_k))$
+- 惩罚函数法：$\min_x f(x) + \frac{\rho}{2}\sum [\max(0, g_i(x))]^2 + \frac{\rho}{2}\sum h_j(x)^2$
+- 增广 Lagrangian：$\mathcal{L}_\rho(x,\lambda) = f(x) + \sum \lambda_i g_i(x) + \frac{\rho}{2}\sum g_i(x)^2$
+- Armijo 线搜索（约束版）：$\alpha$ 满足 $f(\text{proj}_\mathcal{C}(x - \alpha \nabla f)) \leq f(x) - \sigma \alpha \|\nabla f\|^2$
+
+## 适用问题
+
+- 权重范数约束：$\|w\|_2 \leq R$（weight clipping / norm ball）
+- 谱范数约束：$\sigma_{\max}(W) \leq 1$（谱归一化，稳定 GAN/扩散模型）
+- 安全约束 / RLHF：$\mathbb{E}[r_{\text{safety}}] \geq \tau$（KL 约束策略优化）
+- 公平性约束：$|P(\hat{y}|A=0) - P(\hat{y}|A=1)| \leq \epsilon$
+- 资源约束：模型大小 / FLOPs / 推理延迟预算下的性能优化
+
+## AI 设计翻译
+
+- **Weight Clipping (WGAN)**：$W \leftarrow \text{clamp}(W, -c, c)$ 即投影到 $\ell_\infty$-box 约束。实现为 `torch.clamp(W, -c, c)`，elementwise 操作，零额外计算。简单但粗糙，不如谱归一化精细。
+- **谱归一化 (Spectral Normalization)**：约束 $\sigma_{\max}(W) \leq 1$，即投影到算子范数球。用 power iteration 估 $u \leftarrow W^T W u / \|W^T W u\|$，归一化 $W \leftarrow W / \sigma_{\max}$。每步 2 次 matvec + norm，PyTorch 内置 `torch.nn.utils.spectral_norm`。
+- **投影梯度做 $\ell_2$-ball 约束**：$\text{proj}(w) = w \cdot \min(1, R/\|w\|_2)$，实现为 `w * min(1, R / w.norm())`，一次 norm + elementwise，$O(d)$。用于 trust region、对抗鲁棒的 $\epsilon$-ball 约束。
+- **增广 Lagrangian 做 RLHF/PPO**：$\mathcal{L} = -\mathbb{E}[r] + \lambda(\text{KL}(\pi\|\pi_{\text{ref}}) - \epsilon) + \frac{\rho}{2}(\text{KL} - \epsilon)^2$。内层对 $\pi$ 用 PPO 优化，外层 $\lambda \leftarrow \lambda + \rho(\text{KL} - \epsilon)$。KL 计算是 softmax + elementwise log-ratio，GPU 友好。
+- **惩罚法做稀疏/低秩约束**：$\mathcal{L}_{\text{penalty}} = \mathcal{L}_{\text{task}} + \rho \sum_i \max(0, \|w_i\|_1 - \tau)^2$ 约束每层稀疏度不超 $\tau$。惩罚项是 elementwise + reduce，可微且 GPU 友好。$\rho$ 递增策略：$\rho \leftarrow \beta \rho$（$\beta > 1$），每若干步加倍。
+
+## 工程可行性
+
+- **主要操作**：投影 = elementwise + norm（$O(d)$）；惩罚项 = elementwise + reduce（$O(d)$）；KKT 梯度 = 标准反向传播；power iteration = matvec（$O(d^2)$ 或 $O(nd)$）。
+- **GPU 友好度**：高。投影梯度法的投影步骤多为廉价 elementwise（norm-ball、box、$\ell_1$-ball）；惩罚项 / 增广 Lagrangian 仅增加 elementwise 计算；谱归一化的 power iteration 是 matvec。
+- **复杂度**：投影 $O(d)$（norm-ball / box）到 $O(d \log d)$（$\ell_1$-ball）；惩罚评估 $O(d)$；谱归一化 $O(nd)$ per iteration；内点法每步 $O(d^3)$（避免在训练内环使用）。
+- **低精度**：投影操作在 bf16 下稳定（norm 和 clamp 不涉及精细数值）；惩罚项的 $\rho$ 需控制范围避免溢出（$\rho > 10^6$ 时用 fp32）。
+
+## 风险与失效条件
+
+- **投影的不可微性**：$\text{proj}_\mathcal{C}(x)$ 在约束边界处不可微（如 $\|x\|_2 = R$ 时梯度不连续），影响反向传播。解决：用 Moreau envelope 平滑化，或惩罚法 / 增广 Lagrangian 替代硬投影。
+- **惩罚法的病态**：$\rho \to \infty$ 时 Hessian 条件数 $\kappa \sim \rho$，梯度下降收敛变慢。解决：增广 Lagrangian（$\rho$ 不需趋于无穷即可精确满足约束），或对偶上升法。
+- **一般多面体投影昂贵**：$\mathcal{C} = \{x : Ax \leq b\}$ 的投影需解 QP，复杂度 $O(d^3)$，不能每步做。解决：改用惩罚法或增广 Lagrangian，避免显式投影。
+- **KKT 只是必要条件**：非凸问题下 KKT 点不一定是局部最优（可能是鞍点）。需配合二阶充分条件（Lagrangian Hessian 在约束切空间上正定）判断。
+- **互补松弛的数值精度**：$\lambda_i g_i(x) = 0$ 在浮点下只能达到 $\sim \epsilon$，active set 识别困难。增广 Lagrangian 天然避免此问题（不依赖精确互补松弛）。
+
+## 深入参考
+
+- 蒸馏稿：references/books/optimization-ml.md（Ch 20-21 等式/不等式约束与 KKT、Ch 24 约束算法 §24.3 投影梯度 §24.5 增广 Lagrange §24.6 惩罚法）
+- 原书：Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 20-21 (Equality & Inequality Constraints) + Chapter 24 (Algorithms for Constrained Optimization §24.2-24.6)

package/knowledge-base/optimization/convex-optimization.en.md

@@ -0,0 +1,50 @@
+# Convex Optimization
+
+## Minimal Definition
+
+Optimization problems that minimize a convex function over a convex set. Convexity guarantees that a local optimum is also a global optimum, and the first-order condition ($\nabla f = 0$) is both necessary and sufficient. Convex problems are the only class in optimization theory that guarantees global optimality with efficient solvability.
+
+## Core Formulas
+
+- Convex set: $\forall x, y \in C, \theta \in [0,1]: \theta x + (1-\theta)y \in C$
+- Convex function: $f(\theta x + (1-\theta)y) \leq \theta f(x) + (1-\theta)f(y)$
+- First-order condition: $f(y) \geq f(x) + \nabla f(x)^T(y-x)$ (tangent line lies below the function)
+- Second-order condition: $\nabla^2 f(x) \succeq 0$ (Hessian is positive semidefinite)
+- Standard convex problem form: $\min f(x)$ s.t. $g_i(x) \leq 0$ ($g_i$ convex), $Ax = b$
+- Convergence rate (strongly convex + smooth): $\|x_k - x^*\|^2 \leq (1 - \mu/L)^k \|x_0 - x^*\|^2$, condition number $\kappa = L/\mu$
+- Semidefinite programming (SDP): $\min \langle C, X \rangle$ s.t. $\langle A_i, X \rangle = b_i, X \succeq 0$
+
+## Applicable Problems
+
+- Linear/logistic regression: cross-entropy + linear model is a convex problem with a unique global optimum
+- Weight decay / regularization: $\|w\|_2^2$ and $\|w\|_1$ are both convex regularizers
+- SVM: hinge loss + quadratic regularization = convex problem
+- PCA: the maximum eigenvalue problem for the covariance matrix = a special case of SDP
+- Gram matrix optimization in kernel methods: SDP constraint $K \succeq 0$
+
+## AI Design Translation
+
+- **Convexity diagnosis of loss functions**: Cross-entropy is convex with respect to logits (softmax + NLL), and MSE is convex with respect to linear outputs. However, once composed with nonlinear layers (ReLU, attention), the overall problem becomes non-convex. Maintaining convexity from the last layer to the loss provides a convergence guarantee when designing losses.
+- **Learning rate scheduling and convex optimization convergence rates**: Under strong convexity + smoothness, SGD converges at rate $O(1/T)$; without strong convexity, $O(1/\sqrt{T})$. The condition number $\kappa = L/\mu$ determines convergence speed: the role of BatchNorm / LayerNorm is to reduce $\kappa$ (making the Hessian more isotropic), equivalent to preconditioning. Implemented as standard normalization layers.
+- **Convex relaxation**: Relaxing a non-convex problem into a convex one. Examples: $\ell_0$ sparsity $\to \ell_1$ (LASSO); matrix rank minimization $\to$ nuclear norm minimization; integer programming $\to$ LP relaxation. Implemented by substituting the regularizer or constraint.
+- **Projection onto convex sets**: $\text{proj}_C(x) = \arg\min_{y \in C} \|y - x\|^2$. $\ell_2$-ball projection = $x / \max(1, \|x\|_2/R)$ (elementwise + norm); $\ell_1$-ball projection = soft-thresholding + sort ($O(n\log n)$); box constraints = clamp (elementwise). All are GPU-friendly operations.
+- **Mirror descent**: In non-Euclidean geometries, replaces Euclidean distance with Bregman divergence in gradient updates. For $\ell_1$ constraints (sparse weights), uses exponential gradient $x_{k+1} \propto x_k \exp(-\eta \nabla f)$, implemented as elementwise exp + normalize.
+
+## Engineering Feasibility
+
+- **Primary operations**: Gradient computation = backpropagation (matmul chains); projection = elementwise + norm; convex function evaluation = forward pass. Overall isomorphic to the standard training loop.
+- **GPU friendliness**: Extremely high. First-order methods for convex optimization (gradient descent, projected gradient, mirror descent) map entirely to GPU operators. Second-order methods (Newton, interior point) are viable at moderate scale ($d < 10000$) via cuSOLVER.
+- **Complexity**: Gradient descent per step $O(d)$ (gradient computation $O(\text{model FLOPs})$); projection $O(d)$ to $O(d\log d)$; interior-point methods per step $O(d^3)$.
+- **Low precision**: Strongly convex problems are stable under bf16 (gradients are Lipschitz); when the condition number is large, normalization/preconditioning is needed, otherwise low precision amplifies ill-conditioning.
+
+## Risks and Failure Conditions
+
+- **False convexity**: A seemingly convex loss becomes non-convex after composition with nonlinearities (e.g., $f(W_2 \sigma(W_1 x))$ is non-convex in $W_1, W_2$). Convexity from the last layer to the loss does not imply global convexity.
+- **Condition number degradation**: As $\kappa = L/\mu \to \infty$ ($\mu \to 0$, weakly convex or flat directions), the convergence rate degrades to $O(1/\sqrt{T})$. Solution: add $\ell_2$ regularization to make the problem strongly convex ($\mu \geq \lambda$), or use normalization layers to improve $\kappa$.
+- **Poor scalability of SDP solvers**: Interior-point SDP solvers have complexity $O(n^6)$ ($n$ being the matrix dimension), becoming infeasible beyond $n = 500$. Large-scale settings require first-order methods (ADMM) or approximate convex relaxations.
+- **Convex relaxation gap**: $\ell_1$ relaxation does not necessarily recover the $\ell_0$ sparse solution (requires RIP conditions); nuclear norm relaxation does not necessarily yield the lowest-rank solution. Relaxation quality depends on problem structure.
+
+## Further References
+
+- Distilled notes: references/books/optimization-ml.md (Ch 22 Convex Optimization, Section 22.2 Convex Functions, Section 22.3 Convex Problems, Section 22.4 SDP/LMI)
+- Original text: Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 22 (Convex Optimization Problems Section 22.1-22.4)

package/knowledge-base/optimization/convex-optimization.md

@@ -0,0 +1,50 @@
+# 凸优化 (Convex Optimization)
+
+## 最小定义
+
+在凸集上最小化凸函数的优化问题。凸性保证局部最优即全局最优，且一阶条件（$\nabla f = 0$）是充分必要的。凸问题是优化理论中唯一能保证全局最优且可高效求解的类别。
+
+## 核心公式
+
+- 凸集：$\forall x, y \in C, \theta \in [0,1]: \theta x + (1-\theta)y \in C$
+- 凸函数：$f(\theta x + (1-\theta)y) \leq \theta f(x) + (1-\theta)f(y)$
+- 一阶条件：$f(y) \geq f(x) + \nabla f(x)^T(y-x)$（切线在下方的上方）
+- 二阶条件：$\nabla^2 f(x) \succeq 0$（Hessian 半正定）
+- 标准凸问题形式：$\min f(x)$ s.t. $g_i(x) \leq 0$（$g_i$ 凸）, $Ax = b$
+- 收敛速率（强凸 + 光滑）：$\|x_k - x^*\|^2 \leq (1 - \mu/L)^k \|x_0 - x^*\|^2$，条件数 $\kappa = L/\mu$
+- 半定规划 (SDP)：$\min \langle C, X \rangle$ s.t. $\langle A_i, X \rangle = b_i, X \succeq 0$
+
+## 适用问题
+
+- 线性/逻辑回归：交叉熵 + 线性模型是凸问题，有唯一全局最优
+- 权重衰减 / 正则化：$\|w\|_2^2$、$\|w\|_1$ 都是凸正则项
+- SVM：hinge loss + 二次正则 = 凸问题
+- PCA：协方差矩阵的最大特征值问题 = SDP 的特例
+- 核方法的 Gram 矩阵优化：SDP 约束 $K \succeq 0$
+
+## AI 设计翻译
+
+- **Loss 函数的凸性诊断**：交叉熵对 logits 是凸的（softmax + NLL），MSE 对线性输出是凸的。但一旦过非线性层（ReLU、attention），整体变为非凸。设计 loss 时保持最后一层到 loss 的凸性是收敛保障。
+- **学习率调度与凸优化收敛率**：强凸 + 光滑下 SGD 收敛率 $O(1/T)$，非强凸 $O(1/\sqrt{T})$。条件数 $\kappa = L/\mu$ 决定收敛速度：BatchNorm / LayerNorm 的作用是减小 $\kappa$（使 Hessian 更圆），等价于预条件。实现为标准归一化层。
+- **凸松弛 (Convex Relaxation)**：将非凸问题松弛为凸问题求解。例：$\ell_0$ 稀疏 $\to \ell_1$（LASSO）；矩阵秩最小化 $\to$ 核范数最小化；整数规划 $\to$ LP 松弛。实现为替换正则项或约束。
+- **投影到凸集 (Convex Projection)**：$\text{proj}_C(x) = \arg\min_{y \in C} \|y - x\|^2$。$\ell_2$-ball 投影 = $x / \max(1, \|x\|_2/R)$（elementwise + norm）；$\ell_1$-ball 投影 = soft-thresholding + sort（$O(n\log n)$）；box 约束 = clamp（elementwise）。全是 GPU 友好操作。
+- **镜像下降 (Mirror Descent)**：在非欧几何下用 Bregman 散度代替欧氏距离做梯度更新。对 $\ell_1$ 约束（稀疏权重），用指数梯度 $x_{k+1} \propto x_k \exp(-\eta \nabla f)$，实现为 elementwise exp + normalize。
+
+## 工程可行性
+
+- **主要操作**：梯度计算 = 反向传播（matmul 链）；投影 = elementwise + norm；凸函数评估 = 前向传播。整体与标准训练循环同构。
+- **GPU 友好度**：极高。凸优化的一阶方法（梯度下降、投影梯度、镜像下降）完全映射到 GPU 算子。二阶方法（Newton、内点法）在中等规模（$d < 10000$）可用 cuSOLVER。
+- **复杂度**：梯度下降每步 $O(d)$（梯度计算 $O(\text{model FLOPs})$）；投影 $O(d)$ 到 $O(d\log d)$；内点法每步 $O(d^3)$。
+- **低精度**：强凸问题在 bf16 下稳定（梯度是 Lipschitz 的）；条件数大时需归一化/预条件，否则低精度放大病态。
+
+## 风险与失效条件
+
+- **假凸性**：看似凸的 loss 在复合非线性后变成非凸（如 $f(W_2 \sigma(W_1 x))$ 对 $W_1, W_2$ 非凸）。仅最后一层到 loss 是凸的不代表全局凸。
+- **条件数退化**：$\kappa = L/\mu \to \infty$ 时（$\mu \to 0$，弱凸或平坦方向），收敛率退化为 $O(1/\sqrt{T})$。解决：加 $\ell_2$ 正则使问题强凸（$\mu \geq \lambda$），或用归一化层改善 $\kappa$。
+- **SDP 求解器扩展性差**：内点法 SDP 求解器复杂度 $O(n^6)$（$n$ 为矩阵维度），超过 $n = 500$ 基本不可行。大规模场景需一阶方法（ADMM）或凸松弛后近似。
+- **凸松弛的间隙**：$\ell_1$ 松弛不一定恢复 $\ell_0$ 稀疏解（需 RIP 条件）；核范数松弛不一定给出最低秩解。松弛质量依赖问题结构。
+
+## 深入参考
+
+- 蒸馏稿：references/books/optimization-ml.md（Ch 22 Convex Optimization、§22.2 凸函数、§22.3 凸问题、§22.4 SDP/LMI）
+- 原书：Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 22 (Convex Optimization Problems §22.1-22.4)

package/knowledge-base/optimization/lagrangian-duality.en.md

@@ -0,0 +1,50 @@
+# Lagrangian Duality
+
+## Minimal Definition
+
+Transforms a constrained optimization problem (primal) into a maximization problem over dual variables (dual). The dual function $g(\lambda, \nu) = \inf_x L(x, \lambda, \nu)$ provides a lower bound on the primal optimal value (weak duality); when the duality gap is zero (strong duality), the dual optimal value equals the primal optimal value.
+
+## Core Formulas
+
+- Primal problem: $\min_x f(x) \quad \text{s.t.} \quad g_i(x) \leq 0, \ h_j(x) = 0$
+- Lagrangian: $L(x, \lambda, \nu) = f(x) + \sum_i \lambda_i g_i(x) + \sum_j \nu_j h_j(x)$
+- Dual function: $g(\lambda, \nu) = \inf_x L(x, \lambda, \nu)$ (pointwise infimum over $x$, naturally concave)
+- Weak duality: $d^* \leq p^*$ (dual optimum $\leq$ primal optimum, always holds)
+- Strong duality condition (Slater): convex problem + existence of a strictly feasible point $g_i(x_0) < 0 \implies d^* = p^*$
+- Complementary slackness: $\lambda_i^* g_i(x^*) = 0$ (at optimality, either the constraint is tight or the multiplier is zero)
+- Minimax equivalence: strong duality $\iff \min_x \max_{\lambda \geq 0} L = \max_{\lambda \geq 0} \min_x L$
+
+## Applicable Problems
+
+- GAN / adversarial training: $\min_G \max_D V(D,G)$ is a minimax game; under strong duality, the saddle point = Nash equilibrium
+- SVM dual: primal $O(d)$ dimensions converted to dual $O(n)$ dimensions + kernel Gram matrix; profitable when the number of samples $\ll$ dimensionality
+- Constraint decomposition: large-scale problems decomposed by constraints into subproblems that can be solved in parallel after dual decomposition
+- Dual interpretation of regularization parameters: $\lambda$-regularization $\iff$ a constrained primal problem (e.g., $\|w\|_2^2 \leq C$)
+- Resource allocation / federated learning: global constraints decomposed into Lagrangian relaxations of local subproblems
+
+## AI Design Translation
+
+- **Minimax framework for adversarial training**: $L(\theta, \phi) = \mathbb{E}[\log D_\phi(x)] + \mathbb{E}[\log(1 - D_\phi(G_\theta(z)))]$. $G$ and $D$ alternate between gradient ascent and descent; the core operations are forward + backward passes (matmul chains). Gradient penalty / spectral norm constrains the Lipschitz constant of $D$ to ensure the existence of a saddle point.
+- **SVM dual + kernel trick**: Primal $\min_w \frac{1}{2}\|w\|^2 + C\sum\xi_i$ converted to dual $\max_\alpha \sum\alpha_i - \frac{1}{2}\alpha^T(K \circ yy^T)\alpha$. The kernel Gram matrix $K_{ij} = k(x_i, x_j)$ is computed via matmul (linear kernel) or elementwise operations (RBF kernel). Dual variables $\alpha \in \mathbb{R}^n$ are solved using SMO or gradient projection.
+- **Augmented Lagrangian constrained training**: $\mathcal{L}_{\text{AL}} = f(x) + \sum \lambda_i g_i(x) + \frac{\rho}{2}\sum g_i(x)^2$. The inner loop optimizes $x$ via SGD; the outer loop updates $\lambda$ via gradient ascent: $\lambda \leftarrow [\lambda + \rho g(x)]_+$. The $\rho$ term ensures inner-loop convexity, enabling convergence even for non-convex original problems.
+- **Dual decomposition**: $\min \sum_k f_k(x_k)$ s.t. $\sum x_k \leq b$ decomposes into independent subproblems $\min_{x_k} f_k(x_k) + \lambda^T x_k$, with the master problem $\max_\lambda g(\lambda)$ solved via subgradient ascent. Naturally parallelizable, suitable for distributed training.
+- **Dual perspective on the information bottleneck**: $\min I(Z;X) - \beta I(Z;Y)$ can be formulated as constrained optimization, where $\beta$ is the dual variable. The variational IB relaxes this via the ELBO, reducing to standard VAE training (reparameterization trick + SGD).
+
+## Engineering Feasibility
+
+- **Primary operations**: Lagrangian evaluation = primal objective + weighted sum of constraint terms (elementwise + reduce); dual gradient ascent = standard gradient update; SMO (SVM) = coordinate-descent-style $2 \times 2$ subproblems.
+- **GPU friendliness**: High (dual function evaluation) to moderate (sequential solvers such as SMO). The inner-loop optimization of the augmented Lagrangian is entirely standard SGD (matmul chains), and dual variable updates are elementwise. SMO uses coordinate-wise updates, which are not GPU-friendly, but CPU-sufficient when $n$ is not large.
+- **Complexity**: Dual function evaluation = one forward pass $O(\text{model FLOPs})$; dual gradient ascent per step $O(n)$ (number of constraints); SMO $O(n^2 d)$.
+- **Low precision**: Dual variables $\lambda$ should be kept in fp32 (multipliers span a wide range, with overflow risk under low precision); model parameters can use bf16.
+
+## Risks and Failure Conditions
+
+- **Duality gap for non-convex problems**: Strong duality is only guaranteed for convex problems + Slater's condition. In non-convex neural network training, $d^* < p^*$ is common, and the dual solution does not yield a primal-feasible solution. Solution: augmented Lagrangian (eliminates the gap when $\rho$ is sufficiently large) or SQP.
+- **Dual variable oscillation**: Improper step sizes for gradient ascent on $\lambda$ can cause dual variable oscillation and primal infeasibility. Solution: use adaptive step sizes (Adam updates for $\lambda$) or an increasing $\rho$ schedule in the augmented Lagrangian.
+- **Numerical determination of complementary slackness**: $\lambda_i g_i(x) = 0$ can only be satisfied to $\sim 10^{-6}$ in floating point; strict complementary slackness is unattainable. This affects SVM support vector identification; a threshold must be set.
+- **Mode collapse in minimax training**: The non-convex-non-concave game in GAN $\min\max$ leads to mode collapse or training instability. Additional regularization such as gradient penalty (WGAN-GP) or spectral normalization is required.
+
+## Further References
+
+- Distilled notes: references/books/optimization-ml.md (Ch 23 Lagrangian Duality, Section 23.5 Strong Duality, Section 23.6.3 Slater's Condition)
+- Original text: Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 23 (Lagrangian Duality Section 23.1-23.6) + Chapter 17 (LP Duality)

package/knowledge-base/optimization/lagrangian-duality.md

@@ -0,0 +1,50 @@
+# Lagrange 对偶 (Lagrangian Duality)
+
+## 最小定义
+
+将约束优化问题（primal）转化为对偶变量的最大化问题（dual）。对偶函数 $g(\lambda, \nu) = \inf_x L(x, \lambda, \nu)$ 是原问题最优值的下界（弱对偶），当对偶间隙为零时（强对偶）对偶问题的最优值等于原问题最优值。
+
+## 核心公式
+
+- 原始问题：$\min_x f(x) \quad \text{s.t.} \quad g_i(x) \leq 0, \ h_j(x) = 0$
+- Lagrangian：$L(x, \lambda, \nu) = f(x) + \sum_i \lambda_i g_i(x) + \sum_j \nu_j h_j(x)$
+- 对偶函数：$g(\lambda, \nu) = \inf_x L(x, \lambda, \nu)$（对 $x$ 的逐点下确界，天然凹）
+- 弱对偶：$d^* \leq p^*$（对偶最优 $\leq$ 原问题最优，恒成立）
+- 强对偶条件（Slater）：凸问题 + 存在严格可行点 $g_i(x_0) < 0 \implies d^* = p^*$
+- 互补松弛：$\lambda_i^* g_i(x^*) = 0$（最优时，要么约束紧、要么乘子为零）
+- Minimax 等价：强对偶 $\iff \min_x \max_{\lambda \geq 0} L = \max_{\lambda \geq 0} \min_x L$
+
+## 适用问题
+
+- GAN / 对抗训练：$\min_G \max_D V(D,G)$ 即 minimax 博弈，强对偶时鞍点 = Nash 均衡
+- SVM 对偶：原始 $O(d)$ 维转对偶 $O(n)$ 维 + 核 Gram 矩阵，样本数 $\ll$ 维度时大赚
+- 约束分解：大规模问题按约束拆成子问题，对偶分解后各子问题可并行求解
+- 正则化参数的对偶解释：$\lambda$-正则化 $\iff$ 带约束的原问题（如 $\|w\|_2^2 \leq C$）
+- 资源分配 / 联邦学习：全局约束分解为本地子问题的 Lagrange 松弛
+
+## AI 设计翻译
+
+- **对抗训练的 minimax 框架**：$L(\theta, \phi) = \mathbb{E}[\log D_\phi(x)] + \mathbb{E}[\log(1 - D_\phi(G_\theta(z)))]$。$G$ 和 $D$ 交替梯度上升/下降，核心操作是前向 + 反向传播（matmul 链）。梯度惩罚 / spectral norm 约束 $D$ 的 Lipschitz 保证 minimax 有鞍点。
+- **SVM 对偶 + 核技巧**：原始 $\min_w \frac{1}{2}\|w\|^2 + C\sum\xi_i$ 转对偶 $\max_\alpha \sum\alpha_i - \frac{1}{2}\alpha^T(K \circ yy^T)\alpha$。核 Gram $K_{ij} = k(x_i, x_j)$ 是 matmul（线性核）或 elementwise（RBF 核）。对偶变量 $\alpha \in \mathbb{R}^n$，用 SMO 或梯度投影求解。
+- **增广 Lagrangian 约束训练**：$\mathcal{L}_{\text{AL}} = f(x) + \sum \lambda_i g_i(x) + \frac{\rho}{2}\sum g_i(x)^2$。内层对 $x$ 用 SGD 优化，外层对 $\lambda$ 梯度上升更新：$\lambda \leftarrow [\lambda + \rho g(x)]_+$。$\rho$ 项保证内层凸性，即使原问题非凸也收敛。
+- **对偶分解 (Dual Decomposition)**：$\min \sum_k f_k(x_k)$ s.t. $\sum x_k \leq b$ 分解为各子问题 $\min_{x_k} f_k(x_k) + \lambda^T x_k$ 独立求解，主问题 $\max_\lambda g(\lambda)$ 用次梯度上升。天然可并行，适合分布式训练。
+- **信息瓶颈的对偶视角**：$\min I(Z;X) - \beta I(Z;Y)$ 可写为带约束优化，$\beta$ 是对偶变量。Variational IB 用 ELBO 松弛后变成标准 VAE 训练（reparameterization trick + SGD）。
+
+## 工程可行性
+
+- **主要操作**：Lagrangian 评估 = 原目标 + 约束项的加权求和（elementwise + reduce）；对偶梯度上升 = 标准梯度更新；SMO（SVM）= 坐标下降式的 $2 \times 2$ 子问题。
+- **GPU 友好度**：高（对偶函数评估）到中等（SMO 等串行求解器）。增广 Lagrangian 的内层优化完全是标准 SGD（matmul 链），对偶变量更新是 elementwise。SMO 是坐标式更新，GPU 不友好，但 $n$ 不大时 CPU 即可。
+- **复杂度**：对偶函数评估 = 一次前向传播 $O(\text{model FLOPs})$；对偶梯度上升每步 $O(n)$（约束数）；SMO $O(n^2 d)$。
+- **低精度**：对偶变量 $\lambda$ 应保持在 fp32（乘子范围大，低精度溢出风险）；模型参数可用 bf16。
+
+## 风险与失效条件
+
+- **非凸问题的对偶间隙**：强对偶仅对凸问题 + Slater 条件保证。非凸神经网络训练中 $d^* < p^*$ 常见，对偶解不给出原始可行解。解决：增广 Lagrangian（$\rho$ 足够大时消除间隙）或 SQP。
+- **对偶变量振荡**：$\lambda$ 的梯度上升步长不当会导致对偶变量振荡、原始不可行。解决：使用自适应步长（Adam 更新 $\lambda$）或增广 Lagrangian 的 $\rho$ 递增策略。
+- **互补松弛的数值判定**：$\lambda_i g_i(x) = 0$ 在浮点下只能达到 $\sim 10^{-6}$，严格互补松弛不可得。影响 SVM 支持向量识别等，需设阈值。
+- **Minimax 训练的 mode collapse**：GAN 中 $\min\max$ 的非凸-非凹博弈导致模式坍缩或训练不稳定。需梯度惩罚（WGAN-GP）、谱归一化等额外正则化。
+
+## 深入参考
+
+- 蒸馏稿：references/books/optimization-ml.md（Ch 23 Lagrangian Duality、§23.5 强对偶、§23.6.3 Slater 条件）
+- 原书：Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 23 (Lagrangian Duality §23.1-23.6) + Chapter 17 (LP Duality)

package/knowledge-base/optimization/proximal-method.en.md

@@ -0,0 +1,52 @@
+# Proximal Methods
+
+## Minimal Definition
+
+For non-smooth or non-differentiable objective functions $f(x) = g(x) + h(x)$ ($g$ smooth, $h$ possibly non-differentiable but "simple"), uses the proximal operator $\text{prox}_{\eta h}(v) = \arg\min_x \{h(x) + \frac{1}{2\eta}\|x - v\|^2\}$ in place of the gradient of $h$. Proximal methods encapsulate the non-differentiable part as a closed-form subproblem.
+
+## Core Formulas
+
+- Proximal operator: $\text{prox}_{\eta h}(v) = \arg\min_x \left\{h(x) + \frac{1}{2\eta}\|x - v\|^2\right\}$
+- Proximal gradient descent (ISTA): $x_{k+1} = \text{prox}_{\eta h}(x_k - \eta \nabla g(x_k))$
+- Accelerated proximal gradient (FISTA): $y_k = x_k + \frac{k-1}{k+2}(x_k - x_{k-1})$, $x_{k+1} = \text{prox}_{\eta h}(y_k - \eta \nabla g(y_k))$, convergence rate $O(1/k^2)$ vs. ISTA's $O(1/k)$
+- Soft-thresholding ($\ell_1$ proximal): $\text{prox}_{\eta\|\cdot\|_1}(v)_i = \text{sign}(v_i)\max(|v_i| - \eta, 0)$
+- Projection (proximal of indicator function): $\text{prox}_{\eta \delta_\mathcal{C}}(v) = \text{proj}_\mathcal{C}(v)$
+- Nuclear norm proximal (singular value soft-thresholding): $\text{prox}_{\eta\|\cdot\|_*}(A) = U(\Sigma - \eta I)_+ V^H$
+- Moreau envelope: $h_\eta(v) = \min_x \{h(x) + \frac{1}{2\eta}\|x-v\|^2\}$ (smooth approximation of $h$)
+- ADMM splitting: $\min f(x) + g(z)$ s.t. $Ax + Bz = c$, alternating updates of $x, z, u$ (dual variable)
+
+## Applicable Problems
+
+- Sparse training / $\ell_1$ regularization: weight sparsification, feature selection
+- Low-rank matrix recovery: nuclear norm regularization (matrix completion, robust PCA)
+- Group sparsity / Group Lasso: $\sum_g \|w_g\|_2$ regularization (structured pruning)
+- Operator splitting for constrained optimization: ADMM decomposes complex constraints into simple subproblems
+- Quantization-aware training: modeling weight quantization as a proximal operator (round + straight-through gradient)
+
+## AI Design Translation
+
+- **Soft-thresholding for sparse training**: $\text{prox}_{\eta\lambda\|\cdot\|_1}(w) = \text{sign}(w) \odot \max(|w| - \eta\lambda, 0)$, implemented as `w.sign() * (w.abs() - eta * lam).clamp(min=0)`, pure elementwise, $O(d)$, zero additional memory. Applying soft-thresholding after each SGD update yields sparse weights.
+- **Singular value soft-thresholding for low-rank regularization**: $\text{prox}_{\eta\|\cdot\|_*}(W) = U(\Sigma - \eta)_+ V^H$. Requires SVD; for large matrices, approximate with randomized SVD: first perform randomized SVD to rank $r$, then apply elementwise soft-thresholding to $\Sigma$, and reconstruct. Core operations are matmul chains + elementwise.
+- **Group Lasso structured pruning**: $\text{prox}_{\eta\sum_g\|w_g\|_2}(w)_g = w_g \cdot \max(1 - \eta/\|w_g\|_2, 0)$. After grouping by channel/head, each group undergoes independent soft-thresholding (norm + elementwise scale), $O(d)$. Implemented as reshape + norm(dim) + clamp + mul.
+- **ADMM for distributed training**: $\min \sum_i f_i(x_i) + g(z)$ s.t. $x_i = z$. Each node independently updates $x_i$ (local SGD), the server updates $z = \text{prox}_{g/\rho}(\bar{x} + u)$ (aggregation + proximal), and $u$ is updated as the dual variable. More communication-efficient than all-reduce (only $x_i$ and $z$ need to be transmitted).
+- **Quantization proximal operator**: Models weight quantization as $\text{prox}(w) = \Delta \cdot \text{round}(w/\Delta)$, with backpropagation using the straight-through estimator (STE): $\partial \text{prox}/\partial w \approx 1$. Implemented as `w_q = (w / delta).round() * delta`; the forward pass is elementwise round + mul, the backward pass is identity.
+
+## Engineering Feasibility
+
+- **Primary operations**: Most proximal operators are elementwise (soft-thresholding, clamp, group norm) or matmul + small SVD (nuclear norm). Gradient steps = standard backpropagation.
+- **GPU friendliness**: Extremely high. $\ell_1$ proximal = elementwise; group lasso proximal = reshape + norm + scale = elementwise; nuclear norm proximal = matmul + small SVD. FISTA's momentum term is also elementwise. ADMM's communication pattern is compatible with data parallelism.
+- **Complexity**: ISTA/FISTA per step = one gradient computation + one proximal operator ($O(d)$ elementwise); nuclear norm proximal = $O(nd^2)$ (reduced to $O(ndk)$ via randomized SVD); ADMM per node = local SGD + $O(d)$ communication.
+- **Low precision**: Elementwise proximal operators are stable under bf16 (no delicate numerical operations). SVD-based proximals must be computed in fp32. FISTA's momentum accumulation may lose precision under bf16; storing $y_k$ in fp32 is recommended.
+
+## Risks and Failure Conditions
+
+- **SVD overhead of nuclear norm proximal**: Full SVD of large matrices at every step is $O(n^3)$, infeasible. Solution: (1) randomized SVD approximation; (2) factorize $\|W\|_* = \min_{W=UV^T} \frac{1}{2}(\|U\|_F^2 + \|V\|_F^2)$, converting to smooth optimization over $U, V$, eliminating the need for SVD.
+- **FISTA restart issues**: FISTA's $y_k$ sequence may oscillate (non-monotonically), causing actual convergence to be slower than theory predicts. Solution: adaptive restart (reset momentum $t_k = 1$ when $f(x_{k+1}) > f(x_k)$), implemented as a simple if-condition.
+- **Sensitivity of ADMM's $\rho$ parameter**: $\rho$ too large causes ill-conditioning of subproblems; $\rho$ too small causes slow dual convergence. Solution: adaptive $\rho$ (automatically adjusted based on the primal/dual residual ratio), updating every several steps as $\rho \leftarrow \rho \cdot \tau$ ($\tau > 1$ if primal residual $>$ dual residual).
+- **No closed-form proximal operator**: Not all $h(x)$ have closed-form prox. Complex regularizers (e.g., TV regularization, overlapping group lasso) require inner-loop iterative solvers. Solution: use Dykstra's algorithm to decompose into compositions of simple prox operators, or switch to ADMM splitting.
+- **Bias of the straight-through estimator**: The STE gradient for the quantization proximal is biased ($\partial \text{round}/\partial w = 0$ almost everywhere), and long-term training may diverge. Solution: compensate with a learnable scale factor, or use stochastic rounding (noisy quantization).
+
+## Further References
+
+- Distilled notes: references/books/optimization-ml.md (Ch 8 Gradient Methods Section 8.3 Convergence Analysis, Ch 11 Quasi-Newton Section 11.5 BFGS, Ch 24 Constrained Algorithms Section 24.5 Augmented Lagrangian)
+- Original text: Chong, Lu, Zak, *An Introduction to Optimization* 5th Ed., Chapter 24 (Constrained Algorithms Section 24.5 Augmented Lagrangian) + Parikh & Boyd, *Proximal Algorithms*, Foundations and Trends in Optimization, 2014