math-skill 2.1.0 → 3.0.1

package/references/books/matrix-analysis.en.md

@@ -0,0 +1,146 @@
+# Matrix Analysis
+
+> Roger A. Horn & Charles R. Johnson, *Matrix Analysis*, 2nd Edition, Cambridge University Press, 2013 (ISBN 978-0-521-83940-2). A graduate-level classic of matrix theory unified by the theme of **canonical forms**.
+
+## Overview
+
+This is the authoritative reference for upgrading "linear algebra" to "matrix analysis": not just computing with matrices, but studying invariants under **similarity / unitary equivalence / congruence** transformations, canonical forms, eigenvalue location and perturbation, norm geometry, and positive-definite/nonnegative structures. For AI/ML/GPU, it is the **backbone closest to the hardware-level operators** in the v2 reading list -- GEMM, numerical stability, low-rank compression, and second-order optimization all trace their roots here.
+
+Actual chapter map (from the table of contents):
+
+- **Ch 0 Review and Miscellanea**: rank, nonsingularity, inner products, block matrices -- quick-reference foundation.
+- **Ch 1 Eigenvalues, Eigenvectors, and Similarity**: characteristic equations, characteristic polynomials, algebraic/geometric multiplicity, similarity (Sec. 1.1-1.4).
+- **Ch 2 Unitary Similarity and Unitary Equivalence**: QR decomposition (Sec. 2.1), Schur triangularization (Sec. 2.4), normal matrices (Sec. 2.5), **SVD (Sec. 2.6)**, CS decomposition (Sec. 2.7).
+- **Ch 3 Canonical Forms for Similarity and Triangular Factorizations**: Jordan canonical form (Sec. 3.1), minimal polynomials and companion matrices (Sec. 3.3), real Jordan and Weyr forms (Sec. 3.4), triangular factorization LU (Sec. 3.5).
+- **Ch 4 Hermitian, Symmetric Matrices, and Congruences**: variational characterization Courant-Fischer (Sec. 4.2), eigenvalue inequalities Weyl/interlacing (Sec. 4.3), congruences and Sylvester's law of inertia (Sec. 4.5).
+- **Ch 5 Norms for Vectors and Matrices**: norms and inner products, dual norms (Sec. 5.5), **matrix norms (Sec. 5.6-5.7)**, **condition numbers (Sec. 5.8)**.
+- **Ch 6 Location and Perturbation of Eigenvalues**: Gershgorin discs (Sec. 6.1-6.2), eigenvalue perturbation theorems (Sec. 6.3).
+- **Ch 7 Positive Definite and Semidefinite Matrices**: polar decomposition and SVD (Sec. 7.3-7.4), **Schur product theorem (Sec. 7.5)**, simultaneous diagonalization (Sec. 7.6), Loewner partial order and block matrices (Sec. 7.7), positive-definite inequalities (Sec. 7.8).
+- **Ch 8 Positive and Nonnegative Matrices**: Perron-Frobenius (Sec. 8.2-8.5), stochastic and doubly stochastic matrices (Sec. 8.7).
+- Appendices A-F: complex numbers, **convex sets and functions (B)**, fundamental theorem of algebra, eigenvalue continuity, compactness, canonical pairs (F).
+
+**Boundary reminder for activation**: this book is **theory-first** -- addressing existence, characterizations, inequalities, and canonical forms, **not numerical algorithm recipes**. Specific algorithm implementations, convergence constants, and stability details (e.g., communication lower bounds for blocked QR, actual complexity coefficients for SVD) require a companion numerical linear algebra textbook (Golub-Van Loan / Trefethen-Bau). What is provided here is an **activation index** for "which structure to use + why + can the GPU compute it"; for implementation details, consult those books.
+
+## Core Structures Transferable to AI/Infra
+
+| Mathematical structure (chapter) | Transfer to ML / algorithms / Infra |
+|---|---|
+| **SVD / low-rank (Sec. 2.6, 7.4)** | The foundation of all low-rank compression: LoRA, PCA/whitening, Eckart-Young optimal low-rank approximation, KV-Cache low-rank reduction, weight compression |
+| **Spectrum and similarity invariants (Ch 1)** | Hessian/gradient covariance spectra, spectral radius determines stability of linear attention / SSM / RNN, trace = parameter counting / regularization |
+| **Schur triangularization + normal matrices (Sec. 2.4-2.5)** | Foundation for numerical EVD algorithms (QR algorithm); normal <=> unitarily diagonalizable, the criterion for "well-behaved spectra" |
+| **Variational characterization Courant-Fischer (Sec. 4.2)** | Rayleigh quotients, spectral normalization, spectral clustering, PCA as min-max; largest singular value = operator norm |
+| **Eigenvalue perturbation Weyl/Bauer-Fike (Sec. 4.3, 6.3)** | Spectral shift bounds under quantization/low-precision/pruning, training perturbation robustness, stability certificates |
+| **Matrix norms + duality (Sec. 5.5-5.7)** | Spectral norm (gradient clipping / Lipschitz), Frobenius (weight decay), **nuclear norm = dual of spectral norm** (low-rank regularization) |
+| **Condition numbers (Sec. 5.8)** | Numerical stability diagnostics, preconditioning, why bf16 training diverges |
+| **Polar decomposition + Newton-Schulz (Sec. 7.3)** | Orthogonalizing gradients/weights (Muon optimizer, orthogonal initialization), computable with pure GEMM |
+| **Positive definiteness / PSD (Ch 7)** | Kernel methods, covariances, attention Gram matrices, second-order method preconditioners, Loewner partial order for matrix inequalities |
+| **Schur product theorem (Sec. 7.5)** | Hadamard products preserve PSD -- learnable kernel engineering, gating that does not destroy positive-definite structure |
+| **Perron-Frobenius / stochastic matrices (Ch 8)** | Row-stochastic attention mixing and collapse (over-smoothing), PageRank, graph propagation, spectral gap = expressivity |
+
+The table above is organized into four **activation families** for convenient retrieval:
+
+- **Spectral family (Ch 1-3)**: eigenvalues / similarity / canonical forms -- answering "is the dynamics stable, what does the spectrum look like." Note that Jordan/Weyr forms are theoretical tools; numerically one turns to Schur/SVD.
+- **Norm family (Ch 5-6)**: norms / duality / condition numbers / perturbation -- answering "how do errors propagate, is low precision stable, should we precondition."
+- **Positive-definite family (Ch 4, 7)**: Hermitian / PSD / polar decomposition / Loewner -- answering "second-order structure, kernels, covariances, nearest orthogonal matrix."
+- **Nonnegative family (Ch 8)**: Perron-Frobenius / stochastic matrices -- answering "propagation, mixing, collapse, stationary distributions."
+
+## Key Bridging Facts (Activation Shorthand)
+
+When "activating" this book into algorithms, these are the most frequently used connecting facts -- memorizing them enables rapid navigation between structures:
+
+- **Condition number kappa_2(A) = sigma_max / sigma_min (Sec. 5.8)**: directly predicts how many orders of magnitude errors are amplified under bf16/fp8.
+- **SVD <=> eigendecomposition of A^H A and A A^H, sigma = sqrt(lambda) (Sec. 2.6, 7.4)**: singular values are the square roots of the Gram matrix eigenvalues.
+- **Spectral norm = sigma_max, Frobenius = sqrt(Sum sigma^2), nuclear norm = Sum sigma (Sec. 5.6)**: all three major matrix norms are determined entirely by singular values; spectral norm and nuclear norm are dual to each other.
+- **Normal matrices <=> unitarily diagonalizable (Sec. 2.5)**: the only class where "eigenvalue = singular value structure is well-conditioned"; for non-normal matrices one must look at pseudospectra.
+- **Positive definite <=> all eigenvalues > 0 <=> all leading principal minors > 0 (Sec. 7.1)**: the latter is a computable positive-definiteness test (Sylvester's criterion).
+- **Polar decomposition A = UP, P = (A^H A)^{1/2} (Sec. 7.3)**: Newton-Schulz iteration converges to the orthogonal factor U.
+- **Weyl perturbation bound |lambda_i(A+E) - lambda_i(A)| <= ||E||_2 (Sec. 4.3)**: one line gives the spectral shift upper bound for quantization/pruning.
+- **Row-stochastic matrix spectral radius = 1 (Sec. 8.7)**: the Perron root of attention is always 1; the spectral gap determines the collapse rate.
+
+## Problem Types Suited for Activation
+
+- **Low-rank / compression**: where is the redundancy in attention, KV-Cache, weights, gradients? How low can the rank go? How to estimate the optimal approximation error from truncation (Eckart-Young)? Should low-rank regularization use the nuclear norm or explicit parameterization?
+- **Numerical stability**: why does low-precision (bf16/fp8) training diverge? How to monitor condition numbers and spectral radii online during training? Are there bounds on spectral shifts from quantization/pruning (Weyl, Bauer-Fike)? Which operators need reparameterization for stability?
+- **Spectral design**: the operator norms behind normalization (spectral normalization / BatchNorm); spectral radius constraints for recurrent / state-space models (SSM); spectral gap determines expressivity and separability.
+- **Second-order optimization**: PSD structure and negative curvature of Hessian / Fisher (inertia law for detecting saddle points); condition number improvement via preconditioners; Kronecker-factor approximations (K-FAC / Shampoo).
+- **Graphs / propagation**: stability and over-smoothing of message passing; mixing time of row-stochastic operators; Markov chain stationary distributions and spectral gaps.
+
+## Possible Algorithmic Inspirations
+
+> Each item is tagged with the **eight-dimension touchpoint** (corresponding to the dimension numbers in `../gpu-friendly-math.md`) for direct entry into the GPU acceptance gate.
+
+1. **Randomized numerical linear algebra (randomized NLA)**: using random projections + QR (Sec. 2.1) for randomized SVD, reducing the O(n^3) full decomposition to sub-quadratic, producing low-rank sketches of very large weights/activations. *Touchpoint: D2/D3 -- all GEMM, manageable complexity.*
+2. **Low-rank attention / KV compression**: using Eckart-Young (Sec. 7.4) to guarantee truncated SVD is the optimal low-rank approximation; using the **nuclear norm (dual of spectral norm, Sec. 5.5)** as low-rank regularization, projecting the KV-Cache into a low-dimensional subspace. *Touchpoint: D2/D4 -- GEMM chains + memory compression.*
+3. **Spectral normalization**: power iteration to estimate the largest singular value (operator norm, Sec. 5.6), constraining per-layer Lipschitz constants -- GANs/diffusion/stable training. *Touchpoint: D1/D6 -- matvec, but serial iteration requires blocking.*
+4. **Newton-Schulz orthogonalization (Muon-style)**: polar decomposition (Sec. 7.3) projects gradient matrices onto the nearest orthogonal matrix, with iterations involving only matrix multiplications -- currently the most GPU-friendly "second-order-flavored" update. *Touchpoint: D2/D6/D8 -- pure GEMM, fusible, bf16-stable.*
+5. **Preconditioning / Shampoo / K-FAC**: condition numbers (Sec. 5.8) diagnose ill-conditioning, using PSD Kronecker factors (Ch 7) to approximate the Hessian for preconditioning, rounding out ill-conditioned loss landscapes. *Touchpoint: D2/D5 -- small-matrix GEMM, watch precision for inverse operations.*
+6. **Gershgorin cheap spectral radius gate (Sec. 6.1)**: using the disc bound O(n^2) in the training loop to quickly estimate the spectral radius as a low-cost stability gate, without running a full EVD. *Touchpoint: D1/D3 -- per-row summation, extremely cheap.*
+7. **PSD kernel engineering (Schur product theorem, Sec. 7.5)**: composing multiple PSD kernels via Hadamard products, guaranteeing that learnable similarity matrices remain positive semidefinite. *Touchpoint: D1 -- element-wise tensor products, naturally friendly.*
+8. **Perron-Frobenius diagnostics (Sec. 8.2-8.5)**: treating row-stochastic attention as a Markov operator, using the spectral gap to quantify over-smoothing / rank collapse, guiding residual and temperature design. *Touchpoint: D1/D3 -- cheap spectral estimation, avoids deep-layer collapse.*
+9. **Blocked / communication-avoiding decompositions**: writing QR, Cholesky (Sec. 3.5) in blocked versions, replacing column-by-column elimination with GEMM, reducing communication rounds across devices. *Touchpoint: D2/D6 -- reforming serial recurrences into parallelism + overlap.*
+
+## GPU Friendliness Warning
+
+> The scoring dimensions reference the **eight-dimension checklist** in `../gpu-friendly-math.md` (Tensorization / GEMM-mappability / Complexity / Memory / Low-precision / Parallelism / Sparsity / Operator fusion); definitions are not repeated here.
+
+**Natively friendly (math beautiful x GPU friendly):**
+- **Truncated SVD / low-rank**: expressed as GEMM chains (D2), compressing KV-Cache/weights (D4).
+- **Frobenius / spectral norms, Gram matrices, Hadamard products**: batched tensor algebra (D1, D2).
+- **Polar decomposition Newton-Schulz**: pure matrix-multiplication iteration (D2, D6, D8 fusible), robust under bf16.
+- **Gershgorin discs**: O(n^2) per-row summation (D1), cheap stability estimation.
+- **Well-conditioned PSD blocked Cholesky / Gram construction**: blocked form yields GEMM chains (D2), commonly used in kernel methods and covariance preconditioning.
+
+**Beautiful but not computable:**
+- **Jordan canonical form (Sec. 3.1)** -- the classic counter-example: eigenvalue multiplicities are extremely sensitive to perturbation, fundamentally unreliable under floating-point, **never use as a numerical tool** (violates D5 low-precision stability). Weyr form has the same issue.
+- **Full EVD / SVD at O(n^3)** -- blows up for large matrices (violates D3), must switch to randomized/iterative methods.
+- **Non-normal matrices (Sec. 2.5 and beyond)** -- eigenvalues do not reflect true behavior, pseudospectra are needed; spectral distortion under low precision (D5).
+- **Ill-conditioned / high condition number (Sec. 5.8)** -- catastrophic cancellation occurs, requiring fp64 for correctness, conflicting with bf16/fp8 training (D5).
+- **Serial dependencies in QR / Cholesky (Sec. 2.1, 3.5)** -- naive implementations are long serial recurrences (violates D6), requiring blocked / communication-avoiding variants.
+- **Serial iteration in power iteration** -- single-vector iteration has low parallelism; must be blocked (block / subspace iteration) to saturate SMs (D6).
+
+**Reform strategies (echoing the Make-It-Computable Toolkit in `../gpu-friendly-math.md`):**
+- Full EVD/SVD -> **randomized + truncated** to reduce complexity (D3);
+- Exact decompositions -> **blocked / GEMM-ified** to eliminate serial dependencies (D2/D6);
+- Ill-conditioned/non-normal -> **reparameterization + spectral normalization** to stabilize low precision (D5);
+- Non-computable canonical forms like Jordan -> retain only for **theoretical proofs**, numerically switch to Schur/SVD.
+
+## Which Design Lens to Invoke
+
+Used in conjunction with the design lenses in `../../lenses/`:
+
+- **`duality`**: similarity / unitary equivalence / congruence, SVD, diagonalization -- the soul of this book is "change coordinates to reveal structure."
+- **`algorithmic`**: power iteration, Newton-Schulz, QR algorithm, randomized NLA -- turning theorems into runnable kernels.
+- **`variational`**: variational characterizations (Sec. 4.2), condition numbers and preconditioning, second-order methods, matrix inequalities under the Loewner partial order.
+- **`symmetry`**: unitary invariance, similarity invariants (eigenvalues/trace/determinant), well-conditioned spectra of normal matrices.
+- **`categorical`**: canonical forms as "representatives of equivalence classes" -- capturing essence through the simplest morphology, ignoring coordinate details.
+- **`probabilistic`**: randomized NLA, random matrix spectra, Perron-Frobenius / Markov chain stationary distributions.
+
+## Anti-patterns
+
+- **Using Jordan form as a numerical algorithm**: it is not computable under floating-point, only useful for theoretical analysis; do not put it into a kernel.
+- **Defaulting to full SVD/EVD**: in large-scale settings one should use randomized/truncated/iterative methods, otherwise O(n^3) will bog everything down.
+- **Looking only at eigenvalues while ignoring non-normality**: eigenvalues of non-normal matrices do not predict transient behavior; look at singular values / pseudospectra instead.
+- **Assuming all matrices are well-conditioned**: deploying bf16/fp8 without monitoring condition numbers, then debugging after divergence.
+- **Using the nuclear norm while forgetting it requires SVD**: the nuclear norm is an elegant low-rank regularizer, but its computation depends on SVD, requiring proximal/randomized techniques.
+- **Assuming exact positive definiteness numerically**: under floating-point, Gram/covariance matrices can lose positive definiteness; add jitter (diagonal perturbation) or use pivoted Cholesky.
+- **Using Frobenius norm as a low-rank regularizer**: Frobenius / weight decay suppresses "energy" not "rank"; for low rank, use the nuclear norm or explicit low-rank parameterization (e.g., LoRA).
+- **Discussing "eigenvalue magnitudes" for non-symmetric matrices**: when measuring energy/norm/stability margins, look at **singular values**; the modulus of eigenvalues of non-normal matrices is seriously misleading (transient growth far exceeds what the spectral radius predicts).
+- **Materializing the full O(n^2) Gram matrix**: attention/kernel matrices without blocking will blow up memory; use FlashAttention-style fusion + blocking (echoing GPU dimensions D4/D8).
+- **Stacking theorems without diagnosing the bottleneck**: first ask "is the algorithmic bottleneck spectral, low-rank, or stability," then select the structure; do not dump matrix theory upfront.
+
+## Deep Dive Entry
+
+> **Bibliographic info**: Roger A. Horn & Charles R. Johnson, *Matrix Analysis*, 2nd Edition, Cambridge University Press, 2013. ISBN 978-0-521-83940-2.
+>
+> **Activation method**: Place `Matrix Analysis.pdf` in the `math_book/` folder at the project root; the Agent can then automatically search the original text. The PDF is not distributed via npm/git (copyright reasons); obtain it independently.
+
+**Full-fidelity lookup = have the Agent automatically search the local PDF `math_book/Matrix Analysis.pdf`**: extract with `pdftotext` -> `grep` to locate keywords/theorem names -> `Read` the relevant pages for close study. This file is an "activation index," not a substitute; when precise statements, proofs, or constants are needed, go back to the original book.
+
+Actual chapters worth deep reading:
+
+- **Sec. 2.6 The singular value decomposition** -- the origin of all low-rank compression / LoRA / PCA.
+- **Sec. 4.2-4.3 Variational characterizations & eigenvalue inequalities** -- Courant-Fischer min-max and Weyl inequalities, the theoretical roots of spectral normalization and perturbation bounds.
+- **Sec. 5.6-5.8 Matrix norms & condition numbers** -- all the criteria for numerical stability, gradient clipping, and preconditioning are here.
+- **Sec. 7.3-7.5 Polar/SVD & the Schur product theorem** -- the direct source for Muon orthogonalization and PSD kernel engineering.
+- **Sec. 6.1-6.3 Gershgorin discs & perturbation theorems** -- cheap spectral localization and perturbation robustness.
+- **Sec. 8.2-8.5 Perron-Frobenius theory** -- row-stochastic attention, graph propagation, over-smoothing analysis.

package/references/books/matrix-analysis.md

@@ -1,4 +1,4 @@
-# 🧮 矩阵分析 / Matrix Analysis
+# 矩阵分析 / Matrix Analysis
 
 > Roger A. Horn & Charles R. Johnson, *Matrix Analysis*, 2nd Edition, Cambridge University Press, 2013（ISBN 978-0-521-83940-2）。以**标准型（canonical forms）作为统一主题**的研究生级矩阵理论经典。
 
@@ -104,16 +104,16 @@
 - 病态/非正规 → **重参数化 + 谱归一化**稳低精度（维度 5）；
 - Jordan 等不可算标准型 → 只留作**理论证明**，数值上换 Schur/SVD。
 
-## 该调用哪个思想武器
+## 该调用哪个思想透镜
 
-配合 `skills/` 下的现有武器使用：
+配合 `../../lenses/` 下的思想透镜使用：
 
-- **`transformation`（变换）**：相似 / 酉等价 / 合同、SVD、对角化——本书的灵魂就是"换坐标让结构显形"。
-- **`algorithmic-thinking`（算法与计算）**：power iteration、Newton–Schulz、QR algorithm、随机化 NLA——把定理变成可跑的 kernel。
-- **`optimization`（优化）**：变分刻画（§4.2）、条件数与预条件、二阶法、Loewner 偏序下的矩阵不等式。
-- **`symmetry-invariance`（对称与不变性）**：酉不变性、相似不变量（特征值/迹/行列式）、正规矩阵的良态谱。
-- **`abstraction`（抽象化）**：标准型即"等价类的代表元"——用最简形态抓住本质、忽略坐标细节。
-- **`probability-statistics`（概率与统计）**：随机化 NLA、随机矩阵谱、Perron–Frobenius/Markov 链平稳分布。
+- **`duality`（对偶）**：相似 / 酉等价 / 合同、SVD、对角化——本书的灵魂就是"换坐标让结构显形"。
+- **`algorithmic`（算法）**：power iteration、Newton–Schulz、QR algorithm、随机化 NLA——把定理变成可跑的 kernel。
+- **`variational`（变分）**：变分刻画（§4.2）、条件数与预条件、二阶法、Loewner 偏序下的矩阵不等式。
+- **`symmetry`（对称与不变性）**：酉不变性、相似不变量（特征值/迹/行列式）、正规矩阵的良态谱。
+- **`categorical`（范畴化）**：标准型即"等价类的代表元"——用最简形态抓住本质、忽略坐标细节。
+- **`probabilistic`（概率统计）**：随机化 NLA、随机矩阵谱、Perron–Frobenius/Markov 链平稳分布。
 
 ## 反模式
 
@@ -130,7 +130,7 @@
 
 ## 深挖入口
 
-> **📖 书目信息**：Roger A. Horn & Charles R. Johnson, *Matrix Analysis*, 2nd Edition, Cambridge University Press, 2013. ISBN 978-0-521-83940-2.
+> **书目信息**：Roger A. Horn & Charles R. Johnson, *Matrix Analysis*, 2nd Edition, Cambridge University Press, 2013. ISBN 978-0-521-83940-2.
 >
 > **启用方式**：将 `Matrix Analysis.pdf` 放入项目根目录的 `math_book/` 文件夹，Agent 即可自动搜索原文。PDF 不随 npm/git 分发（版权原因），需自行获取。
 

package/references/books/micro-lie-theory.en.md

@@ -0,0 +1,116 @@
+# A Micro Lie Theory
+
+> **A micro Lie theory for state estimation in robotics** -- Joan Solà, Jeremie Deray, Dinesh Atchuthan.
+> arXiv:1812.01537v9 [cs.RO], 2021-12-08, main text approx. 17 pages + appendix formula handbook. Companion open-source C++ header library **manif** (<https://github.com/artivis/manif>, implementing SO(2)/SO(3)/SE(2)/SE(3) with analytic Jacobians).
+> This file is an "activation" distillation, **not a restatement of the original**; for full fidelity, consult the local PDF (see (Deep Dive Entry)).
+
+## Overview
+
+This paper **deliberately "amputates"** the vast edifice of Lie theory down to the minimal subset needed for robotic state estimation: it does not even introduce the Lie bracket, instead reducing the Lie algebra directly to the isomorphic vector space R^n, so that the "curved, nonlinear" Lie group becomes "flat, linear" vector algebra in the tangent space. The one-sentence thesis: **virtually every operation on the group (curved) has an exact counterpart in the Lie algebra (flat)** (Fig. 1) -- this is the modern use of Sophus Lie's "dragon-slaying sword" from inspiration.md: the dragon (solving differential equations) was never slain, yet the sword became a supreme tool for chopping vegetables (linearizing nonlinearities), carving (describing symmetries), and building houses (robotic state estimation).
+
+Actual structure map:
+- **II. A micro Lie theory (core)**: A Lie groups (group axioms + manifold) -> B group actions -> C tangent space and Lie algebra (hat `^wedge` / vee `^vee` operators) -> D **exponential map** exp/log and capital Exp/Log (Eq. 23-24) -> E **addition/subtraction operators** oplus/ominus (Eq. 25-28, left and right versions) -> F **adjoint** Ad_X and adjoint matrix (Eq. 30-35) -> G **derivatives on Lie groups**: right Jacobian (Eq. 41a-c) / left Jacobian (Eq. 44) -> H **uncertainty on manifolds and covariance propagation** (Eq. 52) -> I discrete integration on manifolds.
+- **III. Manifold differentiation rules**: A chain rule, B elementary Jacobian blocks, C derived Jacobian blocks.
+- **IV. Composite manifolds / bundles**: assembling heterogeneous large states into non-interacting manifold blocks, differentiating block by block (Eq. 84-90, Ex. 7 comparing SE(n) vs T(n)xSO(n) vs (R^n, SO(n))).
+- **V. Applications**: landmark-based localization and mapping -- error-state Kalman filter on manifolds (Eq. 92-96), graph-optimization SLAM, sensor self-calibration.
+- **VI. Conclusion + Appendix**: complete closed-form formulas for S^1/SO(2)/SE(2), S^3/SO(3), SE(3) (including right/left Jacobians, e.g. SO(3) Eq. 143-145).
+
+## Core Structures Transferable to AI/Infra
+
+The paper welds together the "curved nonlinear group" and the "flat linear vector space" using five tools, each of which can be directly ported into differentiable algorithms.
+
+### 1. exp/log Maps -> Manifold Parameterization (Eq. 23-24, Fig. 1)
+
+`Exp: R^n -> M`, `Log: M -> R^n` are the bijective bridges between the manifold and its tangent space (isomorphic to linear R^n). Transfer usage: **let the network freely predict in the unconstrained tangent space R^n, then map back to the valid manifold via Exp**. Rotations, poses, and unit quaternions thus naturally satisfy constraints (orthogonality, unit norm), eliminating the need for various ad-hoc post-hoc orthogonalization or normalization fixes on regressed matrices/quaternions.
+
+### 2. oplus/ominus Addition/Subtraction Operators -> "Vector Algebra" on Manifolds (Eq. 25-28)
+
+`oplus` (one Exp + one group composition) makes "adding an increment to a manifold state" look like ordinary vector addition; `ominus` computes the difference between two elements (geodesic residual). **Key pitfall**: because group composition is non-commutative, oplus/ominus come in **right versions** (increment in local frame / tangent at X) and **left versions** (increment in global frame / tangent at identity E); the paper defaults to the right version -- in engineering, one must stay consistent throughout; mixing conventions leads to errors.
+
+### 3. Right/Left Jacobians -> Geometrically Correct Backpropagation (Eq. 41a-c, 44)
+
+The linearization of a manifold function with respect to tangent-space perturbations is the Jacobian; the chain rule on manifolds takes the form `D(g o f)/DX = J_g . J_f` (Sec. III-A), isomorphic to the Euclidean chain rule, so it plugs directly into automatic differentiation. The **right Jacobian** (Eq. 41) and **left Jacobian** (Eq. 44) describe the same derivative in local vs. global tangent spaces, related to each other through the adjoint Ad.
+
+### 4. Adjoint Matrix Ad_X -> Transporting Tangent Vectors Across Frames (Eq. 30-35)
+
+`Ad_X` transports the same tangent vector between different points and left/right frames (Eq. 32: `X oplus tau = (Ad_X tau) oplus X`). It is the **algebraic realization of equivariance**, and also the transformation matrix for propagating perturbations/covariances across frames; Ex. 6 gives the closed-form adjoint matrix for SE(3).
+
+### 5. Covariance Propagation -> Uncertainty on Manifolds (Eq. 52)
+
+`Sigma = E[(X om Xbar)(X om Xbar)^T] in R^{m x m}` -- **the covariance lives in the tangent space R^n, not on the group**. Defining variance directly on the group is ill-posed; the tangent-space representation brings the uncertainty computations of Kalman filters / factor graphs back to familiar linear algebra.
+
+Two additional directly usable "components":
+- **SE(3) / SO(3)**: rigid body motion group / rotation group -- the unified differentiable representation of pose = rotation + translation, the standard language for cameras, point clouds, and robotic arm joints.
+- **Composite manifolds (bundles, Eq. 84-90)**: assembling multi-pose + feature + calibration heterogeneous large states into non-interacting manifold blocks, with Jacobians **assembled block by block** (Eq. 89 is block-structured sparse), i.e., "assemble the Jacobian matrix block by block" in engineering practice.
+
+## Problem Types Suited for Activation
+
+- Prediction/regression targets carry **geometric constraints**: rotations, poses, unit vectors, SPD matrices -- forcing them into Euclidean MLPs breaks the constraints.
+- Need **equivariance / invariance**: when the input undergoes rigid body transformations, the output should covary or remain invariant (point clouds, molecules, multi-view geometry).
+- Need **explicit uncertainty**: pose estimation, sensor fusion, SLAM, visual odometry covariance propagation.
+- **Iterative geometric optimization**: camera/point cloud registration, bundle adjustment, inverse kinematics, requiring gradient/Gauss-Newton on manifolds rather than in Euclidean space.
+- States **evolve on Lie groups**: inertial pre-integration, motion models, differentiable physics/control.
+
+## Possible Algorithmic Inspirations
+
+- **SE(3)/SO(3) equivariant networks**: using group elements as hidden states, updating between layers via Exp/opus, transporting features via Ad; equivariance comes from group actions rather than data augmentation.
+- **Manifold loss for pose regression**: using ominus (geodesic error) as the loss, rather than Euclidean MSE on rotation matrices/quaternions; naturally handles manifold topology (e.g., quaternion double cover).
+- **Lie group optimizers**: rewriting Adam/Gauss-Newton in the retraction form "compute gradient in tangent space -> map back via Exp" (error-state approach, corresponding to the manifold EKF in Sec. V-A).
+- **Differentiable registration / alignment**: camera pose, point cloud ICP, multi-view BA formulated as differentiable optimization layers on SE(3), trained end to end.
+- **Manifold-parameterized output heads**: the network outputs an R^n tangent vector, mapped back to SO(3)/SE(3) via Exp, replacing the various ad-hoc orthogonalization schemes for 6D/9D rotation representations.
+- **Uncertainty-aware fusion**: using the tangent-space covariance from Eq. 52 + Ad propagation to build learnable Kalman/factor-graph layers.
+
+## GPU Friendliness Warning
+
+> The sole authority for the acceptance gate: [`../gpu-friendly-math.md`](../gpu-friendly-math.md) (eight dimensions: D1 Tensorization, D2 GEMM-mappability, D3 Complexity, D4 Memory/KV, D5 Low-precision stability, D6 Parallelism & communication, D7 Sparse structure, D8 Operator fusion). Dimension-by-dimension scoring follows.
+
+| # | Dimension | Rating | Notes |
+|---|---|---|---|
+| D1 | Tensorization | [v] | SO(3)/SE(3) exp/log have **closed forms** (Rodrigues), 3x3/4x4 small matrices, batched as `[B,3,3]` / `[B,4,4]` tensors, sample-independent |
+| D2 | GEMM-mappability | [~] | 3x3/4x4 are too small to **saturate Tensor Cores**, bmm/batched small GEMM has low utilization and is easily memory-bandwidth-bound; fix: stack the batch dimension large, use specialized small kernels or einsum fusion |
+| D3 | Complexity | [v] | O(1) per element, linear in batch |
+| D4 | Memory | [v] | Elements are tiny; however, Jacobians/covariance matrices for high-DOF composite states can grow large (block-structured, still manageable) |
+| D5 | Low-precision stability | [x] **critical** | exp/log contain terms like `sin(theta)/theta`, `(1-cos(theta))/theta^2`, which cause catastrophic cancellation at **theta->0** (division by zero) and **theta->pi** (log singularity); fp16/bf16 are highly prone to NaN. **Must** switch to Taylor expansion for small angles -- branching introduces warp divergence (hurts D1, D8) |
+| D6 | Parallelism & communication | [~] | Per-sample exp/log is embarrassingly parallel [v]; however **discrete integration / motion chains on manifolds** (Sec. II-I, chained Exp products) are **serial recurrences**, long sequences must be rewritten as parallel scans to achieve parallelism |
+| D7 | Sparse structure | [v] | Composite manifold Jacobians are **block-structured sparse** (Eq. 89), GPU-friendly |
+| D8 | Operator fusion | [v] | Closed-form exp/log/Jacobians can be fused into a **single kernel**, avoiding materialization of intermediate small matrices |
+
+**Expanded notes (closed-form vs. series / batching / whether small-matrix exponentials can be tensorized)**:
+
+The GPU fate of `exp/log` in Lie theory depends entirely on "whether one is working with a low-dimensional group that admits a closed form."
+
+- **Closed-form vs. series (Conclusion 1)**: The exponential map for SO(3) is precisely the **Rodrigues formula** `exp([theta]_x) = I + (sin(theta)/theta)[theta]_x + ((1-cos(theta))/theta^2)[theta]_x^2` (Ex. 4); SE(3) is similar with an additional left-Jacobian factor V(theta) -- both are **finite-term closed forms**, pure tensor algebra, per-sample independent, no data-dependent loops. The opposite case is the **general matrix exponential**: for arbitrary Lie groups there is no Rodrigues-style closed form, only Taylor series + scaling-and-squaring **iterative** approximation, with data-dependent step counts, divergent control flow, and difficult tensorization -- a classic "beautiful but not computable" case. Closed forms exist only for the small set SO(3)/SE(3)/SE(2)/S^1/S^3; **do not treat the general matrix exp as a cheap O(1) operator**.
+- **Batching (Conclusion 2)**: stacking N rotations/poses into `[B,3,3]` / `[B,4,4]` for batched Exp/Log is clean element-wise + small bmm, which can be **fused with downstream operators into a single kernel**. Small-matrix exponentials **can** be tensorized -- batched stacking gives per-sample parallelism; but 3x3/4x4 are too small for bmm to saturate Tensor Cores, the value lies in "batchable and fusible" rather than "maxing out compute," and to improve utilization one should stack the batch large or use specialized small kernels.
+- **Low-precision singularities (Conclusion 3, top risk)**: theta->0 (`sin(theta)/theta` division by zero) and theta->pi (log is non-unique) cause catastrophic cancellation, fp16/bf16 produce NaN directly. Before low-precision training, one **must** implement Taylor-expansion fallbacks for small angles and run deterministic reproducibility tests; the fallback branches introduce warp divergence, requiring `where`/mask implementations rather than data-dependent jumps.
+- **Serial recurrence**: motion chains / discrete integration on manifolds (Sec. II-I) are serial recurrences of chained Exp products; long sequences must be rewritten as parallel scans to saturate the GPU.
+
+## Which Design Lens to Invoke
+
+- **symmetry (primary)**: Lie groups are by definition **continuous symmetry transformation groups**; this paper is the most direct mathematical ammunition for "symmetry/invariance -> equivariant networks."
+- **duality (primary)**: exp/log equivalently transforms "curved nonlinear manifold" to "flat linear tangent space" -- a paradigmatic example of transforming a hard problem into an easy one (the dragon-slaying sword used for "chopping vegetables").
+- **variational**: optimization on manifolds, retractions, error-state filtering (Sec. V-A).
+- **geometric**: the modeling closed loop from reality (robotic/camera state + noise) to mathematics (manifold + covariance) to interpretation.
+- **categorical**: abstracting from concrete matrices/quaternions to the unified "group" interface (the paper's own writing style -- generic formulation + grounded examples -- is itself an abstraction demonstration).
+
+## Anti-patterns
+
+- **Adding/subtracting rotations/poses directly in Euclidean space**: doing Euclidean MSE or linear interpolation on rotation matrices/quaternions -- breaks manifold constraints; use oplus/ominus and geodesic error instead.
+- **Mixing left and right perturbations**: inconsistent conventions for right/left Jacobians, local/global frames, causing misalignment of covariances and gradients (the paper repeatedly emphasizes defaulting to the right version).
+- **Assuming matrix exp is cheap and closed-form everywhere**: closed forms exist only for SO(3)/SE(3) and a few other groups; general groups require iterative series and cannot be treated as O(1) operators.
+- **Not handling singularities in low precision**: failing to implement Taylor fallbacks for theta->0/pi causes fp16 training to produce NaN immediately.
+- **Treating SE(3) as T(3)xSO(3)**: their tangent-space parameterizations differ (Ex. 7); whether translation and rotation are coupled changes the Jacobian, and misuse leads to systematic bias.
+- **Over-application**: forcing Lie theory onto tasks that do not need uncertainty propagation or geometric constraints, adding complexity and singularity risk for no gain -- mathematical beauty does not mean it should be used.
+
+## Deep Dive Entry
+
+> **Bibliographic info**: Joan Sola, Jeremie Deray, Dinesh Atchuthan, *A micro Lie theory for state estimation in robotics*, arXiv:1812.01537v9, 2021. Companion open-source C++ library [manif](https://github.com/artivis/manif).
+>
+> **Activation method**: Place `A micro Lie theory.pdf` in the `math_book/` folder at the project root; the Agent can then automatically search the original text. The PDF is not distributed via npm/git (copyright reasons); obtain it independently.
+
+> Full-fidelity lookup = have the Agent automatically retrieve the local PDF: `math_book/A micro Lie theory.pdf` (using `pdftotext` or Read PDF pages). The following are actual section/equation-block locations within that PDF:
+
+1. **Sec. II-D Exponential map + Fig. 1**: exp/log and capital Exp/Log operators (Eq. 23-24) -- the manifold-to-tangent-space bridge and closed-form source; with Ex. 3 (SO(3) Lie algebra `[omega]_x`), Ex. 4 (SO(3) exp = Rodrigues).
+2. **Sec. II-E Addition/subtraction operators (Eq. 25-28) + Sec. II-F Adjoint Ad_X (Eq. 30-35)**: "addition/subtraction" on manifolds and left/right perturbation conversions; Ex. 6 (SE(3) adjoint matrix).
+3. **Sec. II-G Derivatives on Lie groups + Sec. III-A Chain rule**: right Jacobian (Eq. 41a-c), left Jacobian (Eq. 44), related through the adjoint -- the geometrically correct form for backpropagation/gradients.
+4. **Sec. II-H Uncertainty on manifolds and covariance propagation (Eq. 52)**: Sigma defined on the tangent space; Sec. V-A manifold error-state EKF (Eq. 92-96) is the applied example.
+5. **Appendix formula handbook**: SO(3) right/left Jacobian closed forms (Eq. 143-145), SE(3) Jacobians (citing Barfoot), and Sec. IV composite manifold block-wise Jacobians (Eq. 84-90, Ex. 7) -- direct lookup tables for engineering implementation, cross-referenced with the manif library.

package/references/books/micro-lie-theory.md

@@ -1,4 +1,4 @@
-# 🌀 李理论速成 / A micro Lie theory
+# 李理论速成 / A micro Lie theory
 
 > **A micro Lie theory for state estimation in robotics** — Joan Solà, Jeremie Deray, Dinesh Atchuthan.
 > arXiv:1812.01537v9 [cs.RO]，2021-12-08，正文约 17 页 + 附录公式手册。配套开源 C++ 头文件库 **manif**（<https://github.com/artivis/manif>，实现 SO(2)/SO(3)/SE(2)/SE(3) 及解析雅可比）。
@@ -66,14 +66,14 @@
 
 | # | 维度 | 评级 | 说明 |
 |---|---|---|---|
-| ① | 张量化 | ✅ | SO(3)/SE(3) 的 exp/log 有**闭式**（Rodrigues），3×3/4×4 小矩阵，批量化为 `[B,3,3]` / `[B,4,4]` 张量逐样本独立 |
-| ② | GEMM 可映射 | ⚠️ | 3×3/4×4 太小，**吃不满 Tensor Core**，bmm/批量小 GEMM 利用率低、易受访存带宽限制；改造：把 batch 维堆大、用专用小核或 einsum 融合 |
-| ③ | 复杂度 | ✅ | 每元素 O(1)，随 batch 线性 |
-| ④ | 显存 | ✅ | 元素极小；但高自由度复合状态的雅可比/协方差矩阵会变大（块结构，仍可控） |
-| ⑤ | 低精度稳定 | ❌ **重点** | exp/log 含 `sinθ/θ`、`(1−cosθ)/θ²` 等项，在 **θ→0**（除零）与 **θ→π**（log 奇异）处灾难性抵消；fp16/bf16 极易 NaN。**必须**对小角做 Taylor 展开切换——而分支会带来 warp divergence（伤①⑧） |
-| ⑥ | 并行与通信 | ⚠️ | 逐样本 exp/log 是 embarrassingly parallel ✅；但**流形上的离散积分/运动链**（§II-I，连乘 Exp）是**串行递推**，长序列需写成并行扫描（parallel scan）才能并行 |
-| ⑦ | 稀疏结构 | ✅ | 复合流形的雅可比是**块结构化稀疏**（式 89），对 GPU 友好 |
-| ⑧ | 算子融合 | ✅ | 闭式 exp/log/雅可比可融进**单 kernel**，避免物化中间小矩阵 |
+| ① | 张量化 | [v] | SO(3)/SE(3) 的 exp/log 有**闭式**（Rodrigues），3×3/4×4 小矩阵，批量化为 `[B,3,3]` / `[B,4,4]` 张量逐样本独立 |
+| ② | GEMM 可映射 | [~] | 3×3/4×4 太小，**吃不满 Tensor Core**，bmm/批量小 GEMM 利用率低、易受访存带宽限制；改造：把 batch 维堆大、用专用小核或 einsum 融合 |
+| ③ | 复杂度 | [v] | 每元素 O(1)，随 batch 线性 |
+| ④ | 显存 | [v] | 元素极小；但高自由度复合状态的雅可比/协方差矩阵会变大（块结构，仍可控） |
+| ⑤ | 低精度稳定 | [x] **重点** | exp/log 含 `sinθ/θ`、`(1−cosθ)/θ²` 等项，在 **θ→0**（除零）与 **θ→π**（log 奇异）处灾难性抵消；fp16/bf16 极易 NaN。**必须**对小角做 Taylor 展开切换——而分支会带来 warp divergence（伤①⑧） |
+| ⑥ | 并行与通信 | [~] | 逐样本 exp/log 是 embarrassingly parallel [v]；但**流形上的离散积分/运动链**（§II-I，连乘 Exp）是**串行递推**，长序列需写成并行扫描（parallel scan）才能并行 |
+| ⑦ | 稀疏结构 | [v] | 复合流形的雅可比是**块结构化稀疏**（式 89），对 GPU 友好 |
+| ⑧ | 算子融合 | [v] | 闭式 exp/log/雅可比可融进**单 kernel**，避免物化中间小矩阵 |
 
 **展开说明（闭式 vs 级数 / 批量化 / 小矩阵指数能否张量化）**：
 
@@ -84,13 +84,13 @@
 - **低精度奇异点（结论 3，头号风险）**：θ→0（`sinθ/θ` 除零）与 θ→π（log 不唯一）会灾难性抵消，fp16/bf16 直接 NaN。低精度训练前**必须**对小角做 Taylor 兜底并做确定性复现测试；兜底分支又会引入 warp divergence，需用 `where`/掩码而非数据相关跳转实现。
 - **串行递推**：流形上的运动链/离散积分（§II-I）是连乘 Exp 的串行递推，长序列要改写成并行扫描（parallel scan）才能喂满 GPU。
 
-## 该调用哪个思想武器
+## 该调用哪个思想透镜
 
-- **⚛️ symmetry-invariance（首选）**：李群本就是**连续对称变换群**，本文是"对称/不变性→等变网络"最直接的数学弹药。
-- **🔄 transformation（首选）**：exp/log 把"曲的非线性流形"等价变换到"平的线性切空间"——把难问题变换成易问题的范式样板（屠龙刀"切菜"）。
-- **⚖️ optimization**：流形上的优化、retraction、误差状态滤波（§V-A）。
-- **🌉 modeling**：现实（机器人/相机状态+噪声）→数学（流形+协方差）→解释的建模闭环。
-- **🧩 abstraction**：从具体矩阵/四元数抽象出"群"的统一接口（论文正文泛型 + 例子接地的写法本身就是抽象示范）。
+- **symmetry（首选）**：李群本就是**连续对称变换群**，本文是"对称/不变性→等变网络"最直接的数学弹药。
+- **duality（首选）**：exp/log 把"曲的非线性流形"等价变换到"平的线性切空间"——把难问题变换成易问题的范式样板（屠龙刀"切菜"）。
+- **variational**：流形上的优化、retraction、误差状态滤波（§V-A）。
+- **geometric**：现实（机器人/相机状态+噪声）→数学（流形+协方差）→解释的建模闭环。
+- **categorical**：从具体矩阵/四元数抽象出"群"的统一接口（论文正文泛型 + 例子接地的写法本身就是抽象示范）。
 
 ## 反模式
 
@@ -103,13 +103,13 @@
 
 ## 深挖入口
 
-> **📖 书目信息**：Joan Solà, Jérémie Deray, Dinesh Atchuthan, *A micro Lie theory for state estimation in robotics*, arXiv:1812.01537v9, 2021. 配套开源 C++ 库 [manif](https://github.com/artivis/manif)。
+> **书目信息**：Joan Solà, Jérémie Deray, Dinesh Atchuthan, *A micro Lie theory for state estimation in robotics*, arXiv:1812.01537v9, 2021. 配套开源 C++ 库 [manif](https://github.com/artivis/manif)。
 >
 > **启用方式**：将 `A micro Lie theory.pdf` 放入项目根目录的 `math_book/` 文件夹，Agent 即可自动搜索原文。PDF 不随 npm/git 分发（版权原因），需自行获取。
 
 > 全保真回查 = 让 Agent 自动检索本地 PDF：`math_book/A micro Lie theory.pdf`（用 `pdftotext` 或 Read PDF pages）。下列为该 PDF 内的真实章节/公式块定位：
 
-1. **§II-D 指数映射 + Fig. 1**：exp/log 与大写 Exp/Log 算子（式 23–24）——流形↔切空间桥梁与闭式来源；配 Ex. 3（SO(3) 的李代数 `[ω]×`）、Ex. 4（SO(3) 的 exp = Rodrigues）。
+1. **§II-D 指数映射 + Fig. 1**：exp/log 与大写 Exp/Log 算子（式 23–24）——流形-> 切空间桥梁与闭式来源；配 Ex. 3（SO(3) 的李代数 `[ω]×`）、Ex. 4（SO(3) 的 exp = Rodrigues）。
 2. **§II-E 加减算子（式 25–28）+ §II-F 伴随 Ad_X（式 30–35）**：流形上的"加/减"与左右扰动互转；Ex. 6（SE(3) 的伴随矩阵）。
 3. **§II-G 李群上的导数 + §III-A 链式法则**：右雅可比（式 41a–c）、左雅可比（式 44），二者由伴随相联——反传/梯度的几何正确形式。
 4. **§II-H 流形上的不确定性与协方差传播（式 52）**：Σ 定义在切空间；§V-A 流形误差状态 EKF（式 92–96）是落地范例。

package/references/books/optimization-ml.en.md

@@ -0,0 +1,163 @@
+# Optimization (Including ML Applications)
+
+> **An Introduction to Optimization, With Applications to Machine Learning** (5th Edition, 2024)
+> Edwin K. P. Chong, Wu-Sheng Lu, Stanislaw H. Zak -- John Wiley & Sons (Hardback ISBN 9781119877639)
+>
+> This file is an "activation reference": synthesized distillation, engineering-oriented, not a verbatim excerpt.
+> Goal -- extract the structures from this classic optimization textbook that **can be applied to algorithms / GPU / training Infra**.
+
+## Overview
+
+The fifth edition of a classic continuous optimization textbook. Compared to earlier editions, the largest addition is a completely new **Part V "Optimization in Machine Learning"**, which fully integrates first-order/second-order optimality, duality, KKT, and convergence theory with training practices such as SGD/SVM/PCA. The book has 31 chapters organized into five parts.
+
+Actual chapter map:
+
+- **Part I -- Mathematical Review (Ch 1-5)**
+  - Ch 1 Proof methods and notation; Ch 2 Vector spaces and matrices (rank, inner products and norms).
+  - Ch 3 Transformations: eigenvalues/eigenvectors, orthogonal projections, quadratic forms S3.4, matrix norms S3.5.
+  - Ch 4 Geometry: hyperplanes, **convex sets S4.3**, polytopes and polyhedra.
+  - Ch 5 Calculus: derivative matrices S5.3, **level sets and gradients S5.5**, Taylor series S5.6.
+- **Part II -- Unconstrained Optimization (Ch 6-14)**
+  - Ch 6 First-order/second-order necessary and sufficient conditions for local minima.
+  - Ch 7 One-dimensional search: golden section, Fibonacci, bisection, Newton, secant, line search.
+  - **Ch 8 Gradient methods** (steepest descent S8.2, convergence analysis S8.3).
+  - **Ch 9 Newton's method** (Levenberg-Marquardt S9.3, nonlinear least squares).
+  - Ch 10 Conjugate direction/conjugate gradient methods; **Ch 11 Quasi-Newton methods** (inverse Hessian approximation, DFP S11.4, **BFGS S11.5**).
+  - Ch 12 Solving linear equations (least squares, RLS, Kaczmarz, minimum-norm solutions).
+  - **Ch 13 Neural networks and backpropagation** (single-neuron training S13.2, Backprop S13.3).
+  - Ch 14 Global search: Nelder-Mead, simulated annealing, PSO, genetic algorithms.
+- **Part III -- Linear Programming (Ch 15-19)**
+  - Ch 15 LP fundamentals and geometry; Ch 16 Simplex method.
+  - **Ch 17 LP duality** (dual LP, matrix games).
+  - Ch 18 Non-simplex/interior-point methods (Khachiyan, affine scaling, Karmarkar); Ch 19 Integer programming.
+- **Part IV -- Nonlinear Constrained Optimization (Ch 20-25)**
+  - Ch 20 Equality constraints: tangent/normal spaces S20.3, **Lagrange conditions S20.4**, second-order conditions.
+  - **Ch 21 Inequality constraints and KKT** (KKT S21.1, second-order conditions S21.2).
+  - **Ch 22 Convex optimization** (convex functions S22.2, convex problems S22.3, SDP/LMI S22.4).
+  - **Ch 23 Lagrangian duality** (weak/strong duality, duality gap S23.4.6, Slater S23.6.3, saddle points).
+  - Ch 24 Constrained optimization algorithms (projection S24.2, **projected gradient S24.3**, Armijo S24.4.4, augmented Lagrangian S24.5, penalty methods S24.6).
+  - Ch 25 Multi-objective / robust LP (Pareto, uncertainty).
+- **Part V -- Optimization in ML (Ch 26-31)**
+  - Ch 26 Feature engineering, PCA, SVD, linear autoencoders.
+  - **Ch 27 SGD algorithms** (SGD S27.1, variance reduction **SVRG S27.2**, **distributed SVRG and communication/computation tradeoffs S27.3**).
+  - Ch 28 Linear regression (regularization S28.3, cross-validation); Ch 29 Logistic regression / Softmax.
+  - **Ch 30 SVM** (hinge loss, hard/soft margin); **Ch 31 Kernel methods and K-Means**.
+
+## Core Structures Transferable to AI/Infra
+
+- **First-order optimality + gradient geometry (Ch 6, S5.5)**
+  - Gradient is perpendicular to level sets and is the steepest ascent direction -- the axiomatic root of all training optimizers.
+  - `nabla f = 0` is the stopping criterion; saddle points / non-convexity are the central pain points of modern deep learning.
+- **Convergence analysis framework (Ch 8 S8.3)**
+  - Uses the condition number kappa = lambda_max/lambda_min to characterize the convergence rate of steepest descent.
+  - Explains "why ill-conditioned networks are hard to train, why normalization / preconditioning is needed."
+- **Second-order curvature and damping (Ch 9)**
+  - Newton's method uses the Hessian to provide affine-invariant step sizes and directions.
+  - Levenberg-Marquardt's `(H+mu I)` damping = trust-region idea, the ancestor of Adam's `epsilon` and second-order optimizers.
+- **Quasi-Newton low-rank updates (Ch 11)**
+  - BFGS/DFP uses only gradient differences, maintaining an inverse Hessian approximation via rank-1/rank-2 updates.
+  - Direct intellectual ancestor of L-BFGS, K-FAC, and Shampoo.
+- **Matrix-free second-order methods (Ch 10)**
+  - Conjugate gradient requires only Hessian-vector products without storing the full Hessian; corresponds to autodiff HVP (Hessian-free).
+- **Constraint geometry (Ch 20-21, 24)**
+  - Tangent/normal spaces -> Lagrange/KKT -> projected gradient + penalty / augmented Lagrangian.
+  - Forms a complete toolkit for "training with constraints."
+- **Duality and saddle points (Ch 17, 23)**
+  - Strong duality <=> minimax = maximin; KKT, complementary slackness, Slater conditions.
+  - Min-max training (GAN / adversarial robustness) and SVM duality both fall here.
+- **Stochastic and distributed methods (Ch 27)**
+  - SGD's unbiased gradient estimation, SVRG variance reduction.
+  - **Explicit "communication vs. computation" tradeoff in distributed settings** -- the theoretical root of data-parallel training.
+- **Low-rank / spectral methods (Ch 26, S3.4)**
+  - SVD/PCA/linear autoencoders, quadratic forms and spectra -- the mathematical foundation of LoRA, KV compression, and spectral normalization.
+
+## Problem Types Suited for Activation
+
+- Selecting / designing optimizers, or explaining training dynamics (why divergence, oscillation, or stagnation at saddle points occurs).
+- Diagnosing ill-conditioning and slow convergence: quantifying "hard to train" via the condition number kappa.
+- Training with constraints: weight-norm balls, spectral norms, safety / budget constraints, requiring projection or penalty methods.
+- Min-max / adversarial / dual perspectives: replacing a hard primal with an easier dual (e.g., kernel Gram matrix).
+- Communication-computation tradeoffs, gradient compression, and variance reduction in distributed training.
+- Scenarios requiring second-order information but unable to compute the full Hessian (curvature-adaptive preconditioning).
+
+## Possible Algorithmic Inspirations
+
+- **Adaptive optimizers**
+  - Levenberg-Marquardt's `(H+mu I)` damping -> Adam's `epsilon`, trust regions, adaptive learning rates.
+  - Making "curvature / damping" explicit rather than staying with purely heuristic hyperparameter tuning.
+- **Feasible second-order for large models**
+  - L-BFGS: Store only the most recent m gradient pairs (low-rank history, memory O(md)).
+  - Hessian-free: Autodiff HVP + CG (matrix-free, no need to materialize the Hessian).
+  - K-FAC / Shampoo: Approximate the Hessian as **block-diagonal / Kronecker factors**, reducing each block to a small GEMM.
+- **Efficient solving via the dual perspective**
+  - SVM converts a high-dimensional primal into a dual that depends only on the kernel Gram matrix (Ch 30-31), a big win when sample count << dimensionality.
+  - Dual decomposition of constrained problems is naturally parallelizable.
+- **Constrained training**
+  - Projected gradient (S24.3) for norm-ball / spectral-norm constraints (weight clipping, spectral normalization).
+  - Augmented Lagrangian / penalty methods (S24.5-S24.6) soften hard constraints into differentiable regularization.
+- **Variance reduction / large batch**
+  - SVRG's (S27.2) control variate idea -> gradient accumulation, stability analysis of large-batch training.
+- **Distributed training**
+  - S27.3 "Communication vs. Computation" directly corresponds to all-reduce and computation overlap, gradient compression, federated / data security.
+
+## GPU Friendliness Warning
+
+> The following item-by-item evaluation uses the **eight-dimension scorecard** from `../gpu-friendly-math.md` (read that first).
+> Conclusion: The book's first-order / stochastic / low-rank content is naturally GPU-friendly; second-order methods and search-based methods require heavy adaptation or elimination.
+
+- **Full Hessian in second-order methods (Ch 9, 11)**
+  - Violates **Dimension 4 (Memory)**: Dense Hessian is O(d^2), impossible to materialize for hundreds of millions of parameters.
+  - Violates **Dimension 2 (GEMM-mappability)**: Inversion / factorization is not batched-GEMM friendly.
+  - Violates **Dimension 5 (Low-precision stability)**: Ill-conditioned inversion causes catastrophic cancellation under bf16/fp16.
+  - Adaptation: Matrix-free HVP + CG, L-BFGS low-rank, K-FAC block-diagonal -> restores **Dimension 2/4** friendliness.
+- **Line search (Ch 7 golden section / Armijo S24.4.4)**
+  - Violates **Dimension 1 (Tensorization)**: Essentially a scalar loop + data-dependent branching (step-by-step comparison of function values).
+  - Violates **Dimension 8 (Operator fusion)**: Frequent small kernels, control-flow divergence.
+  - Adaptation: Large models almost universally use scheduled step sizes (warmup + cosine) instead of step-by-step line search.
+- **First-order / SGD (Ch 8, 27)**
+  - **Dimension 1/2 friendly**: Gradients = tensor algebra, expressible as GEMM.
+  - **Dimension 6 (Parallelism and communication)** is good: Data-parallel all-reduce can overlap with backprop -- the fundamental reason this is the default approach for large models.
+- **Global search (Ch 14: GA/SA/PSO)**
+  - Violates **Dimension 1**: Non-differentiable, discrete stochastic search, blocks end-to-end gradient training.
+  - **Dimension 6** is also poor: Inter-population dependencies, hard to parallelize. Only suitable for black-box hyperparameter search, not for the training inner loop.
+- **LP simplex / interior-point methods (Ch 16, 18)**
+  - Simplex row operations are acceptable; interior-point methods require solving a linear system at each step, **Dimension 2/6** limited by factorization and communication.
+  - Suitable for medium-scale constrained subproblems, not for placement inside every training step.
+- **Projections (S24.2)**
+  - Norm-ball / box projections are cheap elementwise operations (**Dimension 1/8** friendly).
+  - General polyhedral projections require solving a QP (**Dimension 2** degrades).
+- **Convergence depends on condition number kappa**
+  - Large kappa -> **Dimension 5** is further amplified at low precision (ill-conditioned gradients).
+  - Must pair with normalization / preconditioning; otherwise bf16 training diverges.
+
+## Which Thinking Lens to Invoke
+
+- **variational** (primary): The complete objective-constraint-optimality-convergence framework; this book is its core source; translate real-world tasks (classification / regression / constraints) into solvable optimization problems.
+- **algorithmic**: Convergence properties, complexity, step size / stopping criteria for iterative algorithms.
+- **duality**: Duality, kernel methods, variable substitution, SVD/PCA -- "equivalence transformations to simplify problems."
+- **probabilistic**: SGD stochastic gradients, SVRG variance reduction, cross-validation / regularization.
+
+## Anti-patterns
+
+- Applying **full-Hessian Newton** directly to large models -- O(d^2) memory and inversion blow up immediately (must use matrix-free / low-rank / block-diagonal).
+- Using **line search** to tune step sizes at every step of a large model -- scalar serial, control-flow divergence, burns through compute; should switch to scheduled learning rates.
+- Using **GA/SA/PSO to train neural networks** -- non-differentiable, cannot parallelize, extremely sample-inefficient; limited to black-box hyperparameter search only.
+- Discussing convergence while **ignoring the condition number kappa** -- expecting low-precision convergence without normalization / preconditioning is wishful thinking.
+- Forcing **LP simplex thinking** onto continuously differentiable problems.
+- Treating **duality / KKT as universal** -- non-convex problems typically lack strong duality and have duality gaps (S23.4.6); KKT is only a necessary condition.
+- Treating this "activation reference" as a source of rigorous proofs -- details and theorem conditions must be verified against the original book.
+
+## Deep-dive Entry
+
+> **Bibliographic information**: Edwin K. P. Chong, Wu-Sheng Lu, Stanislaw H. Zak, *An Introduction to Optimization, With Applications to Machine Learning*, 5th Edition, John Wiley & Sons, 2024. ISBN 978-1-119-87763-9.
+>
+> **Activation method**: Place `An Introduction to Optimization With Applications to Machine Learning.pdf` in the `math_book/` folder at the project root; the Agent can then automatically search the original text. The PDF is not distributed via npm/git (copyright reasons) and must be obtained separately.
+
+Full-fidelity lookup = have the Agent directly search the local PDF
+`math_book/An Introduction to Optimization With Applications to Machine Learning.pdf`, locating by actual chapter numbers:
+
+- **Ch 8 Gradient Methods** (S8.3 convergence analysis, condition numbers) + **Ch 11 Quasi-Newton Methods** (S11.5 BFGS) -- Optimizers and feasible second-order methods.
+- **Ch 22 Convex Optimization Problems** (S22.2 convex functions, S22.4 SDP/LMI) -- Convexity criteria and semidefinite programming.
+- **Ch 23 Lagrangian Duality** (S23.5 strong duality, S23.6.3 Slater conditions) + **Ch 21 KKT Conditions** -- Duality / KKT / saddle points.
+- **Ch 24 Algorithms for Constrained Optimization** (S24.3 projected gradient, S24.5 augmented Lagrangian, S24.6 penalty methods) -- Constrained training algorithms.
+- **Ch 27 Stochastic Gradient Descent Algorithms** (S27.1 SGD, S27.2 SVRG, S27.3 distributed and communication/computation) -- Large-scale training core.

package/references/books/optimization-ml.md

@@ -1,4 +1,4 @@
-# ⚙️ 最优化（含 ML 应用）/ Optimization with ML
+# 最优化（含 ML 应用）/ Optimization with ML
 
 > **An Introduction to Optimization, With Applications to Machine Learning** (5th Edition, 2024)
 > Edwin K. P. Chong, Wu-Sheng Lu, Stanisław H. Żak — John Wiley & Sons（Hardback ISBN 9781119877639）
@@ -130,13 +130,12 @@
   - κ 大 → **维度 5** 在低精度下被进一步放大（病态梯度）。
   - 必须配归一化 / 预条件，否则 bf16 训练发散。
 
-## 该调用哪个思想武器
+## 该调用哪个思想透镜
 
-- **optimization**（主）：目标-约束-最优性-收敛的完整框架，本书是其核心出处。
-- **algorithmic-thinking**：迭代算法的收敛性、复杂度、步长 / 停机判据。
-- **transformation**：对偶、核技巧、变量替换、SVD/PCA——"等价转换简化问题"。
-- **modeling**：把现实任务（分类 / 回归 / 约束）翻译成可解的优化问题。
-- **probability-statistics**：SGD 随机梯度、SVRG 方差缩减、交叉验证 / 正则化。
+- **variational**（主）：目标-约束-最优性-收敛的完整框架，本书是其核心出处；把现实任务（分类 / 回归 / 约束）翻译成可解的优化问题。
+- **algorithmic**：迭代算法的收敛性、复杂度、步长 / 停机判据。
+- **duality**：对偶、核技巧、变量替换、SVD/PCA——"等价转换简化问题"。
+- **probabilistic**：SGD 随机梯度、SVRG 方差缩减、交叉验证 / 正则化。
 
 ## 反模式
 
@@ -150,7 +149,7 @@
 
 ## 深挖入口
 
-> **📖 书目信息**：Edwin K. P. Chong, Wu-Sheng Lu, Stanisław H. Żak, *An Introduction to Optimization, With Applications to Machine Learning*, 5th Edition, John Wiley & Sons, 2024. ISBN 978-1-119-87763-9.
+> **书目信息**：Edwin K. P. Chong, Wu-Sheng Lu, Stanisław H. Żak, *An Introduction to Optimization, With Applications to Machine Learning*, 5th Edition, John Wiley & Sons, 2024. ISBN 978-1-119-87763-9.
 >
 > **启用方式**：将 `An Introduction to Optimization With Applications to Machine Learning.pdf` 放入项目根目录的 `math_book/` 文件夹，Agent 即可自动搜索原文。PDF 不随 npm/git 分发（版权原因），需自行获取。