math-skill 3.0.0 → 3.1.0

package/references/books/micro-lie-theory.en.md

@@ -0,0 +1,116 @@
+# A Micro Lie Theory
+
+> **A micro Lie theory for state estimation in robotics** -- Joan Solà, Jeremie Deray, Dinesh Atchuthan.
+> arXiv:1812.01537v9 [cs.RO], 2021-12-08, main text approx. 17 pages + appendix formula handbook. Companion open-source C++ header library **manif** (<https://github.com/artivis/manif>, implementing SO(2)/SO(3)/SE(2)/SE(3) with analytic Jacobians).
+> This file is an "activation" distillation, **not a restatement of the original**; for full fidelity, consult the local PDF (see (Deep Dive Entry)).
+
+## Overview
+
+This paper **deliberately "amputates"** the vast edifice of Lie theory down to the minimal subset needed for robotic state estimation: it does not even introduce the Lie bracket, instead reducing the Lie algebra directly to the isomorphic vector space R^n, so that the "curved, nonlinear" Lie group becomes "flat, linear" vector algebra in the tangent space. The one-sentence thesis: **virtually every operation on the group (curved) has an exact counterpart in the Lie algebra (flat)** (Fig. 1) -- this is the modern use of Sophus Lie's "dragon-slaying sword" from inspiration.md: the dragon (solving differential equations) was never slain, yet the sword became a supreme tool for chopping vegetables (linearizing nonlinearities), carving (describing symmetries), and building houses (robotic state estimation).
+
+Actual structure map:
+- **II. A micro Lie theory (core)**: A Lie groups (group axioms + manifold) -> B group actions -> C tangent space and Lie algebra (hat `^wedge` / vee `^vee` operators) -> D **exponential map** exp/log and capital Exp/Log (Eq. 23-24) -> E **addition/subtraction operators** oplus/ominus (Eq. 25-28, left and right versions) -> F **adjoint** Ad_X and adjoint matrix (Eq. 30-35) -> G **derivatives on Lie groups**: right Jacobian (Eq. 41a-c) / left Jacobian (Eq. 44) -> H **uncertainty on manifolds and covariance propagation** (Eq. 52) -> I discrete integration on manifolds.
+- **III. Manifold differentiation rules**: A chain rule, B elementary Jacobian blocks, C derived Jacobian blocks.
+- **IV. Composite manifolds / bundles**: assembling heterogeneous large states into non-interacting manifold blocks, differentiating block by block (Eq. 84-90, Ex. 7 comparing SE(n) vs T(n)xSO(n) vs (R^n, SO(n))).
+- **V. Applications**: landmark-based localization and mapping -- error-state Kalman filter on manifolds (Eq. 92-96), graph-optimization SLAM, sensor self-calibration.
+- **VI. Conclusion + Appendix**: complete closed-form formulas for S^1/SO(2)/SE(2), S^3/SO(3), SE(3) (including right/left Jacobians, e.g. SO(3) Eq. 143-145).
+
+## Core Structures Transferable to AI/Infra
+
+The paper welds together the "curved nonlinear group" and the "flat linear vector space" using five tools, each of which can be directly ported into differentiable algorithms.
+
+### 1. exp/log Maps -> Manifold Parameterization (Eq. 23-24, Fig. 1)
+
+`Exp: R^n -> M`, `Log: M -> R^n` are the bijective bridges between the manifold and its tangent space (isomorphic to linear R^n). Transfer usage: **let the network freely predict in the unconstrained tangent space R^n, then map back to the valid manifold via Exp**. Rotations, poses, and unit quaternions thus naturally satisfy constraints (orthogonality, unit norm), eliminating the need for various ad-hoc post-hoc orthogonalization or normalization fixes on regressed matrices/quaternions.
+
+### 2. oplus/ominus Addition/Subtraction Operators -> "Vector Algebra" on Manifolds (Eq. 25-28)
+
+`oplus` (one Exp + one group composition) makes "adding an increment to a manifold state" look like ordinary vector addition; `ominus` computes the difference between two elements (geodesic residual). **Key pitfall**: because group composition is non-commutative, oplus/ominus come in **right versions** (increment in local frame / tangent at X) and **left versions** (increment in global frame / tangent at identity E); the paper defaults to the right version -- in engineering, one must stay consistent throughout; mixing conventions leads to errors.
+
+### 3. Right/Left Jacobians -> Geometrically Correct Backpropagation (Eq. 41a-c, 44)
+
+The linearization of a manifold function with respect to tangent-space perturbations is the Jacobian; the chain rule on manifolds takes the form `D(g o f)/DX = J_g . J_f` (Sec. III-A), isomorphic to the Euclidean chain rule, so it plugs directly into automatic differentiation. The **right Jacobian** (Eq. 41) and **left Jacobian** (Eq. 44) describe the same derivative in local vs. global tangent spaces, related to each other through the adjoint Ad.
+
+### 4. Adjoint Matrix Ad_X -> Transporting Tangent Vectors Across Frames (Eq. 30-35)
+
+`Ad_X` transports the same tangent vector between different points and left/right frames (Eq. 32: `X oplus tau = (Ad_X tau) oplus X`). It is the **algebraic realization of equivariance**, and also the transformation matrix for propagating perturbations/covariances across frames; Ex. 6 gives the closed-form adjoint matrix for SE(3).
+
+### 5. Covariance Propagation -> Uncertainty on Manifolds (Eq. 52)
+
+`Sigma = E[(X om Xbar)(X om Xbar)^T] in R^{m x m}` -- **the covariance lives in the tangent space R^n, not on the group**. Defining variance directly on the group is ill-posed; the tangent-space representation brings the uncertainty computations of Kalman filters / factor graphs back to familiar linear algebra.
+
+Two additional directly usable "components":
+- **SE(3) / SO(3)**: rigid body motion group / rotation group -- the unified differentiable representation of pose = rotation + translation, the standard language for cameras, point clouds, and robotic arm joints.
+- **Composite manifolds (bundles, Eq. 84-90)**: assembling multi-pose + feature + calibration heterogeneous large states into non-interacting manifold blocks, with Jacobians **assembled block by block** (Eq. 89 is block-structured sparse), i.e., "assemble the Jacobian matrix block by block" in engineering practice.
+
+## Problem Types Suited for Activation
+
+- Prediction/regression targets carry **geometric constraints**: rotations, poses, unit vectors, SPD matrices -- forcing them into Euclidean MLPs breaks the constraints.
+- Need **equivariance / invariance**: when the input undergoes rigid body transformations, the output should covary or remain invariant (point clouds, molecules, multi-view geometry).
+- Need **explicit uncertainty**: pose estimation, sensor fusion, SLAM, visual odometry covariance propagation.
+- **Iterative geometric optimization**: camera/point cloud registration, bundle adjustment, inverse kinematics, requiring gradient/Gauss-Newton on manifolds rather than in Euclidean space.
+- States **evolve on Lie groups**: inertial pre-integration, motion models, differentiable physics/control.
+
+## Possible Algorithmic Inspirations
+
+- **SE(3)/SO(3) equivariant networks**: using group elements as hidden states, updating between layers via Exp/opus, transporting features via Ad; equivariance comes from group actions rather than data augmentation.
+- **Manifold loss for pose regression**: using ominus (geodesic error) as the loss, rather than Euclidean MSE on rotation matrices/quaternions; naturally handles manifold topology (e.g., quaternion double cover).
+- **Lie group optimizers**: rewriting Adam/Gauss-Newton in the retraction form "compute gradient in tangent space -> map back via Exp" (error-state approach, corresponding to the manifold EKF in Sec. V-A).
+- **Differentiable registration / alignment**: camera pose, point cloud ICP, multi-view BA formulated as differentiable optimization layers on SE(3), trained end to end.
+- **Manifold-parameterized output heads**: the network outputs an R^n tangent vector, mapped back to SO(3)/SE(3) via Exp, replacing the various ad-hoc orthogonalization schemes for 6D/9D rotation representations.
+- **Uncertainty-aware fusion**: using the tangent-space covariance from Eq. 52 + Ad propagation to build learnable Kalman/factor-graph layers.
+
+## GPU Friendliness Warning
+
+> The sole authority for the acceptance gate: [`../gpu-friendly-math.en.md`](../gpu-friendly-math.en.md) (eight dimensions: D1 Tensorization, D2 GEMM-mappability, D3 Complexity, D4 Memory/KV, D5 Low-precision stability, D6 Parallelism & communication, D7 Sparse structure, D8 Operator fusion). Dimension-by-dimension scoring follows.
+
+| # | Dimension | Rating | Notes |
+|---|---|---|---|
+| D1 | Tensorization | [v] | SO(3)/SE(3) exp/log have **closed forms** (Rodrigues), 3x3/4x4 small matrices, batched as `[B,3,3]` / `[B,4,4]` tensors, sample-independent |
+| D2 | GEMM-mappability | [~] | 3x3/4x4 are too small to **saturate Tensor Cores**, bmm/batched small GEMM has low utilization and is easily memory-bandwidth-bound; fix: stack the batch dimension large, use specialized small kernels or einsum fusion |
+| D3 | Complexity | [v] | O(1) per element, linear in batch |
+| D4 | Memory | [v] | Elements are tiny; however, Jacobians/covariance matrices for high-DOF composite states can grow large (block-structured, still manageable) |
+| D5 | Low-precision stability | [x] **critical** | exp/log contain terms like `sin(theta)/theta`, `(1-cos(theta))/theta^2`, which cause catastrophic cancellation at **theta->0** (division by zero) and **theta->pi** (log singularity); fp16/bf16 are highly prone to NaN. **Must** switch to Taylor expansion for small angles -- branching introduces warp divergence (hurts D1, D8) |
+| D6 | Parallelism & communication | [~] | Per-sample exp/log is embarrassingly parallel [v]; however **discrete integration / motion chains on manifolds** (Sec. II-I, chained Exp products) are **serial recurrences**, long sequences must be rewritten as parallel scans to achieve parallelism |
+| D7 | Sparse structure | [v] | Composite manifold Jacobians are **block-structured sparse** (Eq. 89), GPU-friendly |
+| D8 | Operator fusion | [v] | Closed-form exp/log/Jacobians can be fused into a **single kernel**, avoiding materialization of intermediate small matrices |
+
+**Expanded notes (closed-form vs. series / batching / whether small-matrix exponentials can be tensorized)**:
+
+The GPU fate of `exp/log` in Lie theory depends entirely on "whether one is working with a low-dimensional group that admits a closed form."
+
+- **Closed-form vs. series (Conclusion 1)**: The exponential map for SO(3) is precisely the **Rodrigues formula** `exp([theta]_x) = I + (sin(theta)/theta)[theta]_x + ((1-cos(theta))/theta^2)[theta]_x^2` (Ex. 4); SE(3) is similar with an additional left-Jacobian factor V(theta) -- both are **finite-term closed forms**, pure tensor algebra, per-sample independent, no data-dependent loops. The opposite case is the **general matrix exponential**: for arbitrary Lie groups there is no Rodrigues-style closed form, only Taylor series + scaling-and-squaring **iterative** approximation, with data-dependent step counts, divergent control flow, and difficult tensorization -- a classic "beautiful but not computable" case. Closed forms exist only for the small set SO(3)/SE(3)/SE(2)/S^1/S^3; **do not treat the general matrix exp as a cheap O(1) operator**.
+- **Batching (Conclusion 2)**: stacking N rotations/poses into `[B,3,3]` / `[B,4,4]` for batched Exp/Log is clean element-wise + small bmm, which can be **fused with downstream operators into a single kernel**. Small-matrix exponentials **can** be tensorized -- batched stacking gives per-sample parallelism; but 3x3/4x4 are too small for bmm to saturate Tensor Cores, the value lies in "batchable and fusible" rather than "maxing out compute," and to improve utilization one should stack the batch large or use specialized small kernels.
+- **Low-precision singularities (Conclusion 3, top risk)**: theta->0 (`sin(theta)/theta` division by zero) and theta->pi (log is non-unique) cause catastrophic cancellation, fp16/bf16 produce NaN directly. Before low-precision training, one **must** implement Taylor-expansion fallbacks for small angles and run deterministic reproducibility tests; the fallback branches introduce warp divergence, requiring `where`/mask implementations rather than data-dependent jumps.
+- **Serial recurrence**: motion chains / discrete integration on manifolds (Sec. II-I) are serial recurrences of chained Exp products; long sequences must be rewritten as parallel scans to saturate the GPU.
+
+## Which Design Lens to Invoke
+
+- **symmetry (primary)**: Lie groups are by definition **continuous symmetry transformation groups**; this paper is the most direct mathematical ammunition for "symmetry/invariance -> equivariant networks."
+- **duality (primary)**: exp/log equivalently transforms "curved nonlinear manifold" to "flat linear tangent space" -- a paradigmatic example of transforming a hard problem into an easy one (the dragon-slaying sword used for "chopping vegetables").
+- **variational**: optimization on manifolds, retractions, error-state filtering (Sec. V-A).
+- **geometric**: the modeling closed loop from reality (robotic/camera state + noise) to mathematics (manifold + covariance) to interpretation.
+- **categorical**: abstracting from concrete matrices/quaternions to the unified "group" interface (the paper's own writing style -- generic formulation + grounded examples -- is itself an abstraction demonstration).
+
+## Anti-patterns
+
+- **Adding/subtracting rotations/poses directly in Euclidean space**: doing Euclidean MSE or linear interpolation on rotation matrices/quaternions -- breaks manifold constraints; use oplus/ominus and geodesic error instead.
+- **Mixing left and right perturbations**: inconsistent conventions for right/left Jacobians, local/global frames, causing misalignment of covariances and gradients (the paper repeatedly emphasizes defaulting to the right version).
+- **Assuming matrix exp is cheap and closed-form everywhere**: closed forms exist only for SO(3)/SE(3) and a few other groups; general groups require iterative series and cannot be treated as O(1) operators.
+- **Not handling singularities in low precision**: failing to implement Taylor fallbacks for theta->0/pi causes fp16 training to produce NaN immediately.
+- **Treating SE(3) as T(3)xSO(3)**: their tangent-space parameterizations differ (Ex. 7); whether translation and rotation are coupled changes the Jacobian, and misuse leads to systematic bias.
+- **Over-application**: forcing Lie theory onto tasks that do not need uncertainty propagation or geometric constraints, adding complexity and singularity risk for no gain -- mathematical beauty does not mean it should be used.
+
+## Deep Dive Entry
+
+> **Bibliographic info**: Joan Sola, Jeremie Deray, Dinesh Atchuthan, *A micro Lie theory for state estimation in robotics*, arXiv:1812.01537v9, 2021. Companion open-source C++ library [manif](https://github.com/artivis/manif).
+>
+> **Activation method**: Place `A micro Lie theory.pdf` in the `math_book/` folder at the project root; the Agent can then automatically search the original text. The PDF is not distributed via npm/git (copyright reasons); obtain it independently.
+
+> Full-fidelity lookup = have the Agent automatically retrieve the local PDF: `math_book/A micro Lie theory.pdf` (using `pdftotext` or Read PDF pages). The following are actual section/equation-block locations within that PDF:
+
+1. **Sec. II-D Exponential map + Fig. 1**: exp/log and capital Exp/Log operators (Eq. 23-24) -- the manifold-to-tangent-space bridge and closed-form source; with Ex. 3 (SO(3) Lie algebra `[omega]_x`), Ex. 4 (SO(3) exp = Rodrigues).
+2. **Sec. II-E Addition/subtraction operators (Eq. 25-28) + Sec. II-F Adjoint Ad_X (Eq. 30-35)**: "addition/subtraction" on manifolds and left/right perturbation conversions; Ex. 6 (SE(3) adjoint matrix).
+3. **Sec. II-G Derivatives on Lie groups + Sec. III-A Chain rule**: right Jacobian (Eq. 41a-c), left Jacobian (Eq. 44), related through the adjoint -- the geometrically correct form for backpropagation/gradients.
+4. **Sec. II-H Uncertainty on manifolds and covariance propagation (Eq. 52)**: Sigma defined on the tangent space; Sec. V-A manifold error-state EKF (Eq. 92-96) is the applied example.
+5. **Appendix formula handbook**: SO(3) right/left Jacobian closed forms (Eq. 143-145), SE(3) Jacobians (citing Barfoot), and Sec. IV composite manifold block-wise Jacobians (Eq. 84-90, Ex. 7) -- direct lookup tables for engineering implementation, cross-referenced with the manif library.

package/references/books/micro-lie-theory.md

@@ -1,4 +1,4 @@
-# 🌀 李理论速成 / A micro Lie theory
+# 李理论速成 / A micro Lie theory
 
 > **A micro Lie theory for state estimation in robotics** — Joan Solà, Jeremie Deray, Dinesh Atchuthan.
 > arXiv:1812.01537v9 [cs.RO]，2021-12-08，正文约 17 页 + 附录公式手册。配套开源 C++ 头文件库 **manif**（<https://github.com/artivis/manif>，实现 SO(2)/SO(3)/SE(2)/SE(3) 及解析雅可比）。
@@ -66,14 +66,14 @@
 
 | # | 维度 | 评级 | 说明 |
 |---|---|---|---|
-| ① | 张量化 | ✅ | SO(3)/SE(3) 的 exp/log 有**闭式**（Rodrigues），3×3/4×4 小矩阵，批量化为 `[B,3,3]` / `[B,4,4]` 张量逐样本独立 |
-| ② | GEMM 可映射 | ⚠️ | 3×3/4×4 太小，**吃不满 Tensor Core**，bmm/批量小 GEMM 利用率低、易受访存带宽限制；改造：把 batch 维堆大、用专用小核或 einsum 融合 |
-| ③ | 复杂度 | ✅ | 每元素 O(1)，随 batch 线性 |
-| ④ | 显存 | ✅ | 元素极小；但高自由度复合状态的雅可比/协方差矩阵会变大（块结构，仍可控） |
-| ⑤ | 低精度稳定 | ❌ **重点** | exp/log 含 `sinθ/θ`、`(1−cosθ)/θ²` 等项，在 **θ→0**（除零）与 **θ→π**（log 奇异）处灾难性抵消；fp16/bf16 极易 NaN。**必须**对小角做 Taylor 展开切换——而分支会带来 warp divergence（伤①⑧） |
-| ⑥ | 并行与通信 | ⚠️ | 逐样本 exp/log 是 embarrassingly parallel ✅；但**流形上的离散积分/运动链**（§II-I，连乘 Exp）是**串行递推**，长序列需写成并行扫描（parallel scan）才能并行 |
-| ⑦ | 稀疏结构 | ✅ | 复合流形的雅可比是**块结构化稀疏**（式 89），对 GPU 友好 |
-| ⑧ | 算子融合 | ✅ | 闭式 exp/log/雅可比可融进**单 kernel**，避免物化中间小矩阵 |
+| ① | 张量化 | [v] | SO(3)/SE(3) 的 exp/log 有**闭式**（Rodrigues），3×3/4×4 小矩阵，批量化为 `[B,3,3]` / `[B,4,4]` 张量逐样本独立 |
+| ② | GEMM 可映射 | [~] | 3×3/4×4 太小，**吃不满 Tensor Core**，bmm/批量小 GEMM 利用率低、易受访存带宽限制；改造：把 batch 维堆大、用专用小核或 einsum 融合 |
+| ③ | 复杂度 | [v] | 每元素 O(1)，随 batch 线性 |
+| ④ | 显存 | [v] | 元素极小；但高自由度复合状态的雅可比/协方差矩阵会变大（块结构，仍可控） |
+| ⑤ | 低精度稳定 | [x] **重点** | exp/log 含 `sinθ/θ`、`(1−cosθ)/θ²` 等项，在 **θ→0**（除零）与 **θ→π**（log 奇异）处灾难性抵消；fp16/bf16 极易 NaN。**必须**对小角做 Taylor 展开切换——而分支会带来 warp divergence（伤①⑧） |
+| ⑥ | 并行与通信 | [~] | 逐样本 exp/log 是 embarrassingly parallel [v]；但**流形上的离散积分/运动链**（§II-I，连乘 Exp）是**串行递推**，长序列需写成并行扫描（parallel scan）才能并行 |
+| ⑦ | 稀疏结构 | [v] | 复合流形的雅可比是**块结构化稀疏**（式 89），对 GPU 友好 |
+| ⑧ | 算子融合 | [v] | 闭式 exp/log/雅可比可融进**单 kernel**，避免物化中间小矩阵 |
 
 **展开说明（闭式 vs 级数 / 批量化 / 小矩阵指数能否张量化）**：
 
@@ -86,11 +86,11 @@
 
 ## 该调用哪个思想透镜
 
-- **⚛️ symmetry（首选）**：李群本就是**连续对称变换群**，本文是"对称/不变性→等变网络"最直接的数学弹药。
-- **🔄 duality（首选）**：exp/log 把"曲的非线性流形"等价变换到"平的线性切空间"——把难问题变换成易问题的范式样板（屠龙刀"切菜"）。
-- **⚖️ optimization**：流形上的优化、retraction、误差状态滤波（§V-A）。
-- **🌉 modeling**：现实（机器人/相机状态+噪声）→数学（流形+协方差）→解释的建模闭环。
-- **🧩 abstraction**：从具体矩阵/四元数抽象出"群"的统一接口（论文正文泛型 + 例子接地的写法本身就是抽象示范）。
+- **symmetry（首选）**：李群本就是**连续对称变换群**，本文是"对称/不变性→等变网络"最直接的数学弹药。
+- **duality（首选）**：exp/log 把"曲的非线性流形"等价变换到"平的线性切空间"——把难问题变换成易问题的范式样板（屠龙刀"切菜"）。
+- **variational**：流形上的优化、retraction、误差状态滤波（§V-A）。
+- **geometric**：现实（机器人/相机状态+噪声）→数学（流形+协方差）→解释的建模闭环。
+- **categorical**：从具体矩阵/四元数抽象出"群"的统一接口（论文正文泛型 + 例子接地的写法本身就是抽象示范）。
 
 ## 反模式
 
@@ -103,13 +103,13 @@
 
 ## 深挖入口
 
-> **📖 书目信息**：Joan Solà, Jérémie Deray, Dinesh Atchuthan, *A micro Lie theory for state estimation in robotics*, arXiv:1812.01537v9, 2021. 配套开源 C++ 库 [manif](https://github.com/artivis/manif)。
+> **书目信息**：Joan Solà, Jérémie Deray, Dinesh Atchuthan, *A micro Lie theory for state estimation in robotics*, arXiv:1812.01537v9, 2021. 配套开源 C++ 库 [manif](https://github.com/artivis/manif)。
 >
 > **启用方式**：将 `A micro Lie theory.pdf` 放入项目根目录的 `math_book/` 文件夹，Agent 即可自动搜索原文。PDF 不随 npm/git 分发（版权原因），需自行获取。
 
 > 全保真回查 = 让 Agent 自动检索本地 PDF：`math_book/A micro Lie theory.pdf`（用 `pdftotext` 或 Read PDF pages）。下列为该 PDF 内的真实章节/公式块定位：
 
-1. **§II-D 指数映射 + Fig. 1**：exp/log 与大写 Exp/Log 算子（式 23–24）——流形↔切空间桥梁与闭式来源；配 Ex. 3（SO(3) 的李代数 `[ω]×`）、Ex. 4（SO(3) 的 exp = Rodrigues）。
+1. **§II-D 指数映射 + Fig. 1**：exp/log 与大写 Exp/Log 算子（式 23–24）——流形-> 切空间桥梁与闭式来源；配 Ex. 3（SO(3) 的李代数 `[ω]×`）、Ex. 4（SO(3) 的 exp = Rodrigues）。
 2. **§II-E 加减算子（式 25–28）+ §II-F 伴随 Ad_X（式 30–35）**：流形上的"加/减"与左右扰动互转；Ex. 6（SE(3) 的伴随矩阵）。
 3. **§II-G 李群上的导数 + §III-A 链式法则**：右雅可比（式 41a–c）、左雅可比（式 44），二者由伴随相联——反传/梯度的几何正确形式。
 4. **§II-H 流形上的不确定性与协方差传播（式 52）**：Σ 定义在切空间；§V-A 流形误差状态 EKF（式 92–96）是落地范例。

package/references/books/optimization-ml.en.md

@@ -0,0 +1,163 @@
+# Optimization (Including ML Applications)
+
+> **An Introduction to Optimization, With Applications to Machine Learning** (5th Edition, 2024)
+> Edwin K. P. Chong, Wu-Sheng Lu, Stanislaw H. Zak -- John Wiley & Sons (Hardback ISBN 9781119877639)
+>
+> This file is an "activation reference": synthesized distillation, engineering-oriented, not a verbatim excerpt.
+> Goal -- extract the structures from this classic optimization textbook that **can be applied to algorithms / GPU / training Infra**.
+
+## Overview
+
+The fifth edition of a classic continuous optimization textbook. Compared to earlier editions, the largest addition is a completely new **Part V "Optimization in Machine Learning"**, which fully integrates first-order/second-order optimality, duality, KKT, and convergence theory with training practices such as SGD/SVM/PCA. The book has 31 chapters organized into five parts.
+
+Actual chapter map:
+
+- **Part I -- Mathematical Review (Ch 1-5)**
+  - Ch 1 Proof methods and notation; Ch 2 Vector spaces and matrices (rank, inner products and norms).
+  - Ch 3 Transformations: eigenvalues/eigenvectors, orthogonal projections, quadratic forms S3.4, matrix norms S3.5.
+  - Ch 4 Geometry: hyperplanes, **convex sets S4.3**, polytopes and polyhedra.
+  - Ch 5 Calculus: derivative matrices S5.3, **level sets and gradients S5.5**, Taylor series S5.6.
+- **Part II -- Unconstrained Optimization (Ch 6-14)**
+  - Ch 6 First-order/second-order necessary and sufficient conditions for local minima.
+  - Ch 7 One-dimensional search: golden section, Fibonacci, bisection, Newton, secant, line search.
+  - **Ch 8 Gradient methods** (steepest descent S8.2, convergence analysis S8.3).
+  - **Ch 9 Newton's method** (Levenberg-Marquardt S9.3, nonlinear least squares).
+  - Ch 10 Conjugate direction/conjugate gradient methods; **Ch 11 Quasi-Newton methods** (inverse Hessian approximation, DFP S11.4, **BFGS S11.5**).
+  - Ch 12 Solving linear equations (least squares, RLS, Kaczmarz, minimum-norm solutions).
+  - **Ch 13 Neural networks and backpropagation** (single-neuron training S13.2, Backprop S13.3).
+  - Ch 14 Global search: Nelder-Mead, simulated annealing, PSO, genetic algorithms.
+- **Part III -- Linear Programming (Ch 15-19)**
+  - Ch 15 LP fundamentals and geometry; Ch 16 Simplex method.
+  - **Ch 17 LP duality** (dual LP, matrix games).
+  - Ch 18 Non-simplex/interior-point methods (Khachiyan, affine scaling, Karmarkar); Ch 19 Integer programming.
+- **Part IV -- Nonlinear Constrained Optimization (Ch 20-25)**
+  - Ch 20 Equality constraints: tangent/normal spaces S20.3, **Lagrange conditions S20.4**, second-order conditions.
+  - **Ch 21 Inequality constraints and KKT** (KKT S21.1, second-order conditions S21.2).
+  - **Ch 22 Convex optimization** (convex functions S22.2, convex problems S22.3, SDP/LMI S22.4).
+  - **Ch 23 Lagrangian duality** (weak/strong duality, duality gap S23.4.6, Slater S23.6.3, saddle points).
+  - Ch 24 Constrained optimization algorithms (projection S24.2, **projected gradient S24.3**, Armijo S24.4.4, augmented Lagrangian S24.5, penalty methods S24.6).
+  - Ch 25 Multi-objective / robust LP (Pareto, uncertainty).
+- **Part V -- Optimization in ML (Ch 26-31)**
+  - Ch 26 Feature engineering, PCA, SVD, linear autoencoders.
+  - **Ch 27 SGD algorithms** (SGD S27.1, variance reduction **SVRG S27.2**, **distributed SVRG and communication/computation tradeoffs S27.3**).
+  - Ch 28 Linear regression (regularization S28.3, cross-validation); Ch 29 Logistic regression / Softmax.
+  - **Ch 30 SVM** (hinge loss, hard/soft margin); **Ch 31 Kernel methods and K-Means**.
+
+## Core Structures Transferable to AI/Infra
+
+- **First-order optimality + gradient geometry (Ch 6, S5.5)**
+  - Gradient is perpendicular to level sets and is the steepest ascent direction -- the axiomatic root of all training optimizers.
+  - `nabla f = 0` is the stopping criterion; saddle points / non-convexity are the central pain points of modern deep learning.
+- **Convergence analysis framework (Ch 8 S8.3)**
+  - Uses the condition number kappa = lambda_max/lambda_min to characterize the convergence rate of steepest descent.
+  - Explains "why ill-conditioned networks are hard to train, why normalization / preconditioning is needed."
+- **Second-order curvature and damping (Ch 9)**
+  - Newton's method uses the Hessian to provide affine-invariant step sizes and directions.
+  - Levenberg-Marquardt's `(H+mu I)` damping = trust-region idea, the ancestor of Adam's `epsilon` and second-order optimizers.
+- **Quasi-Newton low-rank updates (Ch 11)**
+  - BFGS/DFP uses only gradient differences, maintaining an inverse Hessian approximation via rank-1/rank-2 updates.
+  - Direct intellectual ancestor of L-BFGS, K-FAC, and Shampoo.
+- **Matrix-free second-order methods (Ch 10)**
+  - Conjugate gradient requires only Hessian-vector products without storing the full Hessian; corresponds to autodiff HVP (Hessian-free).
+- **Constraint geometry (Ch 20-21, 24)**
+  - Tangent/normal spaces -> Lagrange/KKT -> projected gradient + penalty / augmented Lagrangian.
+  - Forms a complete toolkit for "training with constraints."
+- **Duality and saddle points (Ch 17, 23)**
+  - Strong duality <=> minimax = maximin; KKT, complementary slackness, Slater conditions.
+  - Min-max training (GAN / adversarial robustness) and SVM duality both fall here.
+- **Stochastic and distributed methods (Ch 27)**
+  - SGD's unbiased gradient estimation, SVRG variance reduction.
+  - **Explicit "communication vs. computation" tradeoff in distributed settings** -- the theoretical root of data-parallel training.
+- **Low-rank / spectral methods (Ch 26, S3.4)**
+  - SVD/PCA/linear autoencoders, quadratic forms and spectra -- the mathematical foundation of LoRA, KV compression, and spectral normalization.
+
+## Problem Types Suited for Activation
+
+- Selecting / designing optimizers, or explaining training dynamics (why divergence, oscillation, or stagnation at saddle points occurs).
+- Diagnosing ill-conditioning and slow convergence: quantifying "hard to train" via the condition number kappa.
+- Training with constraints: weight-norm balls, spectral norms, safety / budget constraints, requiring projection or penalty methods.
+- Min-max / adversarial / dual perspectives: replacing a hard primal with an easier dual (e.g., kernel Gram matrix).
+- Communication-computation tradeoffs, gradient compression, and variance reduction in distributed training.
+- Scenarios requiring second-order information but unable to compute the full Hessian (curvature-adaptive preconditioning).
+
+## Possible Algorithmic Inspirations
+
+- **Adaptive optimizers**
+  - Levenberg-Marquardt's `(H+mu I)` damping -> Adam's `epsilon`, trust regions, adaptive learning rates.
+  - Making "curvature / damping" explicit rather than staying with purely heuristic hyperparameter tuning.
+- **Feasible second-order for large models**
+  - L-BFGS: Store only the most recent m gradient pairs (low-rank history, memory O(md)).
+  - Hessian-free: Autodiff HVP + CG (matrix-free, no need to materialize the Hessian).
+  - K-FAC / Shampoo: Approximate the Hessian as **block-diagonal / Kronecker factors**, reducing each block to a small GEMM.
+- **Efficient solving via the dual perspective**
+  - SVM converts a high-dimensional primal into a dual that depends only on the kernel Gram matrix (Ch 30-31), a big win when sample count << dimensionality.
+  - Dual decomposition of constrained problems is naturally parallelizable.
+- **Constrained training**
+  - Projected gradient (S24.3) for norm-ball / spectral-norm constraints (weight clipping, spectral normalization).
+  - Augmented Lagrangian / penalty methods (S24.5-S24.6) soften hard constraints into differentiable regularization.
+- **Variance reduction / large batch**
+  - SVRG's (S27.2) control variate idea -> gradient accumulation, stability analysis of large-batch training.
+- **Distributed training**
+  - S27.3 "Communication vs. Computation" directly corresponds to all-reduce and computation overlap, gradient compression, federated / data security.
+
+## GPU Friendliness Warning
+
+> The following item-by-item evaluation uses the **eight-dimension scorecard** from `../gpu-friendly-math.en.md` (read that first).
+> Conclusion: The book's first-order / stochastic / low-rank content is naturally GPU-friendly; second-order methods and search-based methods require heavy adaptation or elimination.
+
+- **Full Hessian in second-order methods (Ch 9, 11)**
+  - Violates **D4**: Dense Hessian is O(d^2), impossible to materialize for hundreds of millions of parameters.
+  - Violates **D2**: Inversion / factorization is not batched-GEMM friendly.
+  - Violates **D5**: Ill-conditioned inversion causes catastrophic cancellation under bf16/fp16.
+  - Adaptation: Matrix-free HVP + CG, L-BFGS low-rank, K-FAC block-diagonal -> restores **D2/D4** friendliness.
+- **Line search (Ch 7 golden section / Armijo S24.4.4)**
+  - Violates **D1**: Essentially a scalar loop + data-dependent branching (step-by-step comparison of function values).
+  - Violates **D8**: Frequent small kernels, control-flow divergence.
+  - Adaptation: Large models almost universally use scheduled step sizes (warmup + cosine) instead of step-by-step line search.
+- **First-order / SGD (Ch 8, 27)**
+  - **D1/D2 friendly**: Gradients = tensor algebra, expressible as GEMM.
+  - **D6** is good: Data-parallel all-reduce can overlap with backprop -- the fundamental reason this is the default approach for large models.
+- **Global search (Ch 14: GA/SA/PSO)**
+  - Violates **D1**: Non-differentiable, discrete stochastic search, blocks end-to-end gradient training.
+  - **D6** is also poor: Inter-population dependencies, hard to parallelize. Only suitable for black-box hyperparameter search, not for the training inner loop.
+- **LP simplex / interior-point methods (Ch 16, 18)**
+  - Simplex row operations are acceptable; interior-point methods require solving a linear system at each step, **D2/D6** limited by factorization and communication.
+  - Suitable for medium-scale constrained subproblems, not for placement inside every training step.
+- **Projections (S24.2)**
+  - Norm-ball / box projections are cheap elementwise operations (**D1/D8** friendly).
+  - General polyhedral projections require solving a QP (**D2** degrades).
+- **Convergence depends on condition number kappa**
+  - Large kappa -> **D5** is further amplified at low precision (ill-conditioned gradients).
+  - Must pair with normalization / preconditioning; otherwise bf16 training diverges.
+
+## Which Thinking Lens to Invoke
+
+- **variational** (primary): The complete objective-constraint-optimality-convergence framework; this book is its core source; translate real-world tasks (classification / regression / constraints) into solvable optimization problems.
+- **algorithmic**: Convergence properties, complexity, step size / stopping criteria for iterative algorithms.
+- **duality**: Duality, kernel methods, variable substitution, SVD/PCA -- "equivalence transformations to simplify problems."
+- **probabilistic**: SGD stochastic gradients, SVRG variance reduction, cross-validation / regularization.
+
+## Anti-patterns
+
+- Applying **full-Hessian Newton** directly to large models -- O(d^2) memory and inversion blow up immediately (must use matrix-free / low-rank / block-diagonal).
+- Using **line search** to tune step sizes at every step of a large model -- scalar serial, control-flow divergence, burns through compute; should switch to scheduled learning rates.
+- Using **GA/SA/PSO to train neural networks** -- non-differentiable, cannot parallelize, extremely sample-inefficient; limited to black-box hyperparameter search only.
+- Discussing convergence while **ignoring the condition number kappa** -- expecting low-precision convergence without normalization / preconditioning is wishful thinking.
+- Forcing **LP simplex thinking** onto continuously differentiable problems.
+- Treating **duality / KKT as universal** -- non-convex problems typically lack strong duality and have duality gaps (S23.4.6); KKT is only a necessary condition.
+- Treating this "activation reference" as a source of rigorous proofs -- details and theorem conditions must be verified against the original book.
+
+## Deep-dive Entry
+
+> **Bibliographic information**: Edwin K. P. Chong, Wu-Sheng Lu, Stanislaw H. Zak, *An Introduction to Optimization, With Applications to Machine Learning*, 5th Edition, John Wiley & Sons, 2024. ISBN 978-1-119-87763-9.
+>
+> **Activation method**: Place `An Introduction to Optimization With Applications to Machine Learning.pdf` in the `math_book/` folder at the project root; the Agent can then automatically search the original text. The PDF is not distributed via npm/git (copyright reasons) and must be obtained separately.
+
+Full-fidelity lookup = have the Agent directly search the local PDF
+`math_book/An Introduction to Optimization With Applications to Machine Learning.pdf`, locating by actual chapter numbers:
+
+- **Ch 8 Gradient Methods** (S8.3 convergence analysis, condition numbers) + **Ch 11 Quasi-Newton Methods** (S11.5 BFGS) -- Optimizers and feasible second-order methods.
+- **Ch 22 Convex Optimization Problems** (S22.2 convex functions, S22.4 SDP/LMI) -- Convexity criteria and semidefinite programming.
+- **Ch 23 Lagrangian Duality** (S23.5 strong duality, S23.6.3 Slater conditions) + **Ch 21 KKT Conditions** -- Duality / KKT / saddle points.
+- **Ch 24 Algorithms for Constrained Optimization** (S24.3 projected gradient, S24.5 augmented Lagrangian, S24.6 penalty methods) -- Constrained training algorithms.
+- **Ch 27 Stochastic Gradient Descent Algorithms** (S27.1 SGD, S27.2 SVRG, S27.3 distributed and communication/computation) -- Large-scale training core.

package/references/books/optimization-ml.md

@@ -1,4 +1,4 @@
-# ⚙️ 最优化（含 ML 应用）/ Optimization with ML
+# 最优化（含 ML 应用）/ Optimization with ML
 
 > **An Introduction to Optimization, With Applications to Machine Learning** (5th Edition, 2024)
 > Edwin K. P. Chong, Wu-Sheng Lu, Stanisław H. Żak — John Wiley & Sons（Hardback ISBN 9781119877639）
@@ -106,37 +106,36 @@
 > 结论：本书的一阶 / 随机 / 低秩部分天然友好；二阶法与搜索类方法需重度改造或淘汰。
 
 - **二阶法的全 Hessian（Ch 9, 11）**
-  - 违反**维度 4（显存）**：稠密 Hessian 是 O(d²)，亿级参数无法物化。
-  - 违反**维度 2（GEMM 可映射）**：求逆 / 分解非 batched-GEMM 友好。
-  - 违反**维度 5（低精度稳定）**：病态求逆在 bf16/fp16 灾难性抵消。
-  - 改造：matrix-free HVP + CG、L-BFGS 低秩、K-FAC 块对角 → 重回**维度 2/4** 友好。
+  - 违反**D4**：稠密 Hessian 是 O(d²)，亿级参数无法物化。
+  - 违反**D2**：求逆 / 分解非 batched-GEMM 友好。
+  - 违反**D5**：病态求逆在 bf16/fp16 灾难性抵消。
+  - 改造：matrix-free HVP + CG、L-BFGS 低秩、K-FAC 块对角 → 重回**D2/D4** 友好。
 - **线搜索（Ch 7 黄金分割 / Armijo §24.4.4）**
-  - 违反**维度 1（张量化）**：本质是标量循环 + 数据相关分支（逐步比较函数值）。
-  - 违反**维度 8（算子融合）**：频繁小 kernel、控制流发散。
+  - 违反**D1**：本质是标量循环 + 数据相关分支（逐步比较函数值）。
+  - 违反**D8**：频繁小 kernel、控制流发散。
   - 改造：大模型几乎都用调度式步长（warmup + cosine）替代逐步线搜索。
 - **一阶 / SGD（Ch 8, 27）**
-  - **维度 1/2 友好**：梯度 = 张量代数、可写成 GEMM。
-  - **维度 6（并行与通信）**良好：数据并行 all-reduce 可与反向 overlap——大模型默认路线的根本原因。
+  - **D1/D2 友好**：梯度 = 张量代数、可写成 GEMM。
+  - **D6**良好：数据并行 all-reduce 可与反向 overlap——大模型默认路线的根本原因。
 - **全局搜索（Ch 14：GA/SA/PSO）**
-  - 违反**维度 1**：不可微、离散随机搜索，阻断端到端梯度训练。
-  - **维度 6** 也差：种群间依赖、难并行映射。仅适合黑盒超参搜索，不进训练内环。
+  - 违反**D1**：不可微、离散随机搜索，阻断端到端梯度训练。
+  - **D6** 也差：种群间依赖、难并行映射。仅适合黑盒超参搜索，不进训练内环。
 - **LP 单纯形 / 内点法（Ch 16, 18）**
-  - 单纯形的行运算尚可；内点法每步要解线性系统，**维度 2/6** 受限于分解与通信。
+  - 单纯形的行运算尚可；内点法每步要解线性系统，**D2/D6** 受限于分解与通信。
   - 适合中等规模约束子问题，不适合放进每个训练 step。
 - **投影（§24.2）**
-  - norm-ball / box 投影是廉价 elementwise（**维度 1/8** 友好）。
-  - 一般多面体投影要解 QP（**维度 2** 退化）。
+  - norm-ball / box 投影是廉价 elementwise（**D1/D8** 友好）。
+  - 一般多面体投影要解 QP（**D2** 退化）。
 - **收敛依赖条件数 κ**
-  - κ 大 → **维度 5** 在低精度下被进一步放大（病态梯度）。
+  - κ 大 → **D5** 在低精度下被进一步放大（病态梯度）。
   - 必须配归一化 / 预条件，否则 bf16 训练发散。
 
 ## 该调用哪个思想透镜
 
-- **optimization**（主）：目标-约束-最优性-收敛的完整框架，本书是其核心出处。
-- **algorithmic-thinking**：迭代算法的收敛性、复杂度、步长 / 停机判据。
-- **transformation**：对偶、核技巧、变量替换、SVD/PCA——"等价转换简化问题"。
-- **modeling**：把现实任务（分类 / 回归 / 约束）翻译成可解的优化问题。
-- **probability-statistics**：SGD 随机梯度、SVRG 方差缩减、交叉验证 / 正则化。
+- **variational**（主）：目标-约束-最优性-收敛的完整框架，本书是其核心出处；把现实任务（分类 / 回归 / 约束）翻译成可解的优化问题。
+- **algorithmic**：迭代算法的收敛性、复杂度、步长 / 停机判据。
+- **duality**：对偶、核技巧、变量替换、SVD/PCA——"等价转换简化问题"。
+- **probabilistic**：SGD 随机梯度、SVRG 方差缩减、交叉验证 / 正则化。
 
 ## 反模式
 
@@ -150,7 +149,7 @@
 
 ## 深挖入口
 
-> **📖 书目信息**：Edwin K. P. Chong, Wu-Sheng Lu, Stanisław H. Żak, *An Introduction to Optimization, With Applications to Machine Learning*, 5th Edition, John Wiley & Sons, 2024. ISBN 978-1-119-87763-9.
+> **书目信息**：Edwin K. P. Chong, Wu-Sheng Lu, Stanisław H. Żak, *An Introduction to Optimization, With Applications to Machine Learning*, 5th Edition, John Wiley & Sons, 2024. ISBN 978-1-119-87763-9.
 >
 > **启用方式**：将 `An Introduction to Optimization With Applications to Machine Learning.pdf` 放入项目根目录的 `math_book/` 文件夹，Agent 即可自动搜索原文。PDF 不随 npm/git 分发（版权原因），需自行获取。
 

package/references/books/smooth-manifolds.en.md

@@ -0,0 +1,105 @@
+# Smooth Manifolds
+
+> **Book**: John M. Lee, *Introduction to Smooth Manifolds*, 2nd Edition. Graduate Texts in Mathematics 218, Springer, 2013. ISBN 978-1-4419-9981-8 / DOI 10.1007/978-1-4419-9982-5. MSC 53-01 / 58-01 / 57-01.
+> **Positioning**: Equipping objects that "locally look like Euclidean space but can bend globally" (manifolds) with calculus (tangent spaces, vector fields, differential forms, flows, Lie derivatives) -- the mathematical mother of **manifold optimization, latent-space geometry, and differentiable structures**.
+
+## Overview
+
+Smooth manifold = a space that can be locally linearized by coordinate charts, with charts glued together by smooth transition maps. The main thread of the book: **first transplant Euclidean calculus onto curved spaces, then study geometric and topological invariants on them**. For AI, the most valuable part is the "differentiable machinery" in the first half -- tangent/cotangent spaces, vector fields, flows, Riemannian metrics.
+
+Actual chapter map (2nd ed., chapter numbers match the book):
+
+- **Ch 1-2 Smooth Manifolds / Smooth Maps**: topological manifolds, smooth structures (atlases), smooth maps, partitions of unity. -> The language of local linearization + global patching.
+- **Ch 3 Tangent Vectors**: tangent space T_pM, differential / pushforward df_p, tangent bundle TM. -> **The core of local linearization**, the geometric prototype of backpropagation.
+- **Ch 4-5 Submersions, Immersions, Embeddings / Submanifolds**: constant rank theorem, embeddings, regular level sets -> submanifolds. -> Constraint sets = submanifolds.
+- **Ch 6 Sard's Theorem**: critical values have measure zero, Whitney embedding theorem (n-dimensional manifolds embed in R^{2n}). -> Embedding dimensions / manifold hypothesis.
+- **Ch 7 Lie Groups**: both group and manifold (SO(n), U(n), GL(n), Stiefel...), Lie algebra = tangent space at the identity. -> Orthogonal/unitary weight constraints, equivariance.
+- **Ch 8-9 Vector Fields / Integral Curves and Flows**: vector fields, integral curves, flows (one-parameter diffeomorphism groups), Lie derivatives and Lie brackets [X,Y]. -> **The mother structure of Neural ODE / diffusion / continuous normalizing flows**.
+- **Ch 10-12 Vector Bundles / Cotangent Bundle / Tensors**: bundles, covector fields (1-forms), pullbacks, tensors. -> The true nature of gradients is covectors.
+- **Ch 13 Riemannian Metrics**: inner product at each point, length/distance/volume, tangent-cotangent isomorphism (musical sharp/flat, raising/lowering indices). -> **The metric source for natural gradients / Riemannian optimization**.
+- **Ch 14-16 Differential Forms / Orientations / Integration**: k-forms, wedge products, exterior derivative d (d^2=0), orientations, volume forms, integration on manifolds and change of variables. -> The log-det-Jacobian in normalizing flows = pullback of volume forms.
+- **Ch 17-18 De Rham Cohomology / de Rham Theorem**: closed forms modulo exact forms = topological invariants read from differential data. -> Global obstructions / cohomological regularization.
+- **Ch 19-22 Distributions & Foliations / Exponential Map / Quotient Manifolds / Symplectic Manifolds**: integrable distributions (Frobenius), exponential map (retraction prototype), quotient manifolds (Grassmannian etc.), symplectic forms and Hamiltonian flows. -> Retractions, quotient-space constraints, symplectic integrators / HMC.
+
+**Author's stated boundary (preface)**: the book stops at "building the tools," **deliberately omitting** connections, geodesics, curvature, fiber bundles, and Hodge theory -- these are covered in Lee's sequel *Riemannian Manifolds*. So if a problem truly requires the deep geometry of curvature/parallel transport, this book only provides the entry point via metrics and the exponential map; one must continue with a Riemannian geometry text.
+
+## Core Structures Transferable to AI/Infra
+
+- **Tangent space = local linearization of parameter/latent space**. `df_p: T_pM -> T_{f(p)}N` is the Jacobian / pushforward (corresponding to JVP / forward-mode AD); backpropagation = pullback on the cotangent bundle (VJP = vector-Jacobian product = pullback of covectors), i.e., pullback along composed maps (the geometric version of the chain rule). All first-order methods live in the tangent space.
+- **Gradients are covectors, not vectors**. Autodiff produces 1-forms (elements of the cotangent space); to obtain a descent direction (tangent vector) one must use the **metric to raise indices** (sharp). Euclidean metric -> ordinary gradient; Fisher metric -> natural gradient. **This is the manifold-level root cause of natural gradients / mirror descent**.
+- **Constraint sets = submanifolds**. Regular level set theorem: when g is a submersion, the solution set `g(x)=c` is a smooth submanifold; constrained optimization = unconstrained optimization on a submanifold.
+- **Lie groups = differentiable symmetry groups**. SO(n)/U(n)/Stiefel/Grassmannian are all manifolds; their Lie algebras (e.g., skew-symmetric matrices so(n)) are linear spaces, mapped back to the group via `exp` -> **reparameterize "constrained weights" as "unconstrained Lie algebra + exp"**.
+- **Flows = time-parameterized families of diffeomorphisms**. Learning a vector field + integrating along it = Neural ODE / continuous normalizing flows / diffusion sampling. Flow invertibility and volume preservation directly correspond to model properties.
+- **Riemannian metric = designable/learnable "local geometry"**. It determines distances, angles, volumes, and orthogonality relations; changing the metric changes optimization trajectories and sampling measures.
+- **Differential forms + volume forms = the language of change of variables**. The `log|det J|` term in normalizing flows is precisely the pullback of a volume form under a map; choosing the right structure (triangular/coupling Jacobians) makes it cheap.
+
+## Problem Types Suited for Activation
+
+- Parameters should satisfy **geometric constraints**: orthogonality, unit norm, unit determinant, SPD, low-rank manifolds, hyperbolic/spherical latent spaces.
+- Optimization is more natural on **curved spaces**: subspace learning on Stiefel/Grassmannian, rotation/pose estimation, hyperspherical representations.
+- Need **structure-preserving dynamics**: invertible generative models, volume-preserving flows, Hamiltonian systems, energy-conserving long-horizon simulations.
+- **Latent space geometry**: interpolation, geodesics, metric learning, clustering/nearest-neighbor on manifolds.
+- Need to upgrade from correlations to **topological invariants**: detecting "holes" in latent spaces, global obstructions, cohomology-based consistency regularization.
+
+## Possible Algorithmic Inspirations
+
+- **Riemannian/manifold optimizers**: transplanting Adam/SGD onto Stiefel, Grassmannian, SPD, hyperbolic spaces -- projecting gradients to the tangent space + retraction back to the manifold.
+- **Orthogonal/Stiefel constrained weights**: using Cayley transforms or QR-retractions to maintain `W^T W = I`, mitigating gradient explosion/vanishing in RNNs/deep networks; or using so(n) Lie algebra + matrix-exp to reparameterize rotations.
+- **Geodesic interpolation**: using closed-form geodesics for interpolation and mixing in spherical/hyperbolic/SPD latent spaces, replacing Euclidean linear interpolation.
+- **Normalization on manifolds**: interpreting LayerNorm/feature normalization as projection onto spheres/unit manifolds; hyperspherical softmax and spectral normalization are instances of this.
+- **Neural ODE / continuous normalizing flows / diffusion**: learning a vector field X_theta, solving via flows; structured Jacobians make `log-det` cheap.
+- **Symplectic integrators / HMC**: using leapfrog-style symplectic, volume-preserving explicit updates for sampling and "optimization with momentum," stable over long horizons.
+- **Equivariant networks**: using Lie group actions + quotient manifolds to bake symmetries into the architecture (geometric deep learning).
+
+## GPU Friendliness Warning
+
+> The sole authority for the acceptance gate: the **eight dimensions** in `../gpu-friendly-math.en.md`. The success or failure of manifold methods almost entirely hinges on one point: **whether the retraction / exponential map can be tensorized and expressed as GEMM, or must be solved iteratively.**
+
+Dimension-by-dimension comparison:
+
+- **D1-D2 Tensorization / GEMM**: tangent-space operations (pushforward/pullback, Jacobian-vector products, projecting gradients to the tangent space) are **naturally batched GEMM** [v] -- backpropagation itself is pullback (VJP), which is extremely GPU-friendly. **However**, retractions/exp typically require QR, eigendecomposition, matrix exponentials, or small-matrix inverses: QR/eig are **not clean GEMM**, they are decompositions with serial dependencies (cuSOLVER batched small matrices are acceptable, but large matrices are O(n^3) with poor parallelism) -> **reformable** rather than natively friendly.
+- **D3 Complexity**: geodesic distances, parallel transport, general `log|det J|` are all O(n^3) or worse. **Fix**: restrict to manifolds with closed-form geodesics (spheres/hyperbolic/SO(3)); use triangular/coupling layers in normalizing flows so that log-det reduces to a diagonal sum (O(n)).
+- **D5 Low-precision**: [~] **biggest pitfall**. Matrix `exp / log / sqrt`, eigendecompositions, and the affine-invariant metric for SPD are **catastrophically unstable** under bf16/fp16, often silently requiring fp32/fp64. Manifold primitives frequently "appear to run but have numerically diverged long ago."
+- **D6 Parallelism & communication**: the squaring chain in scaling-and-squaring, ODE integration steps, Householder/QR all have **serial recurrences**, making cross-SM/device overlap difficult. Positive counter-example: explicit symplectic integrators (leapfrog) are highly parallel [v].
+- **D4/D7/D8 Memory / Sparsity / Fusion**: if Lie algebra/rotation parameterizations are restricted to **small matrices or block-diagonal** structures (e.g., per-head rotations, SO(3) Rodrigues closed form), they can be fused into kernels and use Tensor Cores; large dense manifold operators require materializing large intermediate tensors.
+
+**Conclusions and reform strategies (echoing the toolbox in gpu-friendly-math.md)**:
+
+1. **Prefer manifolds with closed-form retractions** (spheres, Stiefel-QR, SO(3), hyperbolic).
+2. **Soften whenever possible**: replace hard constraints with pure-GEMM regularization terms (e.g., `lambda ||W^T W - I||^2` instead of strict orthogonal manifold) -- for most training this suffices.
+3. **Small matrices / blocking**: restrict exp/Cayley/QR to small blocks or per-head, batched as batched GEMM.
+4. **Structured Jacobians**: insist on triangular/coupling structures in normalizing flows, avoiding general LU for computing determinants.
+5. **Precision guardrails**: for all matrix exp/log/eig, enforce fp32 accumulation and numerical stabilization (log-sum-exp style).
+
+## Which Design Lens to Invoke
+
+- **variational (primary)**: constrained optimization, Riemannian/manifold optimization, retraction selection.
+- **symmetry**: Lie groups, equivariance, quotient manifolds, invariants under group actions.
+- **duality**: coordinate chart transitions, pushforward/pullback, change of variables in normalizing flows, diffeomorphisms.
+- **topological**: de Rham cohomology, global obstructions, "holes" and connectivity of latent spaces.
+- **categorical**: extracting the "locally linear + smoothly patched" manifold skeleton from high-dimensional messy ambient data (the manifold hypothesis).
+
+## Anti-patterns
+
+- **Confusing ML "tensors" (arrays) with mathematical tensors (multilinear, with covariant/contravariant transformation laws)**, mistakenly assuming coordinate-independent invariance comes for free.
+- **Putting exp / geodesics / matrix-log into bf16 hot training loops**: both slow (serial decomposition) and silently divergent. Ask first: "is there a closed-form retraction / can it be softened?"
+- **Not softening when you should**: using strict manifold constraints for marginal gains while paying the throughput and stability cost of QR/eig; for many tasks, an orthogonality regularization term suffices.
+- **Confusing gradients (covectors) with descent directions (vectors)**: forgetting the metric and treating raw autodiff output directly as natural gradients.
+- **The illusion of a single global coordinate chart**: covering an entire manifold with one global parameterization inevitably introduces singularities (e.g., Euler angle gimbal lock); manifolds inherently require atlases / redundant parameterizations.
+- **Manifold hypothesis overuse**: deploying the full Riemannian machinery when the parameter space is already flat Euclidean -- pure over-engineering (violates simplicity-first).
+
+## Deep Dive Entry
+
+> **Bibliographic info**: John M. Lee, *Introduction to Smooth Manifolds*, 2nd Edition, Graduate Texts in Mathematics 218, Springer, 2013. ISBN 978-1-4419-9981-8.
+>
+> **Activation method**: Place `Introduction to Smooth Manifolds.pdf` in the `math_book/` folder at the project root; the Agent can then automatically search the original text. The PDF is not distributed via npm/git (copyright reasons); obtain it independently.
+
+> **Full-fidelity lookup**: when original definitions/theorems/proofs are needed, have the Agent **automatically search the local PDF** `math_book/Introduction to Smooth Manifolds.pdf` (locate by chapter number / keywords, do not rely on memory to restate). The following are actual chapter numbers (2nd ed.):
+
+- **Ch 3 Tangent Vectors** -- tangent spaces, differentials/pushforwards, tangent bundles: local linearization and the geometric prototype of backpropagation.
+- **Ch 11 The Cotangent Bundle** -- covector fields (1-forms), `df` as a covector, pullbacks: the true nature of gradients = covectors.
+- **Ch 13 Riemannian Metrics** -- metrics, tangent-cotangent isomorphism (sharp/flat), distances: the root of natural gradients / Riemannian optimization.
+- **Ch 9 Integral Curves and Flows** -- flows, integral curves, Lie derivatives/Lie brackets: Neural ODE / diffusion / structure-preserving dynamics.
+- **Ch 20 The Exponential Map** -- the exponential map: the retraction prototype, and also the main bottleneck for GPU feasibility.
+
+(Extensions: Ch 7 Lie Groups -> orthogonal/unitary constraints and equivariance; Ch 14 Differential Forms -> volume forms and log-det-Jacobian; Ch 22 Symplectic Manifolds -> symplectic integrators / HMC.)