math-skill 2.1.0 → 3.0.0

package/knowledge-base/lie-theory/equivariance.en.md

@@ -0,0 +1,54 @@
+# Equivariance
+
+## Minimal Definition
+
+A map $f: X \to Y$ is equivariant with respect to the action of a group $G$ if $f(g \cdot x) = g \cdot f(x)$ holds for all $g \in G, x \in X$. Equivariance is a finer structure-preserving property than invariance ($f(g \cdot x) = f(x)$): the output "co-moves" with the input under the same group action.
+
+## Core Formulas
+
+- Equivariance condition: $f(\rho_X(g) x) = \rho_Y(g) f(x), \quad \forall g \in G$
+- Invariance = equivariance to the trivial representation: $f(g \cdot x) = f(x)$ ($\rho_Y = \text{id}$)
+- Translation equivariance of convolution: $f(T_a x) = T_a f(x)$, where $T_a$ is the translation operator
+- Gauge equivariance: $f(\alpha \cdot_\omega x) = \alpha \cdot_{f(\omega)} f(x)$, where $\alpha$ is a local gauge transformation and $\omega$ is a connection
+- Adjoint equivariance: $f(X \oplus \tau) = f(X) \oplus (\text{Ad}_X \tau)$ (in the Lie group setting)
+
+## Applicable Problems
+
+- 3D point clouds/molecules: when the input is rotated, the output (segmentation/forces/pose) should rotate accordingly
+- Signal processing on spheres/manifolds: the choice of local coordinates should not affect prediction results
+- Multi-view/multi-sensor: when camera orientation changes, features should covary rather than require relearning
+- Physical simulation: vector quantities such as forces and velocities should rotate correctly under coordinate transformations
+
+## AI Design Translation
+
+- **E(n)-equivariant GNN**: Node features + coordinates; message passing simultaneously updates scalar features and equivariantly updates coordinates $x_i \to x_i + \sum_j \phi(r_{ij}) \cdot (x_i - x_j)$
+- **Gauge-equivariant CNN**: Each edge carries a $G$-connection to align local frames; convolution kernels are invariant to local coordinate choices; applicable to meshes/spheres/graphs
+- **Steerable CNN**: Feature fields are direct sums of group representations $\bigoplus_l \rho_l$; convolution kernels are constrained by Schur's lemma to block structure, yielding few parameters with strict equivariance
+- **Adjoint-equiv output head**: Pose regression $f: X \to SE(3)$ satisfying $f(g \cdot X) = g \cdot f(X)$, implemented using $\exp$ and the Lie algebra
+- **Equivariance verification loss**: $L_{\text{eq}} = \|f(g \cdot x) - g \cdot f(x)\|^2$ as an auxiliary regularizer, enforcing approximate equivariance
+
+## Engineering Feasibility
+
+GPU friendliness depends on the degree of group discretization:
+- **Discrete groups ($C_n$, octahedral group, etc.)**: Group convolution can be expanded into GEMM; equivariance constraints block-diagonalize weights (reducing parameters), GPU-friendly
+- **Translation group (standard CNN)**: Naturally equivariant; weight sharing is the engineering realization of equivariance, perfectly GPU-friendly
+- **Continuous groups SO(3)/SE(3)**: Require frequency-domain expansion (spherical harmonics) or sampling-based discretization; fast algorithms exist for spherical harmonic transforms but implementation is complex
+- **Gauge-equivariant**: One $G$-element action per edge equals small matrix-times-feature-vector, expressible as sparse matmul
+- **Approximate equivariance (regularization)**: The equivariance loss $L_{\text{eq}}$ is a standard MSE, fully GPU-friendly, but equivariance is not exact
+- Key trade-off: strict equivariance (structural constraints) vs. approximate equivariance (regularization) -- the former has fewer parameters but complex implementation, the latter is simple but not guaranteed
+
+## Risks and Failure Conditions
+
+- **Continuous group discretization error**: Improper sampling causes equivariance to silently break, passing verification but failing at inference
+- **Equivariance-expressiveness trade-off**: Strict equivariance constraints reduce the parameter space, potentially insufficient for fitting complex functions
+- **Combinatorial explosion of multiple group actions**: Simultaneously requiring rotation + translation + permutation equivariance leads to complex cross-constraints
+- **Data noise breaking equivariance**: Sensor noise makes the exact computation of $g \cdot x$ unreliable, invalidating the equivariance premise
+- **Numerical precision in equivariant layers**: Floating-point errors in spherical harmonics/CG coefficients at large $l$ break equivariance, requiring fp32 accumulation
+- **Over-constraining with equivariance**: When a task only requires approximate symmetry, enforcing strict equivariance is inferior to using soft regularization constraints
+
+## Further References
+
+- Distillation notes: references/books/micro-lie-theory.md (Section II-F Adjoint $\text{Ad}_X$, algebraic realization of equivariance)
+- Distillation notes: references/books/differential-geometry.md (Section 6.8 Principal Bundles, Section 12.12 G-Connections, gauge equivariance)
+- Distillation notes: references/books/differential-geometry.md (Ch 5 Lie Groups, continuous symmetry as prior)
+- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Section 6.8 + Section 12.12 (geometric foundations of gauge equivariance)

package/knowledge-base/lie-theory/equivariance.md

@@ -0,0 +1,54 @@
+# 等变性 (Equivariance)
+
+## 最小定义
+
+映射 $f: X \to Y$ 关于群 $G$ 的作用等变，若 $f(g \cdot x) = g \cdot f(x)$ 对所有 $g \in G, x \in X$ 成立。等变性是比不变性（$f(g \cdot x) = f(x)$）更精细的结构保持：输出随输入按同一群作用"协动"。
+
+## 核心公式
+
+- 等变性条件：$f(\rho_X(g) x) = \rho_Y(g) f(x), \quad \forall g \in G$
+- 不变性 = 等变到平凡表示：$f(g \cdot x) = f(x)$（$\rho_Y = \text{id}$）
+- 卷积的平移等变性：$f(T_a x) = T_a f(x)$，其中 $T_a$ 是平移算子
+- 规范等变性（gauge equivariance）：$f(\alpha \cdot_\omega x) = \alpha \cdot_{f(\omega)} f(x)$，$\alpha$ 是局部规范变换，$\omega$ 是联络
+- 伴随等变性：$f(X \oplus \tau) = f(X) \oplus (\text{Ad}_X \tau)$（李群场景）
+
+## 适用问题
+
+- 3D 点云/分子：输入旋转后，输出（分割/力/位姿）应同样旋转
+- 球面/流形上的信号处理：局部坐标选择不应影响预测结果
+- 多视角/多传感器：相机朝向变化时，特征应协变而非重新学习
+- 物理模拟：力、速度等矢量应随坐标系变换正确旋转
+
+## AI 设计翻译
+
+- **E(n)-等变 GNN**：节点特征 + 坐标，消息传递同时更新标量特征和等变更新坐标 $x_i \to x_i + \sum_j \phi(r_{ij}) \cdot (x_i - x_j)$
+- **Gauge-equivariant CNN**：每条边携带 $G$-联络对齐局部 frame，卷积核对局部坐标选择不变；适用于 mesh/球面/图
+- **Steerable CNN**：特征场是群表示的直和 $\bigoplus_l \rho_l$，卷积核被 Schur 引理约束为块结构，参数少但严格等变
+- **伴随等变输出头**：位姿回归 $f: X \to SE(3)$ 满足 $f(g \cdot X) = g \cdot f(X)$，用 $\exp$ 和李代数实现
+- **等变性验证损失**：$L_{\text{eq}} = \|f(g \cdot x) - g \cdot f(x)\|^2$ 作为辅助正则，强制近似等变
+
+## 工程可行性
+
+GPU 友好度取决于群的离散化程度：
+- **离散群（$C_n$, 八面体群等）**：群卷积可展开为 GEMM，等变约束使权重块对角化（参数减少），GPU 友好
+- **平移群（标准 CNN）**：天然等变，weight sharing 就是等变性的工程实现，完美 GPU 友好
+- **连续群 SO(3)/SE(3)**：需频域展开（球谐）或采样离散化；球谐变换有快速算法但实现复杂
+- **Gauge-equivariant**：每边一个 $G$-元素作用 = 小矩阵乘特征向量，可表达为 sparse matmul
+- **近似等变（正则化）**：等变性损失 $L_{\text{eq}}$ 是普通的 MSE，完全 GPU 友好，但等变性不严格
+- 关键权衡：严格等变（结构约束）vs 近似等变（正则化）——前者参数少但实现复杂，后者简单但不保证
+
+## 风险与失效条件
+
+- **连续群离散化误差**：采样不当导致等变性悄悄破缺，验证时通过但推理时失败
+- **等变性与表达力的权衡**：严格等变约束减少参数空间，可能不足以拟合复杂函数
+- **多群作用的组合爆炸**：同时要求旋转 + 平移 + 置换等变时，约束交叉复杂
+- **数据噪声破坏等变性**：传感器噪声使 $g \cdot x$ 的精确计算不可靠，等变性前提失效
+- **等变层的数值精度**：球谐/CG 系数在大 $l$ 下的浮点误差会破坏等变性，需 fp32 累加
+- **过度等变约束**：任务只需近似对称时硬上严格等变，不如用正则化软约束
+
+## 深入参考
+
+- 蒸馏稿：references/books/micro-lie-theory.md（§II-F 伴随 Ad_X，等变性的代数实现）
+- 蒸馏稿：references/books/differential-geometry.md（§6.8 Principal Bundles, §12.12 G-Connections, 规范等变）
+- 蒸馏稿：references/books/differential-geometry.md（Ch 5 Lie Groups, 连续对称作为先验）
+- 原书：Jeffrey M. Lee, *Manifolds and Differential Geometry*, §6.8 + §12.12（规范等变的几何基础）

package/knowledge-base/lie-theory/group-action.en.md

@@ -0,0 +1,51 @@
+# Group Action
+
+## Minimal Definition
+
+A group $G$ acting on a set $X$ is a homomorphism $\rho: G \to \text{Bij}(X)$ satisfying $\rho(e) = \text{id}$ and $\rho(g_1 g_2) = \rho(g_1) \circ \rho(g_2)$. It converts group elements into transformations on the set, providing the mathematical realization of "symmetry": the algebraic structure of the group determines the structure of geometric transformations.
+
+## Core Formulas
+
+- Group action: $g \cdot x = \rho(g)(x)$, satisfying $e \cdot x = x$ and $(gh)\cdot x = g \cdot (h \cdot x)$
+- Orbit: $\text{Orb}(x) = \{g \cdot x \mid g \in G\}$
+- Stabilizer subgroup: $\text{Stab}(x) = \{g \in G \mid g \cdot x = x\}$
+- Orbit-stabilizer theorem: $|G| = |\text{Orb}(x)| \cdot |\text{Stab}(x)|$
+- Invariant function: $f(g \cdot x) = f(x), \forall g \in G$
+- Equivariant map: $\phi(g \cdot x) = g \cdot \phi(x)$
+
+## Applicable Problems
+
+- Data possesses known symmetries: rotations, translations, permutations, scale transformations; the model must respect these symmetries
+- Output should covary with input: in pose estimation, when the object rotates, the output pose should rotate accordingly
+- Theoretical foundation for data augmentation: sampling along group orbits is equivalent to traversing the group action
+- Quotient space construction: modding out by the stabilizer subgroup yields an invariant feature space
+
+## AI Design Translation
+
+- **Equivariant network layer**: $f(g \cdot x) = g \cdot f(x)$, hard-coding the group action into the network structure to obtain equivariance without data augmentation
+- **Invariant pooling layer**: Average/max over orbits $\frac{1}{|G|}\sum_g f(g \cdot x)$ to extract invariants from equivariant features
+- **Group convolution**: $(f * h)(g) = \sum_{g'} f(g') h(g'^{-1} g)$, performing convolution directly on the group itself, applicable when signals are defined on the group
+- **Orbit-sampling data augmentation**: Use group actions to generate symmetry-equivalent training samples, effectively enlarging the training set
+
+## Engineering Feasibility
+
+GPU friendliness depends on the type of group:
+- **Finite/discrete groups**: Group convolution can be expanded into batched GEMM or sparse matmul, $O(|G|^2)$ or $O(|G| \cdot d)$, GPU-friendly
+- **Continuous compact groups SO(n)/SU(n)**: Require discrete sampling or frequency-domain expansion (Peter-Weyl theorem); fast algorithms exist for spherical harmonic transforms
+- **Permutation group $S_n$**: Order $n!$, cannot be enumerated; use sort pooling, symmetric functions, and other approximate invariants
+- **Fourier-accelerated group convolution**: The FFT on finite groups reduces convolution from $O(|G|^2)$ to $O(|G| \log |G|)$, but implementation is complex
+- Key bottleneck: if the discretization of a continuous group is not exact, equivariance silently breaks
+
+## Risks and Failure Conditions
+
+- **Naive discretization of continuous groups**: Improper sampling leads to broken equivariance and irregular gather/scatter patterns, GPU-unfriendly
+- **Incorrect group action definition**: Confusing left and right actions or inconsistent group multiplication order causes equivariance verification to pass but inference to fail
+- **Infeasible orbit enumeration**: Orbits of large/continuous groups cannot be fully enumerated; approximate invariants introduce bias
+- **Over-constraining**: Not all tasks require strict equivariance; enforcing group actions on weakly symmetric tasks may sacrifice expressiveness
+
+## Further References
+
+- Distillation notes: references/books/micro-lie-theory.md (Section II-B Group Actions)
+- Distillation notes: references/books/smooth-manifolds.md (Ch 7 Lie Groups)
+- Original text: Joan Sola et al., *A micro Lie theory*, Section II-B (group action definition and applications in robotics)
+- Original text: John M. Lee, *Introduction to Smooth Manifolds*, Ch 7 (Lie groups and group actions)

package/knowledge-base/lie-theory/group-action.md

@@ -0,0 +1,51 @@
+# 群作用 (Group Action)
+
+## 最小定义
+
+群 $G$ 作用在集合 $X$ 上是一个同态 $\rho: G \to \text{Bij}(X)$，满足 $\rho(e) = \text{id}$ 和 $\rho(g_1 g_2) = \rho(g_1) \circ \rho(g_2)$。它将群元素转化为集合上的变换，是"对称性"的数学实现：群的代数结构决定了几何变换的结构。
+
+## 核心公式
+
+- 群作用：$g \cdot x = \rho(g)(x)$，满足 $e \cdot x = x$，$(gh)\cdot x = g \cdot (h \cdot x)$
+- 轨道（orbit）：$\text{Orb}(x) = \{g \cdot x \mid g \in G\}$
+- 稳定子群（stabilizer）：$\text{Stab}(x) = \{g \in G \mid g \cdot x = x\}$
+- 轨道-稳定子定理：$|G| = |\text{Orb}(x)| \cdot |\text{Stab}(x)|$
+- 不变函数：$f(g \cdot x) = f(x), \forall g \in G$
+- 等变映射：$\phi(g \cdot x) = g \cdot \phi(x)$
+
+## 适用问题
+
+- 数据具有已知对称性：旋转、平移、置换、尺度变换，需要模型尊重这些对称
+- 输出应随输入协变：姿态估计中，物体旋转后输出位姿也应相应旋转
+- 数据增广的理论基础：群轨道上的采样等价于群作用的遍历
+- 商空间构造：模去稳定子群得到不变特征空间
+
+## AI 设计翻译
+
+- **等变网络层**：$f(g \cdot x) = g \cdot f(x)$，将群作用硬编码进网络结构，无需数据增广即获得等变性
+- **不变池化层**：对轨道求平均/最大 $\frac{1}{|G|}\sum_g f(g \cdot x)$，从等变特征中提取不变量
+- **群卷积**：$(f * h)(g) = \sum_{g'} f(g') h(g'^{-1} g)$，在群本身上做卷积，适用于信号定义在群上的场景
+- **轨道采样数据增广**：用群作用生成训练样本的对称等价类，扩大有效训练集
+
+## 工程可行性
+
+GPU 友好度取决于群的类型：
+- **有限群/离散群**：群卷积可展开为 batched GEMM 或稀疏 matmul，$O(|G|^2)$ 或 $O(|G| \cdot d)$，GPU 友好
+- **连续紧群 SO(n)/SU(n)**：需离散采样或频域展开（Peter-Wigner 定理），球谐变换有快速算法
+- **置换群 $S_n$**：$n!$ 阶，不可遍历；用排序池化、对称函数等近似不变化
+- **群卷积的 Fourier 加速**：有限群的 FFT 将卷积从 $O(|G|^2)$ 降至 $O(|G| \log |G|)$，但实现复杂
+- 关键瓶颈：连续群的离散化若不精确，等变性会悄悄破缺
+
+## 风险与失效条件
+
+- **连续群无脑离散化**：采样不当导致等变性破缺 + 不规则 gather/scatter，GPU 不友好
+- **群作用定义错误**：左右作用混淆、群乘法顺序不一致导致等变性验证通过但推理失败
+- **轨道遍历不可行**：大群/连续群的轨道无法完整遍历，近似不变化引入偏差
+- **过度约束**：不是所有任务都需要严格等变，弱对称任务硬上群作用可能牺牲表达力
+
+## 深入参考
+
+- 蒸馏稿：references/books/micro-lie-theory.md（§II-B 群作用）
+- 蒸馏稿：references/books/smooth-manifolds.md（Ch 7 Lie Groups）
+- 原书：Joan Sola et al., *A micro Lie theory*, §II-B（群作用定义与机器人中的应用）
+- 原书：John M. Lee, *Introduction to Smooth Manifolds*, Ch 7（李群与群作用）

package/knowledge-base/lie-theory/lie-algebra.en.md

@@ -0,0 +1,51 @@
+# Lie Algebra
+
+## Minimal Definition
+
+The Lie algebra $\mathfrak{g}$ is the tangent space $T_e G$ of a Lie group $G$ at the identity element $e$, equipped with a Lie bracket $[\cdot, \cdot]: \mathfrak{g} \times \mathfrak{g} \to \mathfrak{g}$ satisfying bilinearity, antisymmetry, and the Jacobi identity. It is the linear space of "infinitesimal generators" -- locally linearizing the curved, nonlinear group into a flat vector space.
+
+## Core Formulas
+
+- Lie bracket: $[X, Y] = XY - YX$ (for matrix groups), satisfying $[X,[Y,Z]] + [Y,[Z,X]] + [Z,[X,Y]] = 0$
+- $\text{so}(3)$: skew-symmetric matrices, $[\omega]_\times = \begin{pmatrix} 0 & -\omega_3 & \omega_2 \\ \omega_3 & 0 & -\omega_1 \\ -\omega_2 & \omega_1 & 0 \end{pmatrix}$
+- hat/vee operators: $\hat{\cdot}: \mathbb{R}^n \to \mathfrak{g}$ (vector to skew-symmetric matrix), $\check{\cdot}: \mathfrak{g} \to \mathbb{R}^n$ (inverse)
+- Baker-Campbell-Hausdorff formula: $\log(\exp(X)\exp(Y)) = X + Y + \frac{1}{2}[X,Y] + \cdots$
+- Adjoint representation: $\text{ad}_X(Y) = [X,Y]$, which is the differential of $\text{Ad}$
+
+## Applicable Problems
+
+- Local linearization of rotations/poses: approximate nonlinear group variations in a small neighborhood using Lie algebra vectors $\delta \in \mathbb{R}^n$
+- Reparameterization for constrained optimization: convert orthogonality/rotation constraints into unconstrained Lie algebra parameters + exp map
+- Infinitesimal description of symmetries: continuous symmetry groups are fully characterized by a small set of generators
+- Error-state estimation: covariances are defined on the tangent space $\mathbb{R}^n$ rather than on the group
+
+## AI Design Translation
+
+- **Lie algebra parameterization layer**: The network outputs $\delta \in \mathbb{R}^3$ (so(3)), then applies $\exp$ to obtain a valid rotation matrix; replaces quaternion normalization or 6D representations
+- **Error-state EKF/optimization layer**: Parameterize the error around a nominal state $X$ as $\delta \in \mathfrak{g}$ with $X_{\text{true}} = X \oplus \delta$; Kalman filtering proceeds in the linear tangent space
+- **Lie bracket regularization**: Penalize $[\xi_i, \xi_j] \neq 0$ to constrain the commutativity of generators, or use as a symmetry consistency loss
+- **Generator learning**: Learn a Lie algebra basis $\{E_1, \ldots, E_n\}$ as trainable parameters, enabling data-driven symmetry discovery
+
+## Engineering Feasibility
+
+High GPU friendliness. The core advantage of the Lie algebra is that it is a "linear space":
+- **hat/vee maps**: Pure index operations + sign flips, $O(n)$, perfectly GPU-friendly
+- **Lie bracket $[X,Y] = XY - YX$**: Two small matrix multiplications + subtraction, $O(n^3)$ for small matrices, batchable
+- **Linear combination $\sum c_i E_i$**: Vector addition + scalar multiplication, $O(nd)$, perfectly GPU-friendly
+- **BCH approximation**: The first few terms suffice for engineering precision; $[X,Y]$ computation is a small matrix multiplication
+- Main cost lies in $\exp$ rather than algebra operations: algebra operations are all linear; the bottleneck is the subsequent exp map
+
+## Risks and Failure Conditions
+
+- **BCH series truncation error**: Higher-order terms are non-negligible at large angles; the first-order approximation $X+Y$ is valid only for small perturbations
+- **Misinterpreting non-zero Lie brackets**: $[X,Y] \neq 0$ for non-commutative groups means group composition is order-sensitive; operations cannot be freely interchanged
+- **Basis selection affects optimization**: The choice of Lie algebra basis is not unique; poor conditioning leads to optimization difficulties
+- **Using the Lie algebra as global coordinates**: The exp map is only a local diffeomorphism; global coverage requires multiple charts (an atlas)
+- **Inconsistent left/right conventions**: Right Jacobian vs. left Jacobian, local vs. global frame -- inconsistency leads to misaligned gradients and covariances
+
+## Further References
+
+- Distillation notes: references/books/micro-lie-theory.md (Section II-C Tangent Space and Lie Algebra, hat/vee operators)
+- Distillation notes: references/books/differential-geometry.md (Ch 5 Lie Groups, Lie algebra section)
+- Original text: Joan Sola et al., *A micro Lie theory*, Section II-C (Lie algebra definition and hat/vee), Section II-D (exp/log bridge)
+- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 5 (Lie groups and Lie algebras)

package/knowledge-base/lie-theory/lie-algebra.md

@@ -0,0 +1,51 @@
+# 李代数 (Lie Algebra)
+
+## 最小定义
+
+李代数 $\mathfrak{g}$ 是李群 $G$ 在单位元 $e$ 处的切空间 $T_e G$，配备李括号 $[\cdot, \cdot]: \mathfrak{g} \times \mathfrak{g} \to \mathfrak{g}$ 满足双线性、反对称性和 Jacobi 恒等式。它是"无穷小生成元"的线性空间——把曲的非线性群局部线性化为平的向量空间。
+
+## 核心公式
+
+- 李括号：$[X, Y] = XY - YX$（矩阵群），满足 $[X,[Y,Z]] + [Y,[Z,X]] + [Z,[X,Y]] = 0$
+- $\text{so}(3)$：反对称矩阵，$[\omega]_\times = \begin{pmatrix} 0 & -\omega_3 & \omega_2 \\ \omega_3 & 0 & -\omega_1 \\ -\omega_2 & \omega_1 & 0 \end{pmatrix}$
+- hat/vee 算子：$\hat{\cdot}: \mathbb{R}^n \to \mathfrak{g}$（向量→反对称矩阵），$\check{\cdot}: \mathfrak{g} \to \mathbb{R}^n$（逆）
+- Baker-Campbell-Hausdorff：$\log(\exp(X)\exp(Y)) = X + Y + \frac{1}{2}[X,Y] + \cdots$
+- 伴随表示：$\text{ad}_X(Y) = [X,Y]$，是 $\text{Ad}$ 的微分
+
+## 适用问题
+
+- 旋转/位姿的局部线性化：在小邻域内用李代数向量 $\delta \in \mathbb{R}^n$ 近似群上的非线性变化
+- 约束优化的重参数化：将正交/旋转约束转化为无约束李代数参数 + exp 映射
+- 对称性的无穷小描述：连续对称群由少数生成元完全刻画
+- 误差状态估计：协方差定义在切空间 $\mathbb{R}^n$ 上而非群上
+
+## AI 设计翻译
+
+- **李代数参数化层**：网络输出 $\delta \in \mathbb{R}^3$（so(3)），经 $\exp$ 得到合法旋转矩阵；替代四元数归一化或 6D 表示
+- **误差状态 EKF/优化层**：在标称状态 $X$ 附近用 $\delta \in \mathfrak{g}$ 参数化误差 $X_{\text{true}} = X \oplus \delta$，卡尔曼滤波在线性切空间进行
+- **李括号正则**：惩罚 $[\xi_i, \xi_j] \neq 0$ 来约束生成元的交换性，或用作对称性一致性损失
+- **生成元学习**：学习一组李代数基 $\{E_1, \ldots, E_n\}$ 作为可训练参数，实现数据驱动的对称性发现
+
+## 工程可行性
+
+GPU 友好度高。李代数的核心优势是"线性空间"：
+- **hat/vee 映射**：纯索引操作 + 符号翻转，$O(n)$，完美 GPU 友好
+- **李括号 $[X,Y] = XY - YX$**：两次小矩阵乘法 + 减法，$O(n^3)$ 小矩阵，可 batched
+- **线性组合 $\sum c_i E_i$**：向量加法 + 标量乘，$O(nd)$，完美 GPU 友好
+- **BCH 近似**：取前几项即够工程精度，$[X,Y]$ 计算为小矩阵乘法
+- 主要开销在 $\exp$ 而非代数运算本身：代数运算全部是线性操作，瓶颈在后续的 exp 映射
+
+## 风险与失效条件
+
+- **BCH 级数截断误差**：高阶项在大角度时不可忽略，一阶近似 $X+Y$ 仅对小扰动有效
+- **李括号非零的误读**：非交换群的 $[X,Y] \neq 0$ 意味着群合成顺序敏感，不能随意交换操作
+- **基的选择影响优化**：李代数基的选取不唯一，差的条件数会导致优化困难
+- **把李代数当全局坐标**：exp 映射只是局部微分同胚，大范围需要多张覆盖（atlas）
+- **左右约定不统一**：右雅可比 vs 左雅可比、局部 vs 全局坐标系，不统一则梯度与协方差错位
+
+## 深入参考
+
+- 蒸馏稿：references/books/micro-lie-theory.md（§II-C 切空间与李代数, hat/vee 算子）
+- 蒸馏稿：references/books/differential-geometry.md（Ch 5 Lie Groups, 李代数部分）
+- 原书：Joan Sola et al., *A micro Lie theory*, §II-C（李代数定义与 hat/vee）、§II-D（exp/log 桥梁）
+- 原书：Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 5（李群与李代数）

package/knowledge-base/lie-theory/lie-group.en.md

@@ -0,0 +1,51 @@
+# Lie Group
+
+## Minimal Definition
+
+A Lie group $G$ is a space endowed with both a smooth manifold structure and a group structure, such that group multiplication $\mu: G \times G \to G$ and inversion $\iota: G \to G$ are smooth maps. It is the precise mathematical object of "continuous symmetry transformation groups" -- simultaneously a differentiable space and a multiplicative algebra.
+
+## Core Formulas
+
+- Exponential map: $\exp: \mathfrak{g} \to G$, $\exp(\xi) = \gamma_\xi(1)$, where $\gamma_\xi$ is the one-parameter subgroup
+- SO(3) Rodrigues formula: $\exp([\theta]_\times) = I + \frac{\sin\|\theta\|}{\|\theta\|}[\theta]_\times + \frac{1-\cos\|\theta\|}{\|\theta\|^2}[\theta]_\times^2$
+- SE(3): $\exp\begin{pmatrix} [\omega]_\times & v \\ 0 & 0 \end{pmatrix} = \begin{pmatrix} \exp([\omega]_\times) & V(\theta)v \\ 0 & 1 \end{pmatrix}$
+- $\oplus/\ominus$ operators (right version): $X \oplus \tau = X \cdot \exp(\tau)$, $X \ominus Y = \log(Y^{-1} X)$
+- Adjoint map: $\text{Ad}_X(\tau) = X \tau X^{-1}$ (for matrix groups), satisfying $X \oplus \tau = (\text{Ad}_X \tau) \oplus X$
+
+## Applicable Problems
+
+- Predictions/regression targets carry geometric constraints: rotations, poses, unit quaternions -- forcing them into a Euclidean MLP violates constraints
+- Equivariance/invariance is required: when the input undergoes a rigid-body transformation, the output should covary or remain invariant (point clouds, molecules, multi-view geometry)
+- State evolves on a Lie group: inertial preintegration, motion models, differentiable physics/control
+- Orthogonal/unitary constraints on weight matrices: optimization on the Stiefel manifold
+
+## AI Design Translation
+
+- **Manifold-parameterized output head**: The network makes unconstrained predictions in the tangent space $\mathbb{R}^n$, then projects back to SO(3)/SE(3) via $\exp$, replacing the ad-hoc orthogonalization of 6D/9D rotation representations
+- **Manifold loss function**: Use $\ominus$ (geodesic error) as the loss, $L = \|X_{\text{pred}} \ominus X_{\text{gt}}\|^2$, naturally handling manifold topology
+- **Lie group RNN/ODE**: Hidden state $h_t \in G$, updated as $h_{t+1} = h_t \oplus f_\theta(h_t, x_t) = h_t \cdot \exp(f_\theta(h_t, x_t))$
+- **Orthogonal weight constraints**: $W = \exp(A)$ where $A$ is skew-symmetric, guaranteeing $W^T W = I$; alternatively, the Cayley transform $W = (I-A)(I+A)^{-1}$
+
+## Engineering Feasibility
+
+GPU friendliness: the key factor is whether exp/log has a closed form.
+- **SO(3)/SE(3) closed forms**: The Rodrigues formula is a finite algebraic expression on 3x3/4x4 small matrices, batchable as $[B,3,3]$/ $[B,4,4]$ tensors, per-sample independent, GPU-friendly
+- **General matrix exponential**: Taylor series + scaling-and-squaring iteration, with data-dependent step counts and divergent control flow, difficult to tensorize, GPU-unfriendly
+- **Small-matrix GEMM**: 3x3/4x4 matrices are too small to saturate Tensor Cores; value lies in "batchable and fusible" rather than "maximizing compute throughput"
+- **Low-precision critical issue**: $\sin\theta/\theta$ produces division by zero as $\theta \to 0$; $\log$ is singular as $\theta \to \pi$; fp16/bf16 yields NaN directly -- **must** implement Taylor expansion switching for small angles
+- **Kinematic chain seriality**: Discrete integration via successive $\exp$ multiplication is serial; rewriting as a parallel scan is required for parallelism
+
+## Risks and Failure Conditions
+
+- **Low-precision singularities**: Without Taylor expansion fallbacks at $\theta \to 0/\pi$, fp16 training produces NaN directly; fallback branches introduce warp divergence
+- **Mixing left and right perturbations**: Mixing $\oplus_R$ (right version / local frame) with $\oplus_L$ (left version / global frame) causes misalignment of covariances and gradients
+- **Treating SE(3) as T(3)xSO(3)**: Their tangent-space parameterizations differ; whether translation and rotation are coupled affects the Jacobian, and misuse introduces systematic bias
+- **Treating general matrix exp as an $O(1)$ operator**: Closed forms exist only for a few groups such as SO(3)/SE(3)/SE(2)/S1/S3; general groups require iterative series
+- **Over-application**: Imposing Lie group parameterization on tasks that do not need geometric constraints merely adds complexity and singularity risk
+
+## Further References
+
+- Distillation notes: references/books/micro-lie-theory.md (Section II-A Lie Group Definition, Section II-D Exponential Map, Section II-E Plus/Minus Operators)
+- Distillation notes: references/books/differential-geometry.md (Ch 5 Lie Groups)
+- Distillation notes: references/books/smooth-manifolds.md (Ch 7 Lie Groups)
+- Original text: Joan Sola et al., *A micro Lie theory*, Section II-A through Section II-F (complete Lie group toolkit)

package/knowledge-base/lie-theory/lie-group.md

@@ -0,0 +1,51 @@
+# 李群 (Lie Group)
+
+## 最小定义
+
+李群 $G$ 是同时具有光滑流形结构和群结构的空间，使得群乘法 $\mu: G \times G \to G$ 和求逆 $\iota: G \to G$ 都是光滑映射。它是"连续对称变换群"的精确数学对象——既是可微的空间，又是可乘的代数。
+
+## 核心公式
+
+- 指数映射：$\exp: \mathfrak{g} \to G$，$\exp(\xi) = \gamma_\xi(1)$，其中 $\gamma_\xi$ 是单参数子群
+- SO(3) Rodrigues 公式：$\exp([\theta]_\times) = I + \frac{\sin\|\theta\|}{\|\theta\|}[\theta]_\times + \frac{1-\cos\|\theta\|}{\|\theta\|^2}[\theta]_\times^2$
+- SE(3)：$\exp\begin{pmatrix} [\omega]_\times & v \\ 0 & 0 \end{pmatrix} = \begin{pmatrix} \exp([\omega]_\times) & V(\theta)v \\ 0 & 1 \end{pmatrix}$
+- $\oplus/\ominus$ 算子（右版本）：$X \oplus \tau = X \cdot \exp(\tau)$，$X \ominus Y = \log(Y^{-1} X)$
+- 伴随映射：$\text{Ad}_X(\tau) = X \tau X^{-1}$（矩阵群），满足 $X \oplus \tau = (\text{Ad}_X \tau) \oplus X$
+
+## 适用问题
+
+- 预测/回归量带几何约束：旋转、位姿、单位四元数——硬塞进欧氏 MLP 会破坏约束
+- 需要等变性/不变性：输入做刚体变换，输出应协变或不变（点云、分子、多视几何）
+- 状态在李群上演化：惯性预积分、运动模型、可微物理/控制
+- 正交/酉约束的权重矩阵：Stiefel 流形上的优化
+
+## AI 设计翻译
+
+- **流形参数化输出头**：网络在切空间 $\mathbb{R}^n$ 自由预测，经 $\exp$ 投回 SO(3)/SE(3)，替代 6D/9D 旋转表示的 ad-hoc 正交化
+- **流形损失函数**：用 $\ominus$（测地误差）当 loss，$L = \|X_{\text{pred}} \ominus X_{\text{gt}}\|^2$，天然处理流形拓扑
+- **李群 RNN/ODE**：隐状态 $h_t \in G$，更新 $h_{t+1} = h_t \oplus f_\theta(h_t, x_t) = h_t \cdot \exp(f_\theta(h_t, x_t))$
+- **正交权重约束**：$W = \exp(A)$ 其中 $A$ 反对称，保证 $W^T W = I$；或用 Cayley 变换 $W = (I-A)(I+A)^{-1}$
+
+## 工程可行性
+
+GPU 友好度：关键看 exp/log 是否有闭式。
+- **SO(3)/SE(3) 闭式**：Rodrigues 公式是有限项代数表达式，3x3/4x4 小矩阵，可 batched 为 $[B,3,3]$/ $[B,4,4]$ 张量，逐样本独立，GPU 友好
+- **通用矩阵指数**：Taylor 级数 + scaling-and-squaring 迭代，步数数据相关、控制流发散、难张量化，GPU 不友好
+- **小矩阵 GEMM**：3x3/4x4 太小，吃不满 Tensor Core，价值在"可批量、可融合"而非"打满算力"
+- **低精度致命点**：$\sin\theta/\theta$ 在 $\theta \to 0$ 除零，$\theta \to \pi$ 时 log 奇异；fp16/bf16 直接 NaN，**必须** 对小角做 Taylor 展开切换
+- **运动链串行**：连乘 $\exp$ 的离散积分是串行递推，需改写成 parallel scan 才能并行
+
+## 风险与失效条件
+
+- **低精度奇异点**：$\theta \to 0/\pi$ 处不做 Taylor 兜底，fp16 训练直接 NaN；兜底分支又引入 warp divergence
+- **左右扰动混用**：$\oplus_R$（右版本/局部坐标系）和 $\oplus_L$（左版本/全局坐标系）混用导致协方差与梯度错位
+- **把 SE(3) 当成 T(3)xSO(3)**：二者切空间参数化不同，平移-旋转是否耦合改变雅可比，误用导致系统性偏差
+- **通用 matrix exp 当 O(1) 算子**：闭式只属于 SO(3)/SE(3)/SE(2)/S1/S3 等少数群，一般群需迭代级数
+- **过度套用**：不需要几何约束的任务硬上李群参数化，徒增复杂度与奇异点风险
+
+## 深入参考
+
+- 蒸馏稿：references/books/micro-lie-theory.md（§II-A 李群定义, §II-D 指数映射, §II-E 加减算子）
+- 蒸馏稿：references/books/differential-geometry.md（Ch 5 Lie Groups）
+- 蒸馏稿：references/books/smooth-manifolds.md（Ch 7 Lie Groups）
+- 原书：Joan Sola et al., *A micro Lie theory*, §II-A ~ §II-F（完整李群工具链）

package/knowledge-base/lie-theory/representation.en.md

@@ -0,0 +1,52 @@
+# Representation
+
+## Minimal Definition
+
+A representation of a group $G$ on a vector space $V$ is a group homomorphism $\rho: G \to GL(V)$, realizing abstract group elements as computable linear transformations (matrices). The core problem of representation theory is decomposing a complex group action into a direct sum of irreducible representations (irreps), analogous to factoring integers into primes.
+
+## Core Formulas
+
+- Representation: $\rho(g_1 g_2) = \rho(g_1)\rho(g_2)$, $\rho(e) = I$
+- Character: $\chi_\rho(g) = \text{tr}(\rho(g))$, a class function encoding the essential information of the representation
+- Peter-Weyl theorem (compact groups): $L^2(G) \cong \bigoplus_{\pi \in \hat{G}} \dim(\pi) \cdot \pi$
+- Schur's lemma: $\rho_1, \rho_2$ irreducible and inequivalent $\Rightarrow$ $\text{Hom}_G(\rho_1, \rho_2) = 0$
+- Irreducible representations of SO(3): dimension $2l+1$, $l = 0,1,2,\ldots$, with spherical harmonics $Y_l^m$ as basis functions
+- Clebsch-Gordan decomposition: $\rho_1 \otimes \rho_2 \cong \bigoplus_k m_k \rho_k$
+
+## Applicable Problems
+
+- Signals defined on groups/spheres: require spherical harmonic expansion for frequency-domain analysis
+- Constructing equivariant feature spaces: each feature channel corresponds to an irreducible representation
+- Accelerating group convolutions: using the Fourier transform (irreducible representations) to convert convolution into pointwise multiplication in the frequency domain
+- Molecular/crystal symmetries: representations of point groups/space groups determine orbital symmetries and selection rules
+
+## AI Design Translation
+
+- **Spherical harmonic feature layer**: Expand 3D point cloud/molecular features into a spherical harmonic basis $Y_l^m$; each $(l,m)$ channel transforms according to an SO(3) irreducible representation, achieving strict rotational equivariance
+- **Group Fourier transform layer**: Transform a finite group signal $f: G \to \mathbb{R}$ to the frequency domain via $\hat{f}(\pi) = \sum_g f(g)\pi(g)$, converting convolution into per-irrep matrix multiplication
+- **Schur-constrained weight matrices**: Equivariant inter-layer maps $W: V_1 \to V_2$ must commute with the group action $W\rho_1(g) = \rho_2(g)W$; Schur's lemma forces $W$ to be block-diagonal/scalar, drastically reducing parameters
+- **Character pooling**: Extract invariants using $\chi_\rho(g) = \text{tr}(\rho(g))$ as input features for classification/regression
+
+## Engineering Feasibility
+
+GPU friendliness depends on the group size and representation dimension:
+- **Finite group representations**: Each irrep is a small matrix ($d \times d$); the group Fourier transform equals a batch of small GEMMs, $O(|G| \cdot d^2)$, batchable
+- **SO(3) spherical harmonic transform**: Fast algorithms exist ($O(L^2 \log L)$), but implementation is complex; the real-space to frequency-domain transform can be expressed as sparse matmul
+- **Clebsch-Gordan coefficients**: Once precomputed, they are fixed sparse tensors; contraction with features can be expressed as GEMM
+- **Sparsity from Schur constraints**: Weight matrices of equivariant layers are constrained to be block-diagonal/scalar, greatly reducing parameters but requiring sparse/block GEMM
+- Key bottleneck: the Clebsch-Gordan tensor size grows rapidly for high-dimensional irreps (large $l$ spherical harmonics)
+
+## Risks and Failure Conditions
+
+- **High-frequency spherical harmonic numerical instability**: $Y_l^m$ for large $l$ oscillates violently near the poles, causing severe precision loss under fp16
+- **Irrep completeness truncation**: Keeping only up to $l_{\max}$ spherical harmonics loses high-frequency information; truncation error must be determined experimentally
+- **Infinite-dimensional representations of non-compact groups**: Irreducible representations of groups such as the Lorentz group are infinite-dimensional and must be truncated in practice
+- **Freedom in representation selection**: Which irreps participate and to what order are hyperparameters with no automatic selection method
+- **Clebsch-Gordan tensor storage**: The number of CG coefficients grows as $O(l^3)$ with increasing $l$, increasing precomputation and storage costs
+
+## Further References
+
+- Distillation notes: references/books/micro-lie-theory.md (Section II-F Adjoint $\text{Ad}_X$ and Adjoint Matrix)
+- Distillation notes: references/books/differential-geometry.md (Ch 5 Lie Groups, adjoint representation section)
+- Original text: Joan Sola et al., *A micro Lie theory*, Section II-F (adjoint representation, equations 30--35)
+- Original text: Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 5 (representations of Lie groups)

package/knowledge-base/lie-theory/representation.md

@@ -0,0 +1,52 @@
+# 表示 (Representation)
+
+## 最小定义
+
+群 $G$ 在向量空间 $V$ 上的表示是群同态 $\rho: G \to GL(V)$，将抽象群元素实现为可计算的线性变换（矩阵）。表示论的核心问题是：把复杂的群作用分解为不可约表示（irrep）的直和，如同整数分解为素数。
+
+## 核心公式
+
+- 表示：$\rho(g_1 g_2) = \rho(g_1)\rho(g_2)$，$\rho(e) = I$
+- 特征标：$\chi_\rho(g) = \text{tr}(\rho(g))$，是类函数，编码表示的核心信息
+- Peter-Weyl 定理（紧群）：$L^2(G) \cong \bigoplus_{\pi \in \hat{G}} \dim(\pi) \cdot \pi$
+- Schur 引理：$\rho_1, \rho_2$ 不可约且不等价 $\Rightarrow$ $\text{Hom}_G(\rho_1, \rho_2) = 0$
+- SO(3) 的不可约表示：维数 $2l+1$，$l = 0,1,2,\ldots$，基函数为球谐函数 $Y_l^m$
+- Clebsch-Gordan 分解：$\rho_1 \otimes \rho_2 \cong \bigoplus_k m_k \rho_k$
+
+## 适用问题
+
+- 信号定义在群/球面上：需要用球谐展开做频域分析
+- 构建等变特征空间：每个特征通道对应一个不可约表示
+- 群卷积的加速：利用 Fourier 变换（不可约表示）将卷积变为频域逐点乘法
+- 分子/晶体对称性：点群/空间群的表示决定轨道对称性和选择定则
+
+## AI 设计翻译
+
+- **球谐特征层**：将 3D 点云/分子特征展开为球谐基 $Y_l^m$，每个 $(l,m)$ 通道按 SO(3) 不可约表示变换，实现严格旋转等变
+- **群 Fourier 变换层**：有限群信号 $f: G \to \mathbb{R}$ 经 $\hat{f}(\pi) = \sum_g f(g)\pi(g)$ 变换到频域，卷积变为逐 irrep 矩阵乘
+- **Schur 约束的权重矩阵**：等变层间映射 $W: V_1 \to V_2$ 必须与群作用交换 $W\rho_1(g) = \rho_2(g)W$，Schur 引理强制 $W$ 为块对角/标量，大幅减少参数
+- **特征标池化**：用 $\chi_\rho(g) = \text{tr}(\rho(g))$ 提取不变量作为分类/回归的输入特征
+
+## 工程可行性
+
+GPU 友好度取决于群的规模和表示维数：
+- **有限群表示**：每个 irrep 是小矩阵（$d \times d$），群 Fourier 变换 = 一批小 GEMM，$O(|G| \cdot d^2)$，可 batched
+- **SO(3) 球谐变换**：有快速算法（$O(L^2 \log L)$），但实现复杂；实空间→频域的变换可表达为稀疏 matmul
+- **Clebsch-Gordan 系数**：预计算后是固定的稀疏张量，与特征的缩并可 GEMM 化
+- **Schur 约束的稀疏性**：等变层的权重矩阵被约束为块对角/标量，参数大幅减少，但需要稀疏/分块 GEMM
+- 关键瓶颈：高维 irrep（大 $l$ 的球谐）的 Clebsch-Gordan 张量规模增长快
+
+## 风险与失效条件
+
+- **高频球谐数值不稳定**：大 $l$ 的 $Y_l^m$ 在极点附近振荡剧烈，fp16 下精度损失严重
+- **irrep 完备性截断**：只取到 $l_{\max}$ 阶球谐丢失高频信息，截断误差需要实验标定
+- **非紧群的表示无穷维**：Lorentz 群等的不可约表示是无穷维的，工程上必须截断
+- **表示选择的自由度**：哪些 irrep 参与、阶数多少是超参数，缺乏自动选择方法
+- **Clebsch-Gordan 张量的存储**：$l$ 增大时 CG 系数数量 $O(l^3)$ 增长，预计算和存储开销上升
+
+## 深入参考
+
+- 蒸馏稿：references/books/micro-lie-theory.md（§II-F 伴随 Ad_X 与伴随矩阵）
+- 蒸馏稿：references/books/differential-geometry.md（Ch 5 Lie Groups, 伴随表示部分）
+- 原书：Joan Sola et al., *A micro Lie theory*, §II-F（伴随表示，式 30-35）
+- 原书：Jeffrey M. Lee, *Manifolds and Differential Geometry*, Ch 5（李群的表示）

package/knowledge-base/matrix-analysis/low-rank-approximation.en.md

@@ -0,0 +1,49 @@
+# Low-Rank Approximation
+
+## Minimal Definition
+
+Given a matrix $A \in \mathbb{R}^{m \times n}$, find a matrix $B$ with rank at most $k$ that minimizes $\|A - B\|$. The Eckart-Young-Mirsky theorem guarantees that the truncated SVD provides the unique optimal solution under both the Frobenius norm and the spectral norm: $B_k = \sum_{i=1}^k \sigma_i u_i v_i^H$.
+
+## Core Formulas
+
+- SVD: $A = U\Sigma V^H = \sum_{i=1}^r \sigma_i u_i v_i^H$, $\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r > 0$
+- Truncated SVD (optimal rank-$k$ approximation): $A_k = U_k \Sigma_k V_k^H$
+- Eckart-Young error: $\|A - A_k\|_F = \sqrt{\sum_{i=k+1}^r \sigma_i^2}$, $\|A - A_k\|_2 = \sigma_{k+1}$
+- Randomized SVD: $A \approx Q(Q^HA)$, where $Q$ is the $Q$-factor from the QR decomposition of $A\Omega$ ($\Omega$ random Gaussian)
+- Effective rank: $r_{\text{eff}}(A) = \|A\|_F^2 / \|A\|_2^2 = \sum \sigma_i^2 / \sigma_1^2$
+- Nuclear norm (convex relaxation of rank): $\|A\|_* = \sum \sigma_i$, the dual of the spectral norm
+
+## Applicable Problems
+
+- LoRA weight compression: $W \approx W_0 + BA$, $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times d}$, $r \ll d$
+- KV-Cache compression: projecting Key/Value caches into a low-dimensional subspace, reducing memory from $O(n)$ to $O(k)$
+- PCA / whitening: the top $k$ principal components of the data covariance matrix correspond to the truncated SVD
+- Gradient compression: the effective rank of gradient matrices in large models is often much lower than the nominal rank, allowing safe truncation
+- Recommender systems / matrix completion: low-rank factorization $R \approx UV^H$
+
+## AI Design Translation
+
+- **LoRA (Low-Rank Adaptation)**: Freeze $W_0$, train $\Delta W = BA$ ($r \ll d$), and merge $W = W_0 + BA$ at inference time. Forward pass = two matmul operations ($x \to Ax \to BAx$), reducing trainable parameters from $O(d^2)$ to $O(dr)$. Implemented via `torch.mm(B, torch.mm(A, x))` or merged into a single matmul.
+- **Randomized SVD operator**: For a large matrix $A \in \mathbb{R}^{m \times n}$, first sample $Y = A\Omega$ ($\Omega \in \mathbb{R}^{n \times (k+p)}$ random Gaussian), compute QR decomposition $Y = QR$, then form $B = Q^HA$ (small matrix $O(k \times n)$), and perform SVD on $B$. Total complexity $O(mnk)$ instead of $O(mn^2)$; all core operations are matmul.
+- **KV-Cache low-rank reduction**: Maintain $K_k = K P_k$ ($P_k$ being the projection onto the top $k$ principal components), performing incremental PCA or streaming SVD updates for each new token. Attention computation $\text{softmax}(Q K_k^H / \sqrt{d}) V_k$ involves three matmul operations, reducing the sequence dimension from $L$ to $k$.
+- **Nuclear norm regularization**: $\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \|W\|_*$ promotes low-rank solutions. However, nuclear norm computation requires full SVD ($O(n^3)$). Alternatives: (1) approximate with truncated SVD; (2) factorize $\|W\|_* = \min_{W=UV^H} \frac{1}{2}(\|U\|_F^2 + \|V\|_F^2)$, converting to Frobenius regularization on $U, V$.
+- **Gradient low-rank compression (distributed training)**: Truncate gradient $G$ to $G_k$ (top-$k$ SVD) before all-reduce, reducing communication from $O(d)$ to $O(kd)$. Use randomized SVD locally on each device, then merge.
+
+## Engineering Feasibility
+
+- **Primary operations**: matmul + small-matrix SVD. LoRA forward = two matmul operations; randomized SVD = three matmul operations + one small QR; truncated SVD = $O(k/n)$ fraction of full SVD (using Lanczos).
+- **GPU friendliness**: Extremely high. LoRA forward/backward are all tensor core matmul operations; the dominant cost of randomized SVD is also matmul. Small-matrix SVD has cuSOLVER batched implementations.
+- **Complexity**: LoRA forward $O(dk)$ per sample vs. $O(d^2)$ full rank; randomized SVD $O(mnk)$; full SVD $O(\min(m^2n, mn^2))$.
+- **Memory**: LoRA storage $O(dr)$ vs. $O(d^2)$; KV-Cache low-rank reduction $O(Lk)$ vs. $O(Ld)$.
+
+## Risks and Failure Conditions
+
+- **Incorrect rank selection**: $k$ too small causes information loss ($\sigma_{k+1}$ is non-negligible); $k$ too large negates the compression benefit. Solution: monitor the singular value decay curve and select the elbow point where $\sum_{i>k}\sigma_i^2 / \sum\sigma_i^2 < \epsilon$.
+- **Insufficient randomized SVD accuracy**: When oversampling $p$ is too small (typically $p = 5 \sim 10$) or the number of power iteration steps is insufficient, low-order singular value estimates can be significantly biased. Solution: add $q = 1 \sim 2$ power iteration steps $Y = (AA^H)^q A\Omega$, at the cost of additional matmul operations.
+- **LoRA not applicable to all layers**: Q/K/V in attention is typically effectively low-rank, but the effective rank of FFN layers and embeddings may be close to full rank, and applying LoRA forcefully degrades accuracy. Layer-by-layer effective rank diagnosis is required.
+- **SVD overhead of nuclear norm proximal**: Soft-thresholding $\text{prox}_{\lambda\|\cdot\|_*}(A) = U(\Sigma - \lambda I)_+ V^H$ requires SVD, which is prohibitively expensive for large matrices at every step. Solution: use factorization alternatives or randomized approximations.
+
+## Further References
+
+- Distilled notes: references/books/matrix-analysis.md (Section 2.6 SVD, Section 7.4 Polar Decomposition and SVD, nuclear norm-spectral norm duality Section 5.5)
+- Original text: Horn & Johnson, *Matrix Analysis* 2nd Ed., Chapter 2 Section 2.6 (SVD) + Chapter 7 Section 7.3-7.4 (Polar Decomposition & SVD)