math-skill 2.1.0 → 3.0.0

package/design-patterns/loss/variational-loss.en.md

@@ -0,0 +1,61 @@
+# Variational Loss
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When sampling from latent variable distributions is required to generate diverse outputs. Typical scenarios: (1) Expert selection introduces discrete latent variables $z$ that need end-to-end variational; (2) Representation spaces need to model uncertainty; (3) Generative routing requires sampling from posterior distributions $p(z|x)$; (4) Bayesian mixture of experts. Core objective: **model distributions rather than point estimates in latent space, enabling uncertainty awareness and diversity**.
+
+## Mathematical Inspiration
+- Lenses: lenses/probabilistic.md (variational inference and ELBO), lenses/probabilistic.md (posterior and prior)
+- Knowledge: knowledge-base/probability/entropy.md (KL divergence, variational families), knowledge-base/probability/kl-divergence.md (ELBO derivation)
+
+## Required Mathematical Knowledge
+- **ELBO (Evidence Lower Bound)**: $\log p(x) \geq \mathbb{E}_{q(z|x)}[\log p(x|z)] - \text{KL}(q(z|x) \| p(z))$ -- the first term is reconstruction likelihood, the second regularizes the posterior toward the prior
+- **Reparameterization Trick**: $z = \mu + \sigma \odot \varepsilon$, $\varepsilon \sim \mathcal{N}(0, I)$ -- enables gradient backpropagation through sampling operations
+- **Closed-Form KL Divergence**: When both $q$ and $p$ are Gaussian, $\text{KL}(\mathcal{N}(\mu,\sigma^2) \| \mathcal{N}(0,1)) = -\frac{1}{2} \sum(1 + \log \sigma^2 - \mu^2 - \sigma^2)$
+- **Gumbel-Softmax (Discrete Latent Variables)**: $z = \text{softmax}((\log \pi + g) / \tau)$, $g \sim \text{Gumbel}(0,1)$ -- continuous relaxation of discrete sampling
+
+## AI Module Form
+```
+Module: VariationalLoss
+Input: encoder outputs (mu, log_sigma^2) in R^{B x d_z}, reconstruction output x_hat in R^{B x d_x}, original input x
+
+Method 1 - Gaussian VAE Loss:
+  KL = -0.5 * sum(1 + log_sigma^2 - mu^2 - exp(log_sigma^2), dim=-1)   // closed form
+  recon = -log p(x|x_hat)  // MSE or BCE depending on data distribution
+  L_vae = recon + beta * KL    // beta-VAE controls disentanglement
+
+Method 2 - Gumbel-Softmax (discrete expert selection):
+  logits = encoder(x)  in R^{B x K}    // logits for K experts
+  g = -log(-log(uniform(B x K) + eps) + eps)   // Gumbel noise sampling
+  z_soft = softmax((logits + g) / tau)       // temperature tau annealed from 1.0 to 0.1
+  L_gumbel = CE(task_head(z_soft * features), y)  // straight-through estimator backprop
+
+Method 3 - Variational Information Bottleneck:
+  L_IB = -I(X; Z) + beta * I(Z; Y)  // maximize MI between Z and X, minimize redundancy between Z and Y
+  ~ E[-log q(x|z)] + beta * KL(q(z|x) || p(z))  // variational approximation
+```
+
+## Implementable Architectures
+- **Dual-Head Encoder Output**: Linear(d, 2 * d_z) -> split -> (mu, log_sigma^2), sharing base parameters
+- **Beta Annealing Strategy**: Linearly increase beta from 0 to the target value to prevent KL collapse (posterior collapse)
+- **Free Bits**: Set a KL lower bound lambda per dimension, penalizing only the excess: $\sum \max(\text{KL}_j, \lambda)$
+- **IWAE Multi-Particle**: Use log-mean-exp over $K$ samples instead of single-sample ELBO for a tighter lower bound
+
+## GPU Feasibility
+- **Tensorization**: Computation of mu and sigma^2 is a Linear layer (GEMM); KL involves element-wise operations
+- **GEMM-mappability**: Encoder 1 GEMM -> split -> reparameterization -> decoder 1 GEMM
+- **Complexity**: Same order as standard feedforward networks $O(B \cdot d^2)$; KL computation $O(B \cdot d_z)$ is negligible
+- **Memory & KV-Cache**: Additional storage of mu and sigma^2, two $B \times d_z$ matrices; minimal overhead
+- **Low Precision Stability**: log/exp operations in KL are recommended in fp32; Gumbel softmax log-log requires fp32
+- **Parallelism & Communication**: The $K$ samples in multi-particle IWAE can be sampled and computed in parallel
+- **Sparse Structure**: Discrete latent variables (Gumbel) degenerate to one-hot as $\tau \to 0$, naturally sparse
+- **Operator Fusion**: The Linear layers for mu and sigma^2 can share a single GEMM followed by split; KL exp/sub/add can be fused
+
+## Paper Phrasing
+"We adopt the variational inference framework, geometric expert selection as posterior inference over discrete latent variables $z$. End-to-end variational is achieved through Gumbel-Softmax relaxation, combined with a beta annealing strategy that effectively prevents posterior collapse. The ELBO lower bound converges at rate $O(\sqrt{d/n})$ with respect to latent variable dimensionality."
+
+## Risks
+- Posterior collapse: The KL term converges to 0 prematurely, causing latent variables to degenerate into prior samples and lose information
+- Gumbel-Softmax temperature $\tau$ annealing requires careful scheduling; too fast causes gradient vanishing, too slow loses discreteness
+- IWAE multi-particle log-mean-exp is numerically unstable in high dimensions; log-sum-exp trick is needed
+- Excessively large beta in beta-VAE degrades reconstruction quality; task-specific balancing is required

package/design-patterns/loss/variational-loss.md

@@ -0,0 +1,71 @@
+# Variational Loss（变分损失）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要从隐变量分布中采样并生成多样化输出时使用。典型场景：
+(1) 专家选择引入离散隐变量 z，需要端到端优化；(2) 表示空间需要建模不确定性；
+(3) 生成式路由中需要从后验分布 p(z|x) 采样；(4) 贝叶斯专家混合。
+核心诉求：**在潜空间中建模分布而非点估计，获得不确定性感知与多样性**。
+
+## 数学思想来源
+- 透镜：lenses/probabilistic.md（变分推断与 ELBO）、lenses/probabilistic.md（后验与先验）
+- 知识：knowledge-base/probability/entropy.md（KL 散度、变分族）、
+  knowledge-base/probability/kl-divergence.md（ELBO 推导）
+
+## 需要的数学知识
+- **ELBO（证据下界）**：log p(x) ≥ E_{q(z|x)}[log p(x|z)] - KL(q(z|x) ‖ p(z))
+  第一项为重构似然，第二项为正则项将后验推近先验
+- **重参数化技巧 (Reparameterization Trick)**：z = μ + σ ⊙ ε, ε ~ N(0, I)
+  使梯度可通过采样操作反向传播
+- **KL 散度闭合形式**：当 q 和 p 均为高斯时，
+  KL(N(μ,σ²) ‖ N(0,1)) = -½ Σ(1 + log σ² - μ² - σ²)
+- **Gumbel-Softmax（离散隐变量）**：
+  z = softmax((log π + g) / τ), g ~ Gumbel(0,1)，连续松弛离散采样
+
+## AI 模块形式
+```
+模块：VariationalLoss
+输入：编码器输出 (μ, log_σ²) ∈ R^{B×d_z}，重构输出 x̂ ∈ R^{B×d_x}，原始输入 x
+
+方法1 - 高斯 VAE 损失：
+  KL = -0.5 * sum(1 + log_σ² - μ² - exp(log_σ²), dim=-1)   // 闭合形式
+  recon = -log p(x|x̂)  // MSE 或 BCE 取决于数据分布
+  L_vae = recon + β · KL    // β-VAE 控制解耦程度
+
+方法2 - Gumbel-Softmax（离散专家选择）：
+  logits = encoder(x)  ∈ R^{B×K}    // K 个专家的 logits
+  g = -log(-log(uniform(B×K) + ε) + ε)   // Gumbel 噪声采样
+  z_soft = softmax((logits + g) / τ)       // 温度 τ 从 1.0 退火到 0.1
+  L_gumbel = CE(task_head(z_soft ⊙ features), y)  // 直通估计器反向传播
+
+方法3 - 信息瓶颈变分：
+  L_IB = -I(X; Z) + β · I(Z; Y)  // 最大化 Z 与 X 的互信息，最小化 Z 与 Y 的冗余
+  ≈ E[-log q(x|z)] + β · KL(q(z|x) ‖ p(z))  // 变分近似
+```
+
+## 可实现结构
+- **编码器双头输出**：Linear(d, 2·d_z) → split → (μ, log_σ²)，共享底层参数
+- **β 退火策略**：β 从 0 线性增长到目标值，防止 KL 坍塌（posterior collapse）
+- **Free Bits**：每个维度设定 KL 下界 λ，只惩罚超出部分：Σ max(KL_j, λ)
+- **IWAE 多粒子**：用 K 个样本的 log-mean-exp 替代单样本 ELBO，获得更紧下界
+
+## GPU 可行性
+- **张量化**：μ,σ² 的计算为 Linear 层（GEMM），KL 为 element-wise 运算
+- **GEMM 可映射**：编码器 1 次 GEMM → split → 重参数化 → 解码器 1 次 GEMM
+- **复杂度**：与标准前向网络同阶 O(B·d²)，KL 计算 O(B·d_z) 可忽略
+- **显存与 KV-Cache**：额外存储 μ, σ² 两个 B×d_z 矩阵，开销极小
+- **低精度稳定**：KL 的 log/exp 运算建议 fp32；Gumbel softmax 的 log-log 需 fp32
+- **并行与通信**：多粒子 IWAE 的 K 个样本可并行采样和计算
+- **稀疏结构**：离散隐变量 (Gumbel) 在 τ→0 时退化为 one-hot，天然稀疏
+- **算子融合**：μ 和 σ² 的 Linear 可共享一次 GEMM 后 split；KL 的 exp/sub/add 可融合
+
+## 论文表述方式
+"采用变分推断框架，将专家选择建模为离散隐变量 z 的后验推断，通过 Gumbel-Softmax
+松弛实现端到端优化，配合 β 退火策略有效避免 posterior collapse，
+ELBO 下界随隐变量维度以 O(√d/n) 收敛。"
+
+## 风险
+- Posterior collapse：KL 项过早收敛到 0，隐变量退化为先验采样，丧失信息
+- Gumbel-Softmax 的 τ 退火需要精心调度，过快导致梯度消失，过慢丧失离散性
+- IWAE 多粒子在高维下 log-mean-exp 数值不稳定，需 log-sum-exp trick
+- β-VAE 的 β 过大导致重构质量下降，需根据任务平衡

package/design-patterns/representation/equivariant-split.en.md

@@ -0,0 +1,87 @@
+# Equivariant Split
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when the input possesses symmetries (e.g., permutations, rotations, translations) and the representation should preserve or reflect those symmetries.
+Typical scenarios: (1) Token permutation equivariance -- representation should change accordingly when token order in a sentence changes (positional encoding);
+(2) Symmetry-based grouping of feature dimensions -- certain feature dimensions are invariant under specific transformations, while others are equivariant;
+(3) Multi-expert symmetry specialization -- different experts process different symmetry subspaces;
+(4) Geometric deep learning -- SE(3) equivariance in 3D molecular/protein structures.
+Core requirement: **encode symmetry priors into network architecture to reduce learning burden and improve generalization**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (group actions, invariant/equivariant maps), lenses/probabilistic.md (symmetry and information redundancy)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (group representation theory, irreducible representations),
+  knowledge-base/differential-geometry/manifold.md (Lie groups, homogeneous spaces)
+
+## Required Mathematical Background
+- **Group Action and Equivariance**: A map f is equivariant with respect to group G if and only if f(g * x) = g * f(x) for all g in G
+  Invariance is a special case of equivariance (g * f(x) = f(x), i.e., the trivial representation)
+- **Schur's Lemma and Irreducible Representation Decomposition**:
+  Any finite group representation decomposes as a direct sum of irreducible representations: V = direct_sum_i m_i * V_i
+  Equivariant linear maps are diagonal/block-diagonal between irreducible components
+- **Peter-Weyl Theorem**: Functions on a compact group decompose as a series of irreducible representation matrix elements
+  f(x) = sum_rho sum_{ij} c_{rho,ij} * rho_{ij}(g) (generalized Fourier expansion)
+- **Steerable Feature Spaces**: Features are organized according to irreducible representations of the group;
+  under the action of duality g, each component transforms via the corresponding representation matrix: f_i -> sum_j rho_{ij}(g) f_j
+
+## AI Module Form
+```
+Module: EquivariantSplit
+Input: X in R^{N x d}, symmetry group G (e.g., S_n permutation group, Z_n cyclic group, SO(3) rotation group)
+
+Method 1 - Split feature dimensions by irreducible representations:
+  // Decompose d-dimensional features by irreducible representations of the group
+  irreps = decompose(G, d)  // [(d_1, rho_1), (d_2, rho_2), ...] where sum d_i = d
+  X_split = split(X, [d_1, d_2, ...], dim=-1)  // split by irreducible components
+  // Process each component independently with equivariant layers:
+  for (X_i, rho_i) in zip(X_split, irreps):
+    Y_i = EquivariantLinear(X_i, rho_i)  // weights constrained by Schur's lemma
+  Y = concat(Y_i, dim=-1)                // reassemble
+
+Method 2 - Positional equivariant split (Token permutation group S_n):
+  // Transformer self-attention is naturally permutation equivariant (without positional encoding)
+  // Explicitly introduce controllable permutation equivariance:
+  X_content = X[:, :d_content]            // permutation-invariant content part
+  X_position = X[:, d_content:]           // position-dependent part
+  // Content part uses permutation-invariant pooling:
+  z_inv = mean(X_content, dim=1)          // global invariant features
+  // Position part uses equivariant operations:
+  z_equiv = Attention(X_position, X_position, X_position)  // permutation equivariant
+  output = z_equiv + MLP(z_inv).unsqueeze(1)  // broadcast invariant signal back
+
+Method 3 - Group convolution / group pooling:
+  // Features defined on group G: f: G -> R^c
+  // Group convolution: (f * psi)(g) = sum_{h in G} f(h) * psi(h^{-1} g)
+  // Group pooling: pool over orbits of subgroup H < G
+  // Implemented as matrix multiplication (group multiplication table -> sparse permutation matrices)
+  for g in generators(G):
+    X_g = permutation_matrix(g) @ X       // group generator action
+    features_g = Linear(X_g)              // equivariant processing with shared weights
+  output = aggregate(features_g)          // aggregate along group dimension
+```
+
+## Implementable Structures
+- **e3nn / lie_learn integration**: Use existing libraries for SO(3)/SE(3) irreducible representations and spherical harmonics
+- **Blocked feature storage**: Organize feature dimensions by irreducible representations; each block independently normalized and processed
+- **Precomputed group operations**: Group multiplication tables, Clebsch-Gordan coefficients, etc., computed once and cached
+- **Symmetry augmentation**: Apply random group elements g in G to inputs during training (data augmentation) to encourage equivariance
+
+## GPU Feasibility
+- **Tensorization**: Processing irreducible components is batched GEMM; group convolution is sparse GEMM or batched matmul
+- **GEMM-mappable**: Each block of EquivariantLinear is an independent GEMM (N x d_i) @ (d_i x d_i_out), batchable
+- **Complexity**: Same order as standard networks (Schur constraints actually reduce parameters); group convolution incurs an extra |G| factor
+- **Memory & KV-Cache**: Group convolution requires storing |G| copies of features; significant memory pressure when |G| is large
+- **Low-precision stability**: Spherical harmonics Y_l^m computations involve factorials and square roots; fp32 recommended
+- **Parallelism & Communication**: Irreducible components are independent, perfectly parallel; different g in group convolution can be parallelized
+- **Sparse structure**: Permutation matrices in group convolution are extremely sparse (exactly one nonzero per row/column); SpMM is efficient
+- **Operator fusion**: Split -> batched matmul -> concat can be fused; group pooling scatter + reduce can be fused
+
+## Paper-Worthy Formulation
+"Leveraging Schur's lemma from group representation theory, we decompose the d-dimensional feature space into a direct sum of irreducible representations of symmetry group G, with each component independently processed by equivariance-constrained linear layers. This achieves a |G| / sum(m_i^2) improvement in parameter efficiency while guaranteeing network equivariance under group actions, reducing generalization error by O(1/sqrt(|G|)) on symmetric data."
+
+## Risks
+- Improper group selection (too large constrains cause underfitting, too small fails to capture symmetries)
+- Irreducible representation decomposition requires domain knowledge; implementation is complex for non-standard groups
+- The |G|-fold feature storage in group convolution becomes infeasible for large groups (e.g., S_10 has 3.6M elements)
+- The trade-off between approximate equivariance (soft equivariance) and strict equivariance is difficult to control

package/design-patterns/representation/equivariant-split.md

@@ -0,0 +1,89 @@
+# Equivariant Split（等变分割）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入具有对称性（如排列、旋转、平移），且表示应保持或反映这些对称性时使用。
+典型场景：(1) Token 排列等变——句子中 token 顺序变化，表示应相应变化（位置编码）；
+(2) 特征维度的对称性分组——某些特征维度在特定变换下不变，某些等变；
+(3) 多专家对称分工——不同专家处理不同对称性子空间；
+(4) 几何深度学习——3D 分子/蛋白质结构中 SE(3) 等变性。
+核心诉求：**让网络结构编码对称性先验，减少学习负担，提升泛化**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（群作用、不变/等变映射）、lenses/probabilistic.md（对称性与信息冗余）
+- 知识：knowledge-base/matrix-analysis/projection.md（群表示论、不可约表示）、
+  knowledge-base/differential-geometry/manifold.md（李群、齐性空间）
+
+## 需要的数学知识
+- **群作用与等变性**：映射 f 对群 G 等变 ⟺ f(g·x) = g·f(x), ∀g∈G
+  不变性是等变性的特例（g·f(x) = f(x)，即平凡表示）
+- **Schur 引理与不可约表示分解**：
+  任何有限群表示可分解为不可约表示的直和：V = ⊕_i m_i · V_i
+  等变线性映射在不可约分量间是对角/块对角的
+- **Peter-Weyl 定理**：紧群上的函数可分解为不可约表示矩阵元的级数
+  f(x) = Σ_ρ Σ_{ij} c_{ρ,ij} · ρ_{ij}(g)（广义 Fourier 展开）
+- **Steerable 特征空间**：特征按群的不可约表示组织，
+  变换 g 作用时各分量按对应的表示矩阵变换：f_i → Σ_j ρ_{ij}(g) f_j
+
+## AI 模块形式
+```
+模块：EquivariantSplit
+输入：X ∈ R^{N×d}，对称群 G（如 S_n 排列群、Z_n 循环群、SO(3) 旋转群）
+
+方法1 - 按不可约表示分割特征维度：
+  // 将 d 维特征按群的不可约表示分解
+  irreps = decompose(G, d)  // [(d₁, ρ₁), (d₂, ρ₂), ...] 其中 Σdᵢ = d
+  X_split = split(X, [d₁, d₂, ...], dim=-1)  // 按不可约分量切分
+  // 每个分量用等变层独立处理：
+  for (X_i, ρ_i) in zip(X_split, irreps):
+    Y_i = EquivariantLinear(X_i, ρ_i)  // 权重受 Schur 约束
+  Y = concat(Y_i, dim=-1)              // 重组
+
+方法2 - 位置等变分割（Token 排列群 S_n）：
+  // Transformer 自注意力天然对排列等变（无位置编码时）
+  // 显式引入可控的排列等变性：
+  X_content = X[:, :d_content]           // 排列不变的内容部分
+  X_position = X[:, d_content:]          // 位置相关的部分
+  // 内容部分用排列不变的池化：
+  z_inv = mean(X_content, dim=1)         // 全局不变特征
+  // 位置部分用等变操作：
+  z_equiv = Attention(X_position, X_position, X_position)  // 排列等变
+  output = z_equiv + MLP(z_inv).unsqueeze(1)  // 不变信号广播回去
+
+方法3 - 群卷积/群池化：
+  // 特征定义在群 G 上：f: G → R^c
+  // 群卷积：(f * ψ)(g) = Σ_{h∈G} f(h) · ψ(h⁻¹g)
+  // 群池化：pool over orbits of subgroup H < G
+  // 实现为矩阵乘法（群乘法表→稀疏置换矩阵）
+  for g in generators(G):
+    X_g = permutation_matrix(g) @ X    // 群生成元作用
+    features_g = Linear(X_g)            // 共享权重的等变处理
+  output = aggregate(features_g)        // 沿群维度聚合
+```
+
+## 可实现结构
+- **e3nn / lie_learn 集成**：使用现有库处理 SO(3)/SE(3) 的不可约表示和球谐函数
+- **特征分块存储**：按不可约表示组织特征维度，每个块独立归一化和处理
+- **群操作预计算**：群乘法表、Clebsch-Gordan 系数等一次性计算并缓存
+- **对称性增强**：训练时对输入施加随机群元素 g∈G（数据增强），鼓励等变性
+
+## GPU 可行性
+- **张量化**：不可约分量的处理为 batched GEMM；群卷积为稀疏 GEMM 或 batched matmul
+- **GEMM 可映射**：EquivariantLinear 的每个块为独立 GEMM (N×dᵢ)@(dᵢ×dᵢ_out)，可 batch
+- **复杂度**：与标准网络同阶（Schur 约束反而减少参数），群卷积额外 |G| 倍
+- **显存与 KV-Cache**：群卷积需存储 |G| 份特征，|G| 大时显存压力显著
+- **低精度稳定**：球谐函数 Y_l^m 的计算涉及阶乘和平方根，建议 fp32
+- **并行与通信**：不可约分量间独立，完美并行；群卷积的不同 g 可并行
+- **稀疏结构**：群卷积的置换矩阵极度稀疏（每行/列恰好一个非零），SpMM 高效
+- **算子融合**：split → batched matmul → concat 可融合；群池化的 scatter+reduce 可融合
+
+## 论文表述方式
+"利用群表示论的 Schur 引理，将 d 维特征空间按对称群 G 的不可约表示分解为直和 ⊕mᵢVᵢ，
+每个分量使用受等变约束的线性层独立处理，参数效率提升 |G|/Σmᵢ² 倍，
+同时保证网络对群作用的等变性，在对称数据上泛化误差降低 O(1/√|G|)。"
+
+## 风险
+- 群的选择不当（过大约束导致欠拟合，过小无法捕获对称性）
+- 不可约表示分解需要领域知识，非标准群的实现复杂
+- 群卷积的 |G| 倍特征存储在群阶大时不可行（如 S_10 有 3.6M 个元素）
+- 近似等变（soft equivariance）vs 严格等变的权衡难以控制

package/design-patterns/representation/manifold-representation.en.md

@@ -0,0 +1,85 @@
+# Manifold Representation
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when input data resides in a high-dimensional space but is actually distributed on a low-dimensional manifold. Typical scenarios:
+(1) Semantic space of natural language tokens -- although dimensionality d = 4096, the effective degrees of freedom are far less than d;
+(2) Multi-modal alignment -- text and image distributions lie on different manifolds that need to be aligned;
+(3) Expert feature spaces -- each expert processes a different local region of the manifold;
+(4) Dimensionality reduction / compression -- exploit the low-dimensional manifold nature to reduce parameter count.
+Core requirement: **leverage the low-dimensional manifold structure of data to improve representation efficiency and generalization**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (Riemannian geometry, geodesics, curvature), lenses/probabilistic.md (intrinsic dimensionality)
+- Knowledge: knowledge-base/differential-geometry/manifold.md (manifolds, tangent spaces, exponential maps),
+  knowledge-base/matrix-analysis/projection.md (SVD, low-rank approximation, PCA)
+
+## Required Mathematical Background
+- **Manifold Hypothesis**: Data x in R^D actually lies on a smooth d-dimensional manifold M with d << D
+  Locally approximable by the tangent space T_pM isomorphic to R^d
+- **Local Coordinate Chart**: phi: U subset M -> R^d, mapping a manifold patch to low-dimensional Euclidean space
+  Multiple charts {phi_i} form an atlas covering the entire manifold
+- **Geodesic Distance**: d_M(p, q) = inf integral ||gamma'(t)|| dt, the shortest path between two points on the manifold
+  Approximate computation: Dijkstra/Isomap on a k-NN graph
+- **Exponential Map / Logarithmic Map**:
+  exp_p: T_pM -> M (tangent space to manifold), log_p: M -> T_pM (manifold to tangent space)
+  Used to perform linear operations in the tangent space and map back to the manifold
+
+## AI Module Form
+```
+Module: ManifoldRepresentation
+Input: X in R^{N x D} (high-dimensional input), target manifold dimension d << D
+
+Method 1 - Local Linear Embedding (Chart-based):
+  // Partition the d-dimensional manifold into K local regions, each with a linear projection
+  assignments = cluster(X, K)          // assign inputs to K local regions
+  for k in range(K):
+    z_k = W_k @ X[assignments==k] + b_k  // local linear projection
+  // Equivalent to MoE: K "chart experts" each responsible for one manifold patch
+  z = sum_k g_k(x) * (W_k @ x + b_k)   // g_k is the chart assignment weight
+
+Method 2 - Geodesic Preservation Loss (global structure preservation):
+  // Preserve geodesic distances from high-dimensional space in the low-dimensional representation
+  D_high = geodesic_distance(X, k_nn=10)   // shortest paths on k-NN graph
+  D_low = pairwise_distance(Z)              // Euclidean distances in low-dim representation
+  L_geo = ||D_high - D_low||_F^2 / N^2     // Sammon mapping
+  // Or t-SNE-style KL divergence:
+  p_ij = exp(-D_high^2 / (2 sigma^2)) / sum  // high-dim affinity
+  q_ij = 1 / (1 + D_low^2) / sum             // low-dim t-distribution affinity
+  L_tsne = KL(P || Q)
+
+Method 3 - Riemannian Optimization (optimize directly on the manifold):
+  // Parameters constrained to Stiefel/Grassmann manifold
+  W in St(d, r) i.e. W^T W = I_r          // orthogonality constraint
+  // Riemannian SGD:
+  grad_euclidean = nabla f(W)
+  grad_riemannian = grad_euclidean - W @ (W^T @ grad_euclidean)  // project to tangent space
+  W = retract(W, -lr * grad_riemannian)    // retraction mapping back to manifold
+  // retract can be implemented via QR decomposition or Cayley transform
+```
+
+## Implementable Structures
+- **Chart MoE**: K local linear projections + softmax gating => natural integration with the MoE framework
+- **Manifold regularization**: L_manifold = tr(Z^T L Z) / N^2, where L is the graph Laplacian and Z is the representation
+  Encourages nearby samples to have similar representations
+- **Intrinsic dimension estimation**: Use MLE or two-norm methods to estimate the effective dimensionality d* of the data
+- **Adaptive d**: Local dimensionality varies across regions; estimate locally via PCA
+
+## GPU Feasibility
+- **Tensorization**: Local linear projections are GEMM (d x D) @ (D x N); graph Laplacian regularization is SpMM
+- **GEMM-mappable**: K linear projections in Chart MoE form a batched GEMM (K x d x D) @ (D x N)
+- **Complexity**: k-NN construction O(N * D * log N) requires FAISS; manifold regularization O(N^2) requires sampling approximation
+- **Memory & KV-Cache**: K chart parameters K * d * D typically < 10 MB; k-NN graph N * k * 4 bytes
+- **Low-precision stability**: Distance computations and exp in fp16 require attention to numerical range; Riemannian retract recommended in fp32
+- **Parallelism & Communication**: K charts computed independently, perfectly parallel; k-NN search accelerated with FAISS GPU
+- **Sparse structure**: k-NN graph is naturally sparse; manifold regularization L is a sparse matrix, enabling SpMM acceleration
+- **Operator fusion**: Matmul + bias + activation within a chart can be fused; gating softmax + weighted-sum can be fused
+
+## Paper-Worthy Formulation
+"Based on the manifold hypothesis, we constrain D-dimensional token representations to a d*-dimensional (intrinsic dimension estimate) submanifold, achieving piecewise-linear approximation via K local coordinate charts (Chart MoE) and preserving global geodesic structure through graph Laplacian manifold regularization, with theoretical guarantees that embedding error converges at rate O(N^{-2/d*})."
+
+## Risks
+- Inaccurate intrinsic dimension d* estimation leads to over-compression or dimension waste
+- k-NN graph construction is computationally expensive at large scale, requiring sampling or approximation
+- The N^2 complexity of manifold regularization limits batch size, necessitating mini-batch sampling
+- Discontinuities at local chart boundaries require overlapping regions and smooth transitions

package/design-patterns/representation/manifold-representation.md

@@ -0,0 +1,87 @@
+# Manifold Representation（流形表示）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入数据虽然在高维空间中，但实际分布在低维流形上时使用。典型场景：
+(1) 自然语言 token 的语义空间——虽维度 d=4096，但有效自由度远小于 d；
+(2) 多模态对齐——文本和图像分布在不同的流形上，需要对齐；
+(3) 专家特征空间——每个专家处理流形的不同局部区域；
+(4) 降维/压缩——利用流形低维本质减少参数量。
+核心诉求：**利用数据的低维流形结构，提高表示效率和泛化能力**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（黎曼几何、测地线、曲率）、lenses/probabilistic.md（内在维度）
+- 知识：knowledge-base/differential-geometry/manifold.md（流形、切空间、指数映射）、
+  knowledge-base/matrix-analysis/projection.md（SVD、低秩近似、PCA）
+
+## 需要的数学知识
+- **流形假设**：数据 x ∈ R^D 实际分布在 d << D 维的光滑流形 M 上
+  局部可用切空间 T_pM ≈ R^d 线性近似
+- **局部坐标卡 (Chart)**：φ: U ⊂ M → R^d，将流形局部映射到低维欧氏空间
+  多个坐标卡 {φ_i} 构成图册（Atlas），覆盖整个流形
+- **测地线距离**：d_M(p,q) = inf ∫‖γ'(t)‖dt，流形上两点间最短路径
+  近似计算：k-NN 图上 Dijkstra/Isomap
+- **指数映射/对数映射**：
+  exp_p: T_pM → M（切空间到流形），log_p: M → T_pM（流形到切空间）
+  用于在切空间中做线性运算后映射回流形
+
+## AI 模块形式
+```
+模块：ManifoldRepresentation
+输入：X ∈ R^{N×D}（高维输入），目标流形维度 d << D
+
+方法1 - 局部线性嵌入（Chart-based）：
+  // 将 d 维流形分成 K 个局部区域，每个区域用线性投影
+  assignments = cluster(X, K)         // 将输入分配到 K 个局部区域
+  for k in range(K):
+    z_k = W_k @ X[assignments==k] + b_k  // 局部线性投影 d×D → d×d
+  // 等效为 MoE：K 个"chart expert"各负责流形一个局部
+  z = Σ_k g_k(x) · (W_k @ x + b_k)   // g_k 为 chart 分配权重
+
+方法2 - 测地线保持损失（全局结构保持）：
+  // 保持高维空间的测地线距离在低维表示中不变
+  D_high = geodesic_distance(X, k_nn=10)   // k-NN 图上最短路
+  D_low = pairwise_distance(Z)              // 低维表示的欧氏距离
+  L_geo = ‖D_high - D_low‖_F² / N²         // Sammon mapping
+  // 或用 t-SNE 式 KL 散度：
+  p_ij = exp(-D_high²/2σ²) / Σ  // 高维亲和度
+  q_ij = 1/(1+D_low²) / Σ        // 低维 t 分布亲和度
+  L_tsne = KL(P ‖ Q)
+
+方法3 - 黎曼优化（直接在流形上优化）：
+  // 参数约束在 Stiefel/Grassmann 流形上
+  W ∈ St(d, r)  i.e. W^T W = I_r           // 正交约束
+  // Riemannian SGD：
+  grad_euclidean = ∇f(W)
+  grad_riemannian = grad_euclidean - W @ (W^T @ grad_euclidean)  // 投影到切空间
+  W = retract(W, -lr · grad_riemannian)     // 缩回映射到流形
+  // retract 可用 QR 分解或 Cayley 变换实现
+```
+
+## 可实现结构
+- **Chart MoE**：K 个局部线性投影 + softmax 门控 → 天然与 MoE 框架集成
+- **流形正则化**：L_manifold = tr(Z^T L Z) / N²，L 为图拉普拉斯，Z 为表示
+  鼓励相邻样本的表示相近
+- **内在维度估计**：用 MLE 或 two-norm 方法估计数据的有效维度 d*
+- **自适应 d**：不同区域的局部维度不同，用 PCA 局部估计
+
+## GPU 可行性
+- **张量化**：局部线性投影为 GEMM (d×D)@(D×N)；图拉普拉斯正则为 SpMM
+- **GEMM 可映射**：Chart MoE 的 K 个线性投影为 batched GEMM (K×d×D)@(D×N)
+- **复杂度**：k-NN 构建 O(N·D·log N) 需 FAISS；流形正则 O(N²) 需采样近似
+- **显存与 KV-Cache**：K 个 chart 参数 K·d·D 通常 <10MB；k-NN 图 N·k·4 bytes
+- **低精度稳定**：距离计算和 exp 在 fp16 下需注意数值范围；黎曼 retract 建议 fp32
+- **并行与通信**：K 个 chart 独立计算，完美并行；k-NN 搜索用 FAISS GPU 加速
+- **稀疏结构**：k-NN 图天然稀疏，流形正则 L 为稀疏矩阵，SpMM 加速
+- **算子融合**：chart 内的 matmul+bias+activation 可融合；门控 softmax+weighted-sum 可融合
+
+## 论文表述方式
+"基于流形假设将 D 维 token 表示约束到 d*（内在维度估计）维子流形上，
+通过 K 个局部坐标卡（Chart MoE）实现分段线性近似，配合图拉普拉斯流形正则
+保持全局测地线结构，理论上保证嵌入误差以 O(N^{-2/d*}) 收敛。"
+
+## 风险
+- 内在维度 d* 估计不准确导致过度压缩或维度浪费
+- k-NN 图构建在大规模数据下计算代价高，需采样或近似
+- 流形正则的 N² 复杂度限制 batch size，需 mini-batch 采样
+- 局部 chart 边界处表示不连续，需重叠区域和平滑过渡

package/design-patterns/representation/shared-private-decomposition.en.md

@@ -0,0 +1,84 @@
+# Shared-Private Decomposition
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use in multi-task/multi-domain learning when representations need to be decomposed into a "cross-task common component" and a "task-specific component."
+Typical scenarios: (1) Multi-task MoE -- Shared experts handle linguistic commonalities, Private experts handle task-specific logic;
+(2) Multi-domain adaptation -- Shared representations capture general semantics, Private representations capture domain terminology;
+(3) Continual learning -- Shared retains stable knowledge, Private accommodates new knowledge without interfering with old knowledge.
+Core requirement: **explicitly separate commonality from individuality to prevent negative transfer and catastrophic forgetting**.
+
+## Mathematical Inspiration
+- Lenses: lenses/projection.md (subspace decomposition, direct sum decomposition), lenses/probabilistic.md (information decomposition)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (direct sum decomposition V = U + W, projection operators),
+  knowledge-base/probability/kl-divergence.md (information decomposition: shared/synergy/unique)
+
+## Required Mathematical Background
+- **Direct Sum Decomposition**: R^d = S + P, where S intersect P = {0}, and every x = x_S + x_P is unique
+  Projection matrices P_S + P_P = I, P_S * P_P = 0
+- **Information Decomposition (Williams & Beer PID)**:
+  I(X; Y_1, Y_2) = Shared + Unique_1 + Unique_2 + Synergy
+  Shared = the redundant information component min(I(X; Y_1), I(X; Y_2))
+- **Low-Rank + Sparse Decomposition (RPCA)**: M = L + S, where L is low-rank (common) + S is sparse (specific)
+  Solved via nuclear norm + L1 norm convex relaxation
+- **CCA (Canonical Correlation Analysis)**: max corr(W_1^T X, W_2^T Y), extracting shared variation between two sets of variables
+
+## AI Module Form
+```
+Module: SharedPrivateDecomposer
+Input: X in R^{N x d}, task identifier t in {1, ..., T}
+
+Method 1 - Additive Decomposition (most common):
+  z_shared = E_shared(X)           // shared encoder: MLP or Transformer block
+  z_private = E_private[t](X)      // private encoder: independent parameters per task
+  z = z_shared + z_private         // additive fusion
+  // Training objective: L_task(z, y) + lambda_1 * OrthLoss(z_shared, z_private)
+  // Orthogonality ensures shared and private learn different information
+
+Method 2 - Gated Decomposition (dynamic weighting):
+  z_shared = E_shared(X)
+  z_private = E_private[t](X)
+  gate = sigmoid(Linear(z_shared + z_private))  // dynamic fusion gate
+  z = gate * z_shared + (1 - gate) * z_private
+  // Gating allows per-dimension selection of shared/private contribution ratios
+
+Method 3 - Adversarial Decomposition (information-theoretic guarantee):
+  z_shared = E_shared(X)
+  z_private = E_private[t](X)
+  // Shared should be indistinguishable across tasks (adversarial gradient):
+  task_pred = classifier(z_shared.flip_gradient())
+  L_adv = -CE(task_pred, t)        // Shared contains no task information
+  // Private should be discriminative across tasks:
+  L_private = CE(classifier(z_private), t)
+  L = L_task + lambda_adv * L_adv + lambda_priv * L_private
+
+Dimension Allocation Principle:
+  d_shared = d * T / (T + 1)      // With T tasks, shared occupies the majority
+  d_private = d * 1 / (T + 1)     // Each private occupies a smaller portion
+  // Or dynamically allocate based on PCA variance explained ratio
+```
+
+## Implementable Structures
+- **Dual encoder + fusion layer**: shared_encoder (large) + T private_encoders (small) + fusion
+- **Parameter efficiency**: Private uses LoRA (Low-Rank Adaptation) instead of full encoders, O(d * r) parameters per task
+- **Dynamic routing integration**: Shared experts + private experts selected via MoE routing
+- **Progressive expansion**: For new tasks, only add private encoders with frozen shared parameters
+
+## GPU Feasibility
+- **Tensorization**: Two encoder forward passes are independent GEMM chains, executable in parallel
+- **GEMM-mappable**: Shared/private encoders are each standard Transformer FFNs (2x GEMM)
+- **Complexity**: Shared O(N * d^2) + T private encoders O(N * d^2 / T), total approximately 2x a single encoder
+- **Memory & KV-Cache**: All T private encoder parameters stored; LoRA compression needed when T is large
+- **Low-precision stability**: Additive/gated fusion is safe in fp16; adversarial training gradient reversal requires fp32
+- **Parallelism & Communication**: Shared and private encoders can be assigned to different GPUs; multi-task batches mixed for training
+- **Sparse structure**: Private encoders can be sparsified (only the current task's is activated); only 1 out of T activated
+- **Operator fusion**: Additive fusion is trivial; gated fusion sigmoid -> multiply -> add can be fused
+
+## Paper-Worthy Formulation
+"We decompose the multi-task representation space R^d into a direct sum S + P. The Shared subspace ensures task-invariance through adversarial training (H(T|Z_s) -> log T), while the Private subspace guarantees information complementarity with Shared via orthogonal regularization. Theoretical analysis shows that negative transfer decays as O(||P_S * P_P||_F) with increasing orthogonality."
+
+## Risks
+- Min-max variational in adversarial training is unstable; the gradient reversal scale and lambda_adv require careful tuning
+- Over-compression of Shared leads to insufficient common information, placing excessive burden on Private
+- Total Private parameters grow linearly with T, requiring LoRA or adapter modules to control
+- When task similarity is low, Shared may learn a vacuous "common component"