math-skill 2.0.1 → 3.0.0

package/design-patterns/representation/equivariant-split.md

@@ -0,0 +1,89 @@
+# Equivariant Split（等变分割）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入具有对称性（如排列、旋转、平移），且表示应保持或反映这些对称性时使用。
+典型场景：(1) Token 排列等变——句子中 token 顺序变化，表示应相应变化（位置编码）；
+(2) 特征维度的对称性分组——某些特征维度在特定变换下不变，某些等变；
+(3) 多专家对称分工——不同专家处理不同对称性子空间；
+(4) 几何深度学习——3D 分子/蛋白质结构中 SE(3) 等变性。
+核心诉求：**让网络结构编码对称性先验，减少学习负担，提升泛化**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（群作用、不变/等变映射）、lenses/probabilistic.md（对称性与信息冗余）
+- 知识：knowledge-base/matrix-analysis/projection.md（群表示论、不可约表示）、
+  knowledge-base/differential-geometry/manifold.md（李群、齐性空间）
+
+## 需要的数学知识
+- **群作用与等变性**：映射 f 对群 G 等变 ⟺ f(g·x) = g·f(x), ∀g∈G
+  不变性是等变性的特例（g·f(x) = f(x)，即平凡表示）
+- **Schur 引理与不可约表示分解**：
+  任何有限群表示可分解为不可约表示的直和：V = ⊕_i m_i · V_i
+  等变线性映射在不可约分量间是对角/块对角的
+- **Peter-Weyl 定理**：紧群上的函数可分解为不可约表示矩阵元的级数
+  f(x) = Σ_ρ Σ_{ij} c_{ρ,ij} · ρ_{ij}(g)（广义 Fourier 展开）
+- **Steerable 特征空间**：特征按群的不可约表示组织，
+  变换 g 作用时各分量按对应的表示矩阵变换：f_i → Σ_j ρ_{ij}(g) f_j
+
+## AI 模块形式
+```
+模块：EquivariantSplit
+输入：X ∈ R^{N×d}，对称群 G（如 S_n 排列群、Z_n 循环群、SO(3) 旋转群）
+
+方法1 - 按不可约表示分割特征维度：
+  // 将 d 维特征按群的不可约表示分解
+  irreps = decompose(G, d)  // [(d₁, ρ₁), (d₂, ρ₂), ...] 其中 Σdᵢ = d
+  X_split = split(X, [d₁, d₂, ...], dim=-1)  // 按不可约分量切分
+  // 每个分量用等变层独立处理：
+  for (X_i, ρ_i) in zip(X_split, irreps):
+    Y_i = EquivariantLinear(X_i, ρ_i)  // 权重受 Schur 约束
+  Y = concat(Y_i, dim=-1)              // 重组
+
+方法2 - 位置等变分割（Token 排列群 S_n）：
+  // Transformer 自注意力天然对排列等变（无位置编码时）
+  // 显式引入可控的排列等变性：
+  X_content = X[:, :d_content]           // 排列不变的内容部分
+  X_position = X[:, d_content:]          // 位置相关的部分
+  // 内容部分用排列不变的池化：
+  z_inv = mean(X_content, dim=1)         // 全局不变特征
+  // 位置部分用等变操作：
+  z_equiv = Attention(X_position, X_position, X_position)  // 排列等变
+  output = z_equiv + MLP(z_inv).unsqueeze(1)  // 不变信号广播回去
+
+方法3 - 群卷积/群池化：
+  // 特征定义在群 G 上：f: G → R^c
+  // 群卷积：(f * ψ)(g) = Σ_{h∈G} f(h) · ψ(h⁻¹g)
+  // 群池化：pool over orbits of subgroup H < G
+  // 实现为矩阵乘法（群乘法表→稀疏置换矩阵）
+  for g in generators(G):
+    X_g = permutation_matrix(g) @ X    // 群生成元作用
+    features_g = Linear(X_g)            // 共享权重的等变处理
+  output = aggregate(features_g)        // 沿群维度聚合
+```
+
+## 可实现结构
+- **e3nn / lie_learn 集成**：使用现有库处理 SO(3)/SE(3) 的不可约表示和球谐函数
+- **特征分块存储**：按不可约表示组织特征维度，每个块独立归一化和处理
+- **群操作预计算**：群乘法表、Clebsch-Gordan 系数等一次性计算并缓存
+- **对称性增强**：训练时对输入施加随机群元素 g∈G（数据增强），鼓励等变性
+
+## GPU 可行性
+- **张量化**：不可约分量的处理为 batched GEMM；群卷积为稀疏 GEMM 或 batched matmul
+- **GEMM 可映射**：EquivariantLinear 的每个块为独立 GEMM (N×dᵢ)@(dᵢ×dᵢ_out)，可 batch
+- **复杂度**：与标准网络同阶（Schur 约束反而减少参数），群卷积额外 |G| 倍
+- **显存与 KV-Cache**：群卷积需存储 |G| 份特征，|G| 大时显存压力显著
+- **低精度稳定**：球谐函数 Y_l^m 的计算涉及阶乘和平方根，建议 fp32
+- **并行与通信**：不可约分量间独立，完美并行；群卷积的不同 g 可并行
+- **稀疏结构**：群卷积的置换矩阵极度稀疏（每行/列恰好一个非零），SpMM 高效
+- **算子融合**：split → batched matmul → concat 可融合；群池化的 scatter+reduce 可融合
+
+## 论文表述方式
+"利用群表示论的 Schur 引理，将 d 维特征空间按对称群 G 的不可约表示分解为直和 ⊕mᵢVᵢ，
+每个分量使用受等变约束的线性层独立处理，参数效率提升 |G|/Σmᵢ² 倍，
+同时保证网络对群作用的等变性，在对称数据上泛化误差降低 O(1/√|G|)。"
+
+## 风险
+- 群的选择不当（过大约束导致欠拟合，过小无法捕获对称性）
+- 不可约表示分解需要领域知识，非标准群的实现复杂
+- 群卷积的 |G| 倍特征存储在群阶大时不可行（如 S_10 有 3.6M 个元素）
+- 近似等变（soft equivariance）vs 严格等变的权衡难以控制

package/design-patterns/representation/manifold-representation.en.md

@@ -0,0 +1,85 @@
+# Manifold Representation
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when input data resides in a high-dimensional space but is actually distributed on a low-dimensional manifold. Typical scenarios:
+(1) Semantic space of natural language tokens -- although dimensionality d = 4096, the effective degrees of freedom are far less than d;
+(2) Multi-modal alignment -- text and image distributions lie on different manifolds that need to be aligned;
+(3) Expert feature spaces -- each expert processes a different local region of the manifold;
+(4) Dimensionality reduction / compression -- exploit the low-dimensional manifold nature to reduce parameter count.
+Core requirement: **leverage the low-dimensional manifold structure of data to improve representation efficiency and generalization**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (Riemannian geometry, geodesics, curvature), lenses/probabilistic.md (intrinsic dimensionality)
+- Knowledge: knowledge-base/differential-geometry/manifold.md (manifolds, tangent spaces, exponential maps),
+  knowledge-base/matrix-analysis/projection.md (SVD, low-rank approximation, PCA)
+
+## Required Mathematical Background
+- **Manifold Hypothesis**: Data x in R^D actually lies on a smooth d-dimensional manifold M with d << D
+  Locally approximable by the tangent space T_pM isomorphic to R^d
+- **Local Coordinate Chart**: phi: U subset M -> R^d, mapping a manifold patch to low-dimensional Euclidean space
+  Multiple charts {phi_i} form an atlas covering the entire manifold
+- **Geodesic Distance**: d_M(p, q) = inf integral ||gamma'(t)|| dt, the shortest path between two points on the manifold
+  Approximate computation: Dijkstra/Isomap on a k-NN graph
+- **Exponential Map / Logarithmic Map**:
+  exp_p: T_pM -> M (tangent space to manifold), log_p: M -> T_pM (manifold to tangent space)
+  Used to perform linear operations in the tangent space and map back to the manifold
+
+## AI Module Form
+```
+Module: ManifoldRepresentation
+Input: X in R^{N x D} (high-dimensional input), target manifold dimension d << D
+
+Method 1 - Local Linear Embedding (Chart-based):
+  // Partition the d-dimensional manifold into K local regions, each with a linear projection
+  assignments = cluster(X, K)          // assign inputs to K local regions
+  for k in range(K):
+    z_k = W_k @ X[assignments==k] + b_k  // local linear projection
+  // Equivalent to MoE: K "chart experts" each responsible for one manifold patch
+  z = sum_k g_k(x) * (W_k @ x + b_k)   // g_k is the chart assignment weight
+
+Method 2 - Geodesic Preservation Loss (global structure preservation):
+  // Preserve geodesic distances from high-dimensional space in the low-dimensional representation
+  D_high = geodesic_distance(X, k_nn=10)   // shortest paths on k-NN graph
+  D_low = pairwise_distance(Z)              // Euclidean distances in low-dim representation
+  L_geo = ||D_high - D_low||_F^2 / N^2     // Sammon mapping
+  // Or t-SNE-style KL divergence:
+  p_ij = exp(-D_high^2 / (2 sigma^2)) / sum  // high-dim affinity
+  q_ij = 1 / (1 + D_low^2) / sum             // low-dim t-distribution affinity
+  L_tsne = KL(P || Q)
+
+Method 3 - Riemannian Optimization (optimize directly on the manifold):
+  // Parameters constrained to Stiefel/Grassmann manifold
+  W in St(d, r) i.e. W^T W = I_r          // orthogonality constraint
+  // Riemannian SGD:
+  grad_euclidean = nabla f(W)
+  grad_riemannian = grad_euclidean - W @ (W^T @ grad_euclidean)  // project to tangent space
+  W = retract(W, -lr * grad_riemannian)    // retraction mapping back to manifold
+  // retract can be implemented via QR decomposition or Cayley transform
+```
+
+## Implementable Structures
+- **Chart MoE**: K local linear projections + softmax gating => natural integration with the MoE framework
+- **Manifold regularization**: L_manifold = tr(Z^T L Z) / N^2, where L is the graph Laplacian and Z is the representation
+  Encourages nearby samples to have similar representations
+- **Intrinsic dimension estimation**: Use MLE or two-norm methods to estimate the effective dimensionality d* of the data
+- **Adaptive d**: Local dimensionality varies across regions; estimate locally via PCA
+
+## GPU Feasibility
+- **Tensorization**: Local linear projections are GEMM (d x D) @ (D x N); graph Laplacian regularization is SpMM
+- **GEMM-mappable**: K linear projections in Chart MoE form a batched GEMM (K x d x D) @ (D x N)
+- **Complexity**: k-NN construction O(N * D * log N) requires FAISS; manifold regularization O(N^2) requires sampling approximation
+- **Memory & KV-Cache**: K chart parameters K * d * D typically < 10 MB; k-NN graph N * k * 4 bytes
+- **Low-precision stability**: Distance computations and exp in fp16 require attention to numerical range; Riemannian retract recommended in fp32
+- **Parallelism & Communication**: K charts computed independently, perfectly parallel; k-NN search accelerated with FAISS GPU
+- **Sparse structure**: k-NN graph is naturally sparse; manifold regularization L is a sparse matrix, enabling SpMM acceleration
+- **Operator fusion**: Matmul + bias + activation within a chart can be fused; gating softmax + weighted-sum can be fused
+
+## Paper-Worthy Formulation
+"Based on the manifold hypothesis, we constrain D-dimensional token representations to a d*-dimensional (intrinsic dimension estimate) submanifold, achieving piecewise-linear approximation via K local coordinate charts (Chart MoE) and preserving global geodesic structure through graph Laplacian manifold regularization, with theoretical guarantees that embedding error converges at rate O(N^{-2/d*})."
+
+## Risks
+- Inaccurate intrinsic dimension d* estimation leads to over-compression or dimension waste
+- k-NN graph construction is computationally expensive at large scale, requiring sampling or approximation
+- The N^2 complexity of manifold regularization limits batch size, necessitating mini-batch sampling
+- Discontinuities at local chart boundaries require overlapping regions and smooth transitions

package/design-patterns/representation/manifold-representation.md

@@ -0,0 +1,87 @@
+# Manifold Representation（流形表示）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入数据虽然在高维空间中，但实际分布在低维流形上时使用。典型场景：
+(1) 自然语言 token 的语义空间——虽维度 d=4096，但有效自由度远小于 d；
+(2) 多模态对齐——文本和图像分布在不同的流形上，需要对齐；
+(3) 专家特征空间——每个专家处理流形的不同局部区域；
+(4) 降维/压缩——利用流形低维本质减少参数量。
+核心诉求：**利用数据的低维流形结构，提高表示效率和泛化能力**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（黎曼几何、测地线、曲率）、lenses/probabilistic.md（内在维度）
+- 知识：knowledge-base/differential-geometry/manifold.md（流形、切空间、指数映射）、
+  knowledge-base/matrix-analysis/projection.md（SVD、低秩近似、PCA）
+
+## 需要的数学知识
+- **流形假设**：数据 x ∈ R^D 实际分布在 d << D 维的光滑流形 M 上
+  局部可用切空间 T_pM ≈ R^d 线性近似
+- **局部坐标卡 (Chart)**：φ: U ⊂ M → R^d，将流形局部映射到低维欧氏空间
+  多个坐标卡 {φ_i} 构成图册（Atlas），覆盖整个流形
+- **测地线距离**：d_M(p,q) = inf ∫‖γ'(t)‖dt，流形上两点间最短路径
+  近似计算：k-NN 图上 Dijkstra/Isomap
+- **指数映射/对数映射**：
+  exp_p: T_pM → M（切空间到流形），log_p: M → T_pM（流形到切空间）
+  用于在切空间中做线性运算后映射回流形
+
+## AI 模块形式
+```
+模块：ManifoldRepresentation
+输入：X ∈ R^{N×D}（高维输入），目标流形维度 d << D
+
+方法1 - 局部线性嵌入（Chart-based）：
+  // 将 d 维流形分成 K 个局部区域，每个区域用线性投影
+  assignments = cluster(X, K)         // 将输入分配到 K 个局部区域
+  for k in range(K):
+    z_k = W_k @ X[assignments==k] + b_k  // 局部线性投影 d×D → d×d
+  // 等效为 MoE：K 个"chart expert"各负责流形一个局部
+  z = Σ_k g_k(x) · (W_k @ x + b_k)   // g_k 为 chart 分配权重
+
+方法2 - 测地线保持损失（全局结构保持）：
+  // 保持高维空间的测地线距离在低维表示中不变
+  D_high = geodesic_distance(X, k_nn=10)   // k-NN 图上最短路
+  D_low = pairwise_distance(Z)              // 低维表示的欧氏距离
+  L_geo = ‖D_high - D_low‖_F² / N²         // Sammon mapping
+  // 或用 t-SNE 式 KL 散度：
+  p_ij = exp(-D_high²/2σ²) / Σ  // 高维亲和度
+  q_ij = 1/(1+D_low²) / Σ        // 低维 t 分布亲和度
+  L_tsne = KL(P ‖ Q)
+
+方法3 - 黎曼优化（直接在流形上优化）：
+  // 参数约束在 Stiefel/Grassmann 流形上
+  W ∈ St(d, r)  i.e. W^T W = I_r           // 正交约束
+  // Riemannian SGD：
+  grad_euclidean = ∇f(W)
+  grad_riemannian = grad_euclidean - W @ (W^T @ grad_euclidean)  // 投影到切空间
+  W = retract(W, -lr · grad_riemannian)     // 缩回映射到流形
+  // retract 可用 QR 分解或 Cayley 变换实现
+```
+
+## 可实现结构
+- **Chart MoE**：K 个局部线性投影 + softmax 门控 → 天然与 MoE 框架集成
+- **流形正则化**：L_manifold = tr(Z^T L Z) / N²，L 为图拉普拉斯，Z 为表示
+  鼓励相邻样本的表示相近
+- **内在维度估计**：用 MLE 或 two-norm 方法估计数据的有效维度 d*
+- **自适应 d**：不同区域的局部维度不同，用 PCA 局部估计
+
+## GPU 可行性
+- **张量化**：局部线性投影为 GEMM (d×D)@(D×N)；图拉普拉斯正则为 SpMM
+- **GEMM 可映射**：Chart MoE 的 K 个线性投影为 batched GEMM (K×d×D)@(D×N)
+- **复杂度**：k-NN 构建 O(N·D·log N) 需 FAISS；流形正则 O(N²) 需采样近似
+- **显存与 KV-Cache**：K 个 chart 参数 K·d·D 通常 <10MB；k-NN 图 N·k·4 bytes
+- **低精度稳定**：距离计算和 exp 在 fp16 下需注意数值范围；黎曼 retract 建议 fp32
+- **并行与通信**：K 个 chart 独立计算，完美并行；k-NN 搜索用 FAISS GPU 加速
+- **稀疏结构**：k-NN 图天然稀疏，流形正则 L 为稀疏矩阵，SpMM 加速
+- **算子融合**：chart 内的 matmul+bias+activation 可融合；门控 softmax+weighted-sum 可融合
+
+## 论文表述方式
+"基于流形假设将 D 维 token 表示约束到 d*（内在维度估计）维子流形上，
+通过 K 个局部坐标卡（Chart MoE）实现分段线性近似，配合图拉普拉斯流形正则
+保持全局测地线结构，理论上保证嵌入误差以 O(N^{-2/d*}) 收敛。"
+
+## 风险
+- 内在维度 d* 估计不准确导致过度压缩或维度浪费
+- k-NN 图构建在大规模数据下计算代价高，需采样或近似
+- 流形正则的 N² 复杂度限制 batch size，需 mini-batch 采样
+- 局部 chart 边界处表示不连续，需重叠区域和平滑过渡

package/design-patterns/representation/shared-private-decomposition.en.md

@@ -0,0 +1,84 @@
+# Shared-Private Decomposition
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use in multi-task/multi-domain learning when representations need to be decomposed into a "cross-task common component" and a "task-specific component."
+Typical scenarios: (1) Multi-task MoE -- Shared experts handle linguistic commonalities, Private experts handle task-specific logic;
+(2) Multi-domain adaptation -- Shared representations capture general semantics, Private representations capture domain terminology;
+(3) Continual learning -- Shared retains stable knowledge, Private accommodates new knowledge without interfering with old knowledge.
+Core requirement: **explicitly separate commonality from individuality to prevent negative transfer and catastrophic forgetting**.
+
+## Mathematical Inspiration
+- Lenses: lenses/projection.md (subspace decomposition, direct sum decomposition), lenses/probabilistic.md (information decomposition)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (direct sum decomposition V = U + W, projection operators),
+  knowledge-base/probability/kl-divergence.md (information decomposition: shared/synergy/unique)
+
+## Required Mathematical Background
+- **Direct Sum Decomposition**: R^d = S + P, where S intersect P = {0}, and every x = x_S + x_P is unique
+  Projection matrices P_S + P_P = I, P_S * P_P = 0
+- **Information Decomposition (Williams & Beer PID)**:
+  I(X; Y_1, Y_2) = Shared + Unique_1 + Unique_2 + Synergy
+  Shared = the redundant information component min(I(X; Y_1), I(X; Y_2))
+- **Low-Rank + Sparse Decomposition (RPCA)**: M = L + S, where L is low-rank (common) + S is sparse (specific)
+  Solved via nuclear norm + L1 norm convex relaxation
+- **CCA (Canonical Correlation Analysis)**: max corr(W_1^T X, W_2^T Y), extracting shared variation between two sets of variables
+
+## AI Module Form
+```
+Module: SharedPrivateDecomposer
+Input: X in R^{N x d}, task identifier t in {1, ..., T}
+
+Method 1 - Additive Decomposition (most common):
+  z_shared = E_shared(X)           // shared encoder: MLP or Transformer block
+  z_private = E_private[t](X)      // private encoder: independent parameters per task
+  z = z_shared + z_private         // additive fusion
+  // Training objective: L_task(z, y) + lambda_1 * OrthLoss(z_shared, z_private)
+  // Orthogonality ensures shared and private learn different information
+
+Method 2 - Gated Decomposition (dynamic weighting):
+  z_shared = E_shared(X)
+  z_private = E_private[t](X)
+  gate = sigmoid(Linear(z_shared + z_private))  // dynamic fusion gate
+  z = gate * z_shared + (1 - gate) * z_private
+  // Gating allows per-dimension selection of shared/private contribution ratios
+
+Method 3 - Adversarial Decomposition (information-theoretic guarantee):
+  z_shared = E_shared(X)
+  z_private = E_private[t](X)
+  // Shared should be indistinguishable across tasks (adversarial gradient):
+  task_pred = classifier(z_shared.flip_gradient())
+  L_adv = -CE(task_pred, t)        // Shared contains no task information
+  // Private should be discriminative across tasks:
+  L_private = CE(classifier(z_private), t)
+  L = L_task + lambda_adv * L_adv + lambda_priv * L_private
+
+Dimension Allocation Principle:
+  d_shared = d * T / (T + 1)      // With T tasks, shared occupies the majority
+  d_private = d * 1 / (T + 1)     // Each private occupies a smaller portion
+  // Or dynamically allocate based on PCA variance explained ratio
+```
+
+## Implementable Structures
+- **Dual encoder + fusion layer**: shared_encoder (large) + T private_encoders (small) + fusion
+- **Parameter efficiency**: Private uses LoRA (Low-Rank Adaptation) instead of full encoders, O(d * r) parameters per task
+- **Dynamic routing integration**: Shared experts + private experts selected via MoE routing
+- **Progressive expansion**: For new tasks, only add private encoders with frozen shared parameters
+
+## GPU Feasibility
+- **Tensorization**: Two encoder forward passes are independent GEMM chains, executable in parallel
+- **GEMM-mappable**: Shared/private encoders are each standard Transformer FFNs (2x GEMM)
+- **Complexity**: Shared O(N * d^2) + T private encoders O(N * d^2 / T), total approximately 2x a single encoder
+- **Memory & KV-Cache**: All T private encoder parameters stored; LoRA compression needed when T is large
+- **Low-precision stability**: Additive/gated fusion is safe in fp16; adversarial training gradient reversal requires fp32
+- **Parallelism & Communication**: Shared and private encoders can be assigned to different GPUs; multi-task batches mixed for training
+- **Sparse structure**: Private encoders can be sparsified (only the current task's is activated); only 1 out of T activated
+- **Operator fusion**: Additive fusion is trivial; gated fusion sigmoid -> multiply -> add can be fused
+
+## Paper-Worthy Formulation
+"We decompose the multi-task representation space R^d into a direct sum S + P. The Shared subspace ensures task-invariance through adversarial training (H(T|Z_s) -> log T), while the Private subspace guarantees information complementarity with Shared via orthogonal regularization. Theoretical analysis shows that negative transfer decays as O(||P_S * P_P||_F) with increasing orthogonality."
+
+## Risks
+- Min-max variational in adversarial training is unstable; the gradient reversal scale and lambda_adv require careful tuning
+- Over-compression of Shared leads to insufficient common information, placing excessive burden on Private
+- Total Private parameters grow linearly with T, requiring LoRA or adapter modules to control
+- When task similarity is low, Shared may learn a vacuous "common component"

package/design-patterns/representation/shared-private-decomposition.md

@@ -0,0 +1,86 @@
+# Shared-Private Decomposition（共享-私有分解）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+多任务/多领域学习中，需要将表示分解为"跨任务共性部分"和"任务特异部分"。
+典型场景：(1) 多任务 MoE——Shared 专家处理语言共性，Private 专家处理任务特异逻辑；
+(2) 多领域适配——Shared 表示捕获通用语义，Private 表示捕获领域术语；
+(3) 持续学习——Shared 保留稳定知识，Private 容纳新知识而不干扰旧知识。
+核心诉求：**显式分离共性与个性，避免负迁移和灾难性遗忘**。
+
+## 数学思想来源
+- 透镜：lenses/projection.md（子空间分解、直和分解）、lenses/probabilistic.md（信息分解）
+- 知识：knowledge-base/matrix-analysis/projection.md（直和分解 V = U ⊕ W、投影算子）、
+  knowledge-base/probability/kl-divergence.md（信息分解：shared/synergy/unique）
+
+## 需要的数学知识
+- **直和分解**：R^d = S ⊕ P，其中 S ∩ P = {0}，每个 x = x_S + x_P 唯一
+  投影矩阵 P_S + P_P = I，P_S · P_P = 0
+- **信息分解 (Williams & Beer PID)**：
+  I(X;Y₁,Y₂) = Shared + Unique₁ + Unique₂ + Synergy
+  Shared = min(I(X;Y₁), I(X;Y₂)) 的冗余信息部分
+- **低秩+稀疏分解 (RPCA)**：M = L + S，L 低秩（共性）+ S 稀疏（特异）
+  通过核范数 + L1 范数凸松弛求解
+- **CCA (典型相关分析)**：max corr(W₁^T X, W₂^T Y)，提取两组变量的共享变异
+
+## AI 模块形式
+```
+模块：SharedPrivateDecomposer
+输入：X ∈ R^{N×d}，任务标识 t ∈ {1,...,T}
+
+方法1 - 加法分解（最常用）：
+  z_shared = E_shared(X)          // 共享编码器：MLP or Transformer block
+  z_private = E_private[t](X)     // 私有编码器：每个任务独立参数
+  z = z_shared + z_private        // 加法融合
+  // 训练目标：L_task(z, y) + λ₁·OrthLoss(z_shared, z_private)
+  // 正交性确保 shared 和 private 学到不同的东西
+
+方法2 - 门控分解（动态权重）：
+  z_shared = E_shared(X)
+  z_private = E_private[t](X)
+  gate = sigmoid(Linear(z_shared ⊕ z_private))  // 动态融合门
+  z = gate ⊙ z_shared + (1 - gate) ⊙ z_private
+  // 门控允许逐维度选择 shared/private 的贡献比例
+
+方法3 - 对抗分解（信息论保证）：
+  z_shared = E_shared(X)
+  z_private = E_private[t](X)
+  // Shared 应无法区分任务（对抗梯度）：
+  task_pred = classifier(z_shared.flip_gradient())
+  L_adv = -CE(task_pred, t)       // Shared 不含任务信息
+  // Private 应能区分任务：
+  L_private = CE(classifier(z_private), t)
+  L = L_task + λ_adv·L_adv + λ_priv·L_private
+
+维度分配原则：
+  d_shared = d · T/(T+1)          // T 个任务时 shared 占大部分
+  d_private = d · 1/(T+1)         // 每个 private 占较小部分
+  // 或用 PCA 变异解释率动态分配
+```
+
+## 可实现结构
+- **双编码器 + 融合层**：shared_encoder (大) + T 个 private_encoder (小) + fusion
+- **参数效率**：private 用 LoRA（低秩适配）而非完整编码器，参数 O(d·r) per task
+- **动态路由集成**：shared 专家 + private 专家通过 MoE 路由选择
+- **渐进扩展**：新任务时只增加 private 编码器，frozen shared 参数
+
+## GPU 可行性
+- **张量化**：两个编码器前向为独立 GEMM 链，可并行执行
+- **GEMM 可映射**：shared/private 编码器各为标准 Transformer FFN（2×GEMM）
+- **复杂度**：shared O(N·d²) + T 个 private O(N·d²/T)，总 ≈ 2× 单编码器
+- **显存与 KV-Cache**：T 个 private 编码器参数全存储，T 大时需 LoRA 压缩
+- **低精度稳定**：加法/门控融合在 fp16 安全；对抗训练的梯度反转需 fp32
+- **并行与通信**：shared 和 private 编码器可分配到不同 GPU；多任务 batch 混合训练
+- **稀疏结构**：private 编码器可稀疏化（仅当前任务激活），T 个中仅激活 1 个
+- **算子融合**：加法融合 trivial；门控融合的 sigmoid→multiply→add 可融合
+
+## 论文表述方式
+"将多任务表示空间 R^d 分解为直和 S ⊕ P，Shared 子空间通过对抗训练确保任务无关性
+（H(T|Z_s)→log T），Private 子空间通过正交正则保证与 Shared 的信息互补，
+理论分析表明负迁移随正交性以 O(‖P_S·P_P‖_F) 衰减。"
+
+## 风险
+- 对抗训练的 min-max 优化不稳定，梯度反转的 scale 和 λ_adv 需精心调节
+- Shared 过度压缩导致共性信息不足，Private 负担过重
+- T 增大时 Private 参数总量线性增长，需要 LoRA 或 adapter 控制
+- 任务相似度低时 Shared 可能学到空洞的"公共部分"

package/design-patterns/representation/subspace-alignment.en.md

@@ -0,0 +1,90 @@
+# Subspace Alignment
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+Use when two or more representation spaces need to be aligned to a common subspace. Typical scenarios:
+(1) Multi-modal alignment -- aligning text and image representations to a shared semantic subspace;
+(2) Cross-layer alignment -- aligning shallow features to the subspace of deep features for residual connections or distillation;
+(3) Expert output alignment -- different experts produce outputs with different dimensions/distributions that must be aligned before fusion;
+(4) Domain adaptation -- source and target domain feature distributions differ, requiring subspace alignment.
+Core requirement: **find the optimal linear/nonlinear mapping between two spaces such that corresponding semantics are aligned**.
+
+## Mathematical Inspiration
+- Lenses: lenses/geometric.md (Grassmann manifold, principal angles), lenses/variational.md (Procrustes problem)
+- Knowledge: knowledge-base/matrix-analysis/projection.md (SVD, orthogonal Procrustes),
+  knowledge-base/differential-geometry/manifold.md (Grassmann distance, geodesics)
+
+## Required Mathematical Background
+- **Orthogonal Procrustes Problem**: min_{W in O(d)} ||AW - B||_F^2
+  Closed-form solution: W* = UV^T, where USV^T = SVD(A^T B)
+- **CCA (Canonical Correlation Analysis)**: max_{W_1, W_2} tr(W_1^T Sigma_{XY} W_2) s.t. W_1^T Sigma_{XX} W_1 = I
+  Solution via generalized eigenvalue problem or SVD(Sigma_{XX}^{-1/2} Sigma_{XY} Sigma_{YY}^{-1/2})
+- **Distance on the Grassmann Manifold**: Gr(d, r) = {r-dimensional subspaces of R^d}
+  Distance between two points (subspaces) determined by principal angles theta_i: d_G(U, V) = sqrt(sum theta_i^2)
+  cos(theta_i) = sigma_i(U^T V) (singular values of U^T V)
+- **Subspace Tracking**: Online update of subspace basis
+  Oja's rule: W_{t+1} = W_t + eta * (x_t x_t^T W_t - W_t diag(W_t^T x_t x_t^T W_t))
+
+## AI Module Form
+```
+Module: SubspaceAligner
+Input: Source representation A in R^{N x d_a}, target representation B in R^{N x d_b} (paired data)
+
+Method 1 - Linear Procrustes Alignment (most common):
+  // Find optimal linear duality W such that A @ W is close to B
+  M = A^T @ B                         // d_a x d_b, single GEMM
+  U, S, V^T = SVD(M)                  // singular value decomposition
+  W* = U @ V^T                        // d_a x d_b optimal orthogonal duality
+  // Differentiable version: parameterize W as nn.Parameter, optimize with SGD
+  L_align = ||A @ W - B||_F^2 / N    // alignment loss
+
+Method 2 - Deep CCA (nonlinear subspace alignment):
+  f_A = MLP_A(A)                      // R^{d_a} -> R^r (nonlinear projection)
+  f_B = MLP_B(B)                      // R^{d_b} -> R^r
+  Sigma_AA = f_A^T @ f_A / N + lambda * I  // r x r auto-covariance
+  Sigma_BB = f_B^T @ f_B / N + lambda * I
+  Sigma_AB = f_A^T @ f_B / N         // r x r cross-covariance
+  T = Sigma_AA^{-1/2} @ Sigma_AB @ Sigma_BB^{-1/2}  // whitened cross-correlation
+  L_cca = -tr(SVD(T).S[:k])          // maximize sum of top k canonical correlations
+  // Equivalent to minimizing Grassmann distance
+
+Method 3 - Subspace Angle Regularization (multi-expert output alignment):
+  // Outputs of multiple experts should be aligned to the same subspace
+  U_k = SVD(expert_k_output)[0][:, :r]  // r-dim principal components of each expert output
+  for i, j in expert_pairs:
+    cos_angles = SVD(U_i^T @ U_j).S     // cosines of principal angles
+    L_subspace = -mean(cos_angles)       // maximize principal angle cosines -> minimize angles
+  // Or use Grassmann distance:
+  L_grass = sum(angles^2)                // angles = arccos(cos_angles)
+
+Online Subspace Tracking (adaptation at inference):
+  // New samples arrive at test time; incrementally update alignment matrix
+  W_new = W_old + eta * (x_new @ (y_new^T - x_new^T @ W_old))  // Oja-like
+  // No need to recompute SVD; O(d^2) online update
+```
+
+## Implementable Structures
+- **Alignment Layer**: nn.Linear(d_a, d_b, bias=False) initialized with the Procrustes solution
+- **Whitening layer**: Use running mean/variance for online whitening, avoiding per-step Sigma^{-1/2} computation
+- **CCA alternative**: Use Barlow Twins-style redundancy reduction loss instead of CCA (avoids matrix inversion)
+  L_BT = ||C - I||_F^2 where C = corr(f_A, f_B)
+- **Chunked SVD**: Use randomized SVD approximation at large scale; sufficient precision and faster
+
+## GPU Feasibility
+- **Tensorization**: A^T @ B is standard GEMM (d x N) @ (N x d); SVD has cuSOLVER implementations
+- **GEMM-mappable**: Procrustes core requires 1 GEMM + 1 SVD; CCA requires 2 GEMMs + 1 SVD
+- **Complexity**: GEMM O(N * d^2); SVD O(d^3) (d typically < 1024, acceptable); online tracking O(d^2)
+- **Memory & KV-Cache**: Storing covariance matrix d x d (~4 MB for d = 1024); no KV-Cache overhead
+- **Low-precision stability**: SVD strongly recommended in fp32; whitening matrix inversion requires fp32 + epsilon regularization
+- **Parallelism & Communication**: Subspace angle computations for multiple expert pairs are independent and parallel; CCA GEMMs are highly parallel
+- **Sparse structure**: When source/target representations are sparse, covariance matrix Sigma is sparse, enabling sparse SVD
+- **Operator fusion**: Whitening (mean-sub -> cov -> inv_sqrt -> transform) can be partially fused
+
+## Paper-Worthy Formulation
+"Based on orthogonal Procrustes theory, we obtain the optimal isometric mapping W* = UV^T (where USV^T = SVD(A^T B)) between source and target representations. Extending this to Deep CCA for nonlinear subspace alignment, principal angle analysis on the Grassmann manifold shows that post-alignment subspace distance converges at rate O(r/sqrt(N)), while the Barlow Twins redundancy reduction objective ensures feature dimension decorrelation."
+
+## Risks
+- SVD during backpropagation produces undefined gradients when singular values coincide; epsilon regularization is needed
+- CCA whitening requires matrix inversion; numerically unstable when the covariance matrix is near-singular
+- Nonlinear CCA (Deep CCA) may overfit, particularly on small datasets
+- Online subspace tracking learning rate eta requires a decay schedule to prevent continuous drift

package/design-patterns/representation/subspace-alignment.md

@@ -0,0 +1,92 @@
+# Subspace Alignment（子空间对齐）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当两个或多个表示空间需要对齐到共同的子空间时使用。典型场景：
+(1) 多模态对齐——将文本和图像的表示对齐到共享语义子空间；
+(2) 跨层对齐——将浅层特征对齐到深层特征的子空间，便于残差连接或蒸馏；
+(3) 专家输出对齐——不同专家的输出维度/分布不同，需要对齐后融合；
+(4) 领域适配——源域和目标域的特征分布不同，需要对齐子空间。
+核心诉求：**找到两个空间之间的最优线性/非线性映射，使对应语义对齐**。
+
+## 数学思想来源
+- 透镜：lenses/geometric.md（Grassmann 流形、主角度）、lenses/variational.md（Procrustes 问题）
+- 知识：knowledge-base/matrix-analysis/projection.md（SVD、正交 Procrustes）、
+  knowledge-base/differential-geometry/manifold.md（Grassmann 距离、测地线）
+
+## 需要的数学知识
+- **正交 Procrustes 问题**：min_{W∈O(d)} ‖AW - B‖_F²
+  闭合解：W* = UV^T，其中 USV^T = SVD(A^T B)
+- **CCA (典型相关分析)**：max_W₁,W₂ tr(W₁^T Σ_XY W₂) s.t. W₁^T Σ_XX W₁ = I
+  解为广义特征值问题或 SVD(Σ_XX^{-1/2} Σ_XY Σ_YY^{-1/2})
+- **Grassmann 流形上的距离**：Gr(d, r) = {r 维子空间 ⊂ R^d}
+  两点（子空间）的距离由主角度 θ_i 决定：d_G(U,V) = √(Σ θ_i²)
+  cos(θ_i) = σ_i(U^T V)（U^T V 的奇异值）
+- **子空间追踪 (Subspace Tracking)**：在线更新子空间基
+  Oja 规则：W_{t+1} = W_t + η(x_t x_t^T W_t - W_t diag(W_t^T x_t x_t^T W_t))
+
+## AI 模块形式
+```
+模块：SubspaceAligner
+输入：源表示 A ∈ R^{N×d_a}，目标表示 B ∈ R^{N×d_b}（配对数据）
+
+方法1 - 线性 Procrustes 对齐（最常用）：
+  // 找最优线性变换 W 使 A@W ≈ B
+  M = A^T @ B                       // d_a × d_b，一次 GEMM
+  U, S, V^T = SVD(M)                // 奇异值分解
+  W* = U @ V^T                      // d_a × d_b 最优正交变换
+  // 可微版本：将 W 参数化为 nn.Parameter，用 SGD 优化
+  L_align = ‖A @ W - B‖_F² / N     // 对齐损失
+
+方法2 - 深度 CCA（非线性子空间对齐）：
+  f_A = MLP_A(A)                    // R^{d_a} → R^r（非线性投影）
+  f_B = MLP_B(B)                    // R^{d_b} → R^r
+  Σ_AA = f_A^T @ f_A / N + λI       // r×r 自协方差
+  Σ_BB = f_B^T @ f_B / N + λI
+  Σ_AB = f_A^T @ f_B / N            // r×r 互协方差
+  T = Σ_AA^{-1/2} @ Σ_AB @ Σ_BB^{-1/2}  // 白化互相关
+  L_cca = -tr(SVD(T).S[:k])         // 最大化前 k 个典型相关的和
+  // 等效于最小化 Grassmann 距离
+
+方法3 - 子空间角度正则（多专家输出对齐）：
+  // 多个专家的输出应对齐到同一子空间
+  U_k = SVD(expert_k_output)[0][:, :r]  // 各专家输出的 r 维主成分
+  for i, j in expert_pairs:
+    cos_angles = SVD(U_i^T @ U_j).S    // 主角度的余弦
+    L_subspace = -mean(cos_angles)       // 最大化主角度余弦 → 最小化角度
+  // 或使用 Grassmann 距离：
+  L_grass = sum(angles²)                 // angles = arccos(cos_angles)
+
+在线子空间追踪（推理时适配）：
+  // 测试时新样本到来，增量更新对齐矩阵
+  W_new = W_old + η · (x_new @ (y_new^T - x_new^T @ W_old))  // Oja-like
+  // 无需重新 SVD，O(d²) 在线更新
+```
+
+## 可实现结构
+- **对齐层 (AlignmentLayer)**：nn.Linear(d_a, d_b, bias=False) 初始化为 Procrustes 解
+- **白化层**：用 running mean/variance 做在线白化，避免每步计算 Σ^{-1/2}
+- **CCA 替代方案**：用 Barlow Twins 式的冗余减少损失替代 CCA（避免矩阵逆）
+  L_BT = ‖C - I‖_F² 其中 C = corr(f_A, f_B)
+- **分块 SVD**：大规模时用 randomized SVD 近似，精度足够且更快
+
+## GPU 可行性
+- **张量化**：A^T@B 为标准 GEMM (d×N)@(N×d)；SVD 有 cuSOLVER 实现
+- **GEMM 可映射**：Procrustes 核心 1 次 GEMM + 1 次 SVD；CCA 2 次 GEMM + 1 次 SVD
+- **复杂度**：GEMM O(N·d²)；SVD O(d³)（d 通常 <1024，可接受）；在线追踪 O(d²)
+- **显存与 KV-Cache**：存储协方差矩阵 d×d（~4MB for d=1024），不增加 KV-Cache
+- **低精度稳定**：SVD 强烈建议 fp32；白化的矩阵逆需 fp32 + ε 正则化
+- **并行与通信**：多专家对的子空间角度计算独立并行；CCA 的 GEMM 高度并行
+- **稀疏结构**：当源/目标表示稀疏时，协方差矩阵 Σ 稀疏，可用稀疏 SVD
+- **算子融合**：白化 (mean-sub → cov → inv_sqrt → transform) 可部分融合
+
+## 论文表述方式
+"基于正交 Procrustes 理论求得源-目标表示间的最优等距映射 W*=UV^T（USV^T=SVD(A^TB)），
+将其扩展为深度 CCA 实现非线性子空间对齐，Grassmann 流形上的主角度分析表明
+对齐后子空间距离以 O(r/√N) 收敛，同时 Barlow Twins 冗余减少目标保证特征维度解耦。"
+
+## 风险
+- SVD 在反向传播时奇异值重合导致梯度未定义，需添加 ε 正则化
+- CCA 的白化步骤需要矩阵逆，协方差矩阵近奇异时数值不稳定
+- 非线性 CCA（深度 CCA）可能过拟合，特别是在小数据集上
+- 在线子空间追踪的学习率 η 需要衰减调度，否则持续漂移