math-skill 2.1.0 → 3.0.0

package/design-patterns/attention/equivariant-attention.en.md

@@ -0,0 +1,72 @@
+# Equivariant Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the input possesses an **explicit symmetry group $G$ action** (rotation, translation, permutation, reflection, etc.), and the desired model output should be **equivariant** (covariant) rather than invariant under the same transformations, equivariant constraints must be directly encoded into the attention mechanism. Typical scenarios include: 3D point clouds / molecules ($E(3)$ rigid-body group), image classification ($D_n$ rotation/reflection group), set data ($S_n$ permutation group), and multi-view / multi-sensor fusion.
+
+## Mathematical Inspiration
+- Lenses: [symmetry, categorical (unified framework for group actions)]
+- Knowledge: [`../../knowledge-base/probability/concentration-inequality.md` (sample efficiency gains under equivariant constraints -- data equivalence along orbits), `../../knowledge-base/probability/entropy.md` (equivariant constraints reduce output distribution entropy, yielding stronger inductive bias)]
+
+## Required Mathematical Knowledge
+- **Group Representation Theory Basics**: Linear representation of a group $G$, $\rho: G \to GL(V)$, and irreducible representation decomposition
+- **Equivariant Map Definition**: $f(g \cdot x) = \rho_{\text{out}}(g) \cdot f(x)$ for all $g \in G$
+- **Orbit-Stabilizer Theorem** (see `references/books/abstract-algebra.md` Ch.5): $|orbit| = |G|/|stab|$, giving the parameter sharing multiplier
+- **Schur's Lemma**: An equivariant linear map between irreducible representations is either zero or a scalar multiple
+
+## AI Module Form
+
+**Core Idea**: Replace the standard attention $Q, K, V$ with **steerable features**, ensuring that attention weights are invariant under group actions and that outputs are equivariant under group actions.
+
+**Scheme A: Permutation-Equivariant Attention ($S_n$ group, set data)**:
+```python
+# DeepSets / Set Transformer style
+# Attention weights are permutation-invariant: pi(Q)pi(K)^T = QK^T (permutations cancel)
+# Output is permutation-equivariant: pi(softmax(QK^T) V) = softmax(QK^T) pi(V)
+Q, K, V = W_q(X), W_k(X), W_v(X)  # pointwise linear transform
+scores = Q @ K.T / sqrt(d)         # permutation-invariant
+attn = softmax(scores)             # permutation-invariant
+output = attn @ V                  # permutation-equivariant (V is equivariant)
+```
+
+**Scheme B: $E(3)$-Equivariant Attention (3D point clouds / molecules)**:
+```python
+# Decompose features into scalars + vectors + higher-order tensors (spherical harmonic basis)
+# Attention weights computed using only scalar features (rotation-invariant)
+scalar_Q = scalar_proj(X_scalar)   # scalars only -> rotation-invariant
+scalar_K = scalar_proj(X_scalar)
+scores = scalar_Q @ scalar_K.T / sqrt(d_s)  # rotation-invariant attention weights
+
+# V contains equivariant features (scalars + vectors), weighted by invariant weights
+output_scalar = softmax(scores) @ V_scalar   # scalar -> invariant
+output_vector = softmax(scores) @ V_vector   # vector -> equivariant (rotation-covariant)
+```
+
+**Scheme C: $D_n$-Equivariant Attention (image rotation / reflection)**:
+```python
+# G-CNN style: for each group element g in D_n, transform input with rho(g) then compute attention
+# Weights shared along orbits (same W_q/W_k/W_v), aggregate outputs over all group elements
+output = mean(softmax((rho(g)@X@W_q) @ (rho(g)@X@W_k).T/sqrt(d)) @ (rho(g)@X@W_v) for g in D_n)
+```
+
+## Implementable Architectures
+- **SE(3)-Transformer / Equiformer**: Spherical harmonic features + equivariant attention for molecular property prediction and protein structure
+- **Set Transformer**: $S_n$ permutation-equivariant attention + Induced Set Attention (low-rank inducing points for complexity reduction)
+- **G-CNN Attention**: $D_n$ rotation/reflection equivariance for remote sensing imagery and medical imaging
+
+## GPU Feasibility
+- **Dimension 1 Tensorization**: Group actions implemented as $\rho(g)$ matrix multiplications; equivariant features stored as batched tensors
+- **Dimension 2 GEMM-mappability**: $\rho(g) X$ and $Q K^T$ are both GEMM operations; $|G|$ group elements map to batched GEMM
+- **Dimension 3 Complexity**: $|G|$-fold computation overhead; acceptable for small groups ($|D_4|=8$), infeasible for large groups ($|S_n|=n!$). Remedy: use generators + Cayley graph propagation instead of full group enumeration
+- **Dimension 4 Memory**: Requires storing $|G|$ copies of intermediate features; mitigated by chunking + gradient checkpointing
+- **Dimension 5 Low Precision**: Orthogonal representation matrices are numerically stable under bf16
+- **Dimension 6 Parallelism**: $|G|$ group elements are naturally parallelizable (along the batch dimension)
+- **Dimension 7 Sparsity**: Permutation $\rho(g)$ is extremely sparse and can be encoded as gather indices
+- **Dimension 8 Operator Fusion**: Group action + linear duality can be fused into a single batched GEMM
+
+## Paper Phrasing
+"We propose an equivariant attention mechanism that constrains attention weights to be group invariants and attention outputs to be group equivariants, directly encoding the inductive bias of symmetry group $G$ into the model architecture without additional data augmentation, achieving a $|G|/|stab|$-fold improvement in parameter efficiency."
+
+## Risks
+- **Cost of Incorrect Group Selection**: If the data does not possess the assumed symmetry (e.g., molecules lacking full $E(3)$ symmetry), equivariant constraints will impair expressiveness. The symmetry assumption must be validated first, or "approximate equivariance" (soft equivariance) should be used.
+- **Memory Explosion of High-Order Representations**: The $L$-th order spherical harmonic representation of $SO(3)$ has dimension $(2L+1)^2$, causing storage and computation to grow rapidly for high-order features ($L \geq 3$). In practice, truncation to $L \leq 2$ is standard.

package/design-patterns/attention/equivariant-attention.md

@@ -0,0 +1,72 @@
+# 等变注意力 / Equivariant Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入具有**明确的对称群 $G$ 作用**（旋转、平移、置换、反射等），且期望模型输出在相同变换下**协变**（equivariant）而非不变时，需要将等变约束直接编入注意力机制。典型场景：3D 点云/分子（$E(3)$ 刚体群）、图像分类（$D_n$ 旋转/镜像群）、集合数据（$S_n$ 置换群）、多视角/多传感器融合。
+
+## 数学思想来源
+- 透镜：[symmetry（对称透镜）, categorical（范畴化透镜 — 群作用的统一框架）]
+- 知识：[`../../knowledge-base/probability/concentration-inequality.md`（等变约束下的样本效率提升 — 轨道上的数据等价性）, `../../knowledge-base/probability/entropy.md`（等变约束降低输出分布熵 → 更强的归纳偏置）]
+
+## 需要的数学知识
+- **群表示论基础**：群 $G$ 的线性表示 $\rho: G \to GL(V)$，不可约表示分解
+- **等变映射定义**：$f(g \cdot x) = \rho_{\text{out}}(g) \cdot f(x)$，对所有 $g \in G$
+- **轨道-稳定子定理**（参见 `references/books/abstract-algebra.md` Ch.5）：$|orbit| = |G|/|stab|$，参数共享倍率
+- **Schur 引理**：不可约表示间的等变线性映射要么为零要么为标量乘
+
+## AI 模块形式
+
+**核心思路**：将标准注意力的 $Q, K, V$ 替换为**等变特征**（steerable features），确保注意力权重在群作用下不变，输出在群作用下等变。
+
+**方案 A：置换等变注意力（$S_n$ 群，集合数据）**：
+```python
+# DeepSets / Set Transformer 风格
+# 注意力权重对置换不变：π(Q)π(K)^T = QK^T（置换抵消）
+# 输出对置换等变：π(softmax(QK^T) V) = softmax(QK^T) π(V)
+Q, K, V = W_q(X), W_k(X), W_v(X)  # 逐点线性变换
+scores = Q @ K.T / sqrt(d)         # 置换不变
+attn = softmax(scores)             # 置换不变
+output = attn @ V                  # 置换等变（V 是等变的）
+```
+
+**方案 B：$E(3)$ 等变注意力（3D 点云/分子）**：
+```python
+# 将特征分解为标量 + 向量 + 高阶张量（球谐基）
+# 注意力权重只用标量特征计算（旋转不变）
+scalar_Q = scalar_proj(X_scalar)   # 仅标量 → 旋转不变
+scalar_K = scalar_proj(X_scalar)
+scores = scalar_Q @ scalar_K.T / sqrt(d_s)  # 旋转不变的注意力权重
+
+# V 包含等变特征（标量 + 向量），用不变的权重加权
+output_scalar = softmax(scores) @ V_scalar   # 标量 → 不变
+output_vector = softmax(scores) @ V_vector   # 向量 → 等变（旋转协变）
+```
+
+**方案 C：$D_n$ 等变注意力（图像旋转/镜像）**：
+```python
+# G-CNN 风格：对每个群元素 g ∈ D_n，用 ρ(g) 变换输入后计算注意力
+# 权重沿轨道共享（同一组 W_q/W_k/W_v），聚合所有群元素输出
+output = mean(softmax((rho(g)@X@W_q) @ (rho(g)@X@W_k).T/sqrt(d)) @ (rho(g)@X@W_v) for g in D_n)
+```
+
+## 可实现结构
+- **SE(3)-Transformer / Equiformer**：球谐特征 + 等变注意力，用于分子性质预测和蛋白质结构
+- **Set Transformer**：$S_n$ 置换等变注意力 + Induced Set Attention（低秩诱导点降复杂度）
+- **G-CNN 注意力**：$D_n$ 旋转/镜像等变，用于遥感图像和医学影像
+
+## GPU 可行性
+- **维度 1 张量化 ✅**：群作用实现为 $\rho(g)$ 矩阵乘，等变特征为 batched 张量
+- **维度 2 GEMM 可映射 ✅**：$\rho(g) X$ 和 $Q K^T$ 均为 GEMM；$|G|$ 个群元素 → batched GEMM
+- **维度 3 复杂度 ⚠️**：$|G|$ 倍计算量，小群（$|D_4|=8$）可接受，大群（$|S_n|=n!$）不可。改造：用生成元 + Cayley 图传播替代整群枚举
+- **维度 4 显存 ⚠️**：需存储 $|G|$ 份中间特征，可分块 + gradient checkpointing
+- **维度 5 低精度 ✅**：正交表示矩阵在 bf16 下数值稳定
+- **维度 6 并行 ✅**：$|G|$ 个群元素天然并行（batch 维）
+- **维度 7 稀疏 ⚠️**：置换 $\rho(g)$ 极度稀疏，可编码为 gather 索引
+- **维度 8 算子融合 ✅**：群作用 + 线性变换可融合为单次 batched GEMM
+
+## 论文表述方式
+"我们提出等变注意力机制，通过将注意力权重约束为群不变量、注意力输出约束为群等变量，在不增加数据增强的情况下将对称群 $G$ 的归纳偏置直接编码进模型结构，参数效率提升 $|G|/|stab|$ 倍。"
+
+## 风险
+- **群选择错误的代价**：若数据不具有假定的对称性（如分子不具有完全 $E(3)$ 对称性），等变约束会伤害表达能力。需先验证对称性假设，或使用"近似等变"（soft equivariance）。
+- **高阶表示的显存爆炸**：$SO(3)$ 的 $L$ 阶球谐表示维度为 $(2L+1)^2$，高阶特征 ($L \geq 3$) 的存储和计算量急剧增长。实践中通常截断到 $L \leq 2$。

package/design-patterns/attention/geometry-aware-attention.en.md

@@ -0,0 +1,73 @@
+# Geometry-Aware Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When **known geometric relationships** exist between tokens/keys (spatial distance, manifold geodesic distance, hierarchical structure, temporal interval), the position-agnostic inner product of standard attention cannot exploit this structural information. Geometry-aware attention **directly injects geometric priors into attention weight computation**, enabling the model to naturally respect the metric structure of the underlying space. Typical scenarios include: 3D scene understanding, molecular conformation, temporal forecasting, hierarchical text structure (paragraph-sentence-word), and knowledge graphs.
+
+## Mathematical Inspiration
+- Lenses: [symmetry (metric invariance), duality (coordinate-system-independent representation)]
+- Knowledge: [`../../knowledge-base/information-geometry/fisher-metric.md` (geometric metrics on distribution spaces), `../../knowledge-base/probability/concentration-inequality.md` (concentration behavior under geometric constraints)]
+
+## Required Mathematical Knowledge
+- **Metric Spaces and Distance Functions**: Euclidean distance, geodesic distance, tree distance, Wasserstein distance
+- **RBF / Kernel Methods**: $k(x, y) = \exp(-d(x,y)^2 / 2\sigma^2)$, converting distance to similarity
+- **Geometric Interpretation of Positional Encoding**: RoPE = action of the rotation group $SO(2)$ (relative distance encoded as rotation angle difference); ALiBi = exponentially decaying distance bias
+
+## AI Module Form
+
+**Core Idea**: Explicitly introduce a geometric distance term into attention scores so that attention to distant tokens naturally decays:
+
+$$\text{GeoAttn}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}} + \text{GeoBias}(i, j)\right) V$$
+
+**Scheme A: Distance-Biased Attention (General Metric Spaces)**:
+```python
+# d_ij is the geometric distance between token i and j (precomputed or computed online)
+# Learnable distance bias function
+distance_bias = MLP(d_ij)  # or simply -alpha * d_ij
+scores = (Q @ K.T) / sqrt(d) + distance_bias
+attn = softmax(scores) @ V
+```
+
+**Scheme B: Relative Positional Encoding (Geometric Generalization of RoPE / ALiBi)**:
+```python
+# Generalize RoPE: position -> group element g_i, relative position -> g_i g_j^{-1}
+# Learnable relative position bias on arbitrary metric spaces
+def geo_attention(Q, K, V, positions):
+    rel_pos = pairwise_difference(positions)       # (n, n, coord_dim)
+    geo_bias = geo_encoder(rel_pos)               # (n, n) learnable mapping
+    scores = (Q @ K.T) / sqrt(d) + geo_bias
+    return softmax(scores) @ V
+```
+
+**Scheme C: Manifold-Aware Attention (Non-Euclidean Spaces)**:
+```python
+# When tokens reside on a non-Euclidean manifold (hyperbolic space, sphere),
+# use geodesic distance instead of Euclidean distance
+def manifold_attention(Q, K, V, manifold):
+    geodist = manifold.geodesic_distance_matrix(positions)  # (n, n)
+    geo_kernel = exp(-geodist^2 / (2 * sigma^2))  # RBF kernel
+    scores = (Q @ K.T) / sqrt(d) + log(geo_kernel + eps)
+    return softmax(scores) @ V
+```
+
+## Implementable Architectures
+- **RoPE Extensions**: From $SO(2)$ rotations to higher-dimensional rotation groups $SO(2k)$, encoding multi-dimensional positional information (2D image patches, 3D voxels)
+- **Hierarchical Positional Bias**: Use LCA (Lowest Common Ancestor) depth as distance in tree structures, suitable for hierarchical geometric of documents and code
+- **Molecular Conformation Attention**: 3D atomic coordinates -> distance matrix -> geometric bias, used in molecular GNNs and protein structure prediction
+
+## GPU Feasibility
+- **Dimension 1 Tensorization**: Distance matrices and bias matrices are dense tensors with element-wise operations
+- **Dimension 2 GEMM-mappability**: The main body $Q K^T$ is standard GEMM; geometric bias is additive and does not interrupt GEMM
+- **Dimension 3 Complexity**: Pairwise distance matrix construction and storage are $O(n^2)$; however, block-wise computation + online softmax can avoid materializing the full matrix
+- **Dimension 4 Memory**: The $n \times n$ distance matrix occupies GPU memory; mitigated by block/streaming computation (compatible with FlashAttention)
+- **Dimension 5 Low Precision**: Distance computation and bias addition are stable under bf16; the exp in RBF requires overflow caution (clamp distances)
+- **Dimension 6 Parallelism**: Distance computation and attention can be pipelined; pairwise distances can be computed in parallel blocks
+- **Dimension 7 Sparsity**: Distant biases tend toward $-\infty$ (approaching zero after softmax), naturally inducing structured sparsity (local attention windows)
+- **Dimension 8 Operator Fusion**: Geometric bias can be fused into FlashAttention's online softmax loop (added after $QK^T$, before softmax)
+
+## Paper Phrasing
+"We propose geometry-aware attention, which explicitly injects learnable geometric distance bias terms into attention scores, enabling the model to naturally respect the metric structure of the input space. This achieves exponential decay of attention to distant tokens without increasing parameter count, while maintaining compatibility with FlashAttention."
+
+## Risks
+- **Geometric Prior Conflicting with Data**: If geometric distance is inconsistent with semantic relevance (e.g., distant but semantically related tokens in text), strong geometric bias will impair model expressiveness. The bias should be learnable and overridable by content-based attention.
+- **Non-Differentiability of Distance Computation**: Certain distances (graph shortest path, tree LCA depth) are non-differentiable or computationally expensive. Differentiable relaxations (softmin instead of min) or precomputed distance matrices are needed.

package/design-patterns/attention/geometry-aware-attention.md

@@ -0,0 +1,72 @@
+# 几何感知注意力 / Geometry-Aware Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当 token/key 之间存在**已知的几何关系**（空间距离、流形测地距、层级结构、时序间隔）时，标准注意力的位置无关内积无法利用这些结构信息。几何感知注意力将**几何先验直接注入注意力权重计算**，使模型天然尊重底空间的度量结构。典型场景：3D 场景理解、分子构象、时序预测、层级文本结构（段落-句子-词）、知识图谱。
+
+## 数学思想来源
+- 透镜：[symmetry（对称透镜 — 度量不变性）, duality（对偶透镜 — 坐标系无关表达）]
+- 知识：[`../../knowledge-base/information-geometry/fisher-metric.md`（分布空间的几何度量）, `../../knowledge-base/probability/concentration-inequality.md`（几何约束下的浓度行为）]
+
+## 需要的数学知识
+- **度量空间与距离函数**：欧氏距离、测地距、树距离、Wasserstein 距离
+- **RBF / 核方法**：$k(x, y) = \exp(-d(x,y)^2 / 2\sigma^2)$，距离→相似度的转换
+- **位置编码的几何解释**：RoPE = 旋转群 $SO(2)$ 的作用（相对距离编码为旋转角度差）；ALiBi = 指数衰减的距离偏置
+
+## AI 模块形式
+
+**核心思路**：在注意力分数中显式引入几何距离项，使远距离 token 的注意力天然衰减：
+
+$$\text{GeoAttn}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}} + \text{GeoBias}(i, j)\right) V$$
+
+**方案 A：距离偏置注意力（通用度量空间）**：
+```python
+# d_ij 为 token i 和 j 之间的几何距离（预计算或在线计算）
+# 可学习距离偏置函数
+distance_bias = MLP(d_ij)  # 或简单的 -α * d_ij
+scores = (Q @ K.T) / sqrt(d) + distance_bias
+attn = softmax(scores) @ V
+```
+
+**方案 B：相对位置编码（RoPE / ALiBi 的几何推广）**：
+```python
+# 推广 RoPE：位置 → 群元素 g_i，相对位置 → g_i g_j^{-1}
+# 可学习任意度量空间上的相对位置偏置
+def geo_attention(Q, K, V, positions):
+    rel_pos = pairwise_difference(positions)       # (n, n, coord_dim)
+    geo_bias = geo_encoder(rel_pos)               # (n, n) 可学习映射
+    scores = (Q @ K.T) / sqrt(d) + geo_bias
+    return softmax(scores) @ V
+```
+
+**方案 C：流形感知注意力（非欧空间）**：
+```python
+# token 位于非欧流形（双曲空间、球面）时，用测地距替代欧氏距
+def manifold_attention(Q, K, V, manifold):
+    geodist = manifold.geodesic_distance_matrix(positions)  # (n, n)
+    geo_kernel = exp(-geodist^2 / (2 * sigma^2))  # RBF 核
+    scores = (Q @ K.T) / sqrt(d) + log(geo_kernel + eps)
+    return softmax(scores) @ V
+```
+
+## 可实现结构
+- **RoPE 扩展**：从 $SO(2)$ 旋转到更高维旋转群 $SO(2k)$，编码多维位置信息（2D 图像 patch、3D voxel）
+- **层级位置偏置**：树结构中用 LCA（最近公共祖先）深度作为距离，适合文档/代码的层级建模
+- **分子构象注意力**：3D 原子坐标 → 距离矩阵 → 几何偏置，用于分子 GNN 和蛋白质结构预测
+
+## GPU 可行性
+- **维度 1 张量化 ✅**：距离矩阵和偏置矩阵均为稠密张量，逐元素运算
+- **维度 2 GEMM 可映射 ✅**：主体 $Q K^T$ 为标准 GEMM；几何偏置为加法，不阻断 GEMM
+- **维度 3 复杂度 ⚠️**：成对距离矩阵 $O(n^2)$ 构建和存储；但可用分块计算 + online softmax 避免物化完整矩阵
+- **维度 4 显存 ⚠️**：$n \times n$ 距离矩阵占用显存；可分块/流式计算（与 FlashAttention 兼容）
+- **维度 5 低精度 ✅**：距离计算和偏置加法在 bf16 下稳定；RBF 的 exp 需注意上溢（clamp 距离）
+- **维度 6 并行 ✅**：距离计算和注意力可流水线并行，成对距离可分块并行
+- **维度 7 稀疏 ✅**：远距离偏置趋于 $-\infty$（softmax 后趋零），天然诱导结构化稀疏（局部注意力窗口）
+- **维度 8 算子融合 ✅**：几何偏置可融入 FlashAttention 的 online softmax 循环中（加在 $QK^T$ 之后、softmax 之前）
+
+## 论文表述方式
+"我们提出几何感知注意力，通过在注意力分数中显式注入可学习的几何距离偏置项，使模型天然尊重输入空间的度量结构，在不增加参数量的情况下将远距离 token 的注意力指数衰减，同时保持与 FlashAttention 的兼容性。"
+
+## 风险
+- **几何先验与数据冲突**：若几何距离与语义相关性不一致（如文本中远距离但语义相关的 token），强几何偏置会伤害模型表达。需让偏置可学习且可被内容注意力覆盖。
+- **距离计算的非可微性**：某些距离（图最短路径、树 LCA 深度）不可微或计算复杂。需使用可微松弛（softmin 替代 min）或预计算距离矩阵。

package/design-patterns/attention/information-bottleneck-attention.en.md

@@ -0,0 +1,72 @@
+# Information Bottleneck Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the attention mechanism needs to **selectively transmit useful information while suppressing redundant/noisy information**, information bottleneck theory can guide the learning of attention weights -- maximizing the mutual information $I(Z;Y)$ of the attention distribution with respect to the target $Y$, while minimizing the mutual information $I(X;Z)$ with respect to the input $X$. Typical scenarios include: long-document summarization (filtering large numbers of irrelevant tokens), multimodal alignment (cross-modal noise suppression), and interpretability (attention weights as visualization of information flow).
+
+## Mathematical Inspiration
+- Lenses: [categorical (information-theoretic framework unifying attention design), variational (constrained variational and Lagrangian duality)]
+- Knowledge: [`../../knowledge-base/probability/information-bottleneck.md` (IB objective and variational lower bound), `../../knowledge-base/probability/kl-divergence.md` (implementation of KL regularization), `../../knowledge-base/probability/entropy.md` (mutual information estimation)]
+
+## Required Mathematical Knowledge
+- **Information Bottleneck Objective**: $\min I(X;Z) - \beta I(Z;Y)$, balancing compression and prediction
+- **Variational Information Bottleneck (VIB)**: Replacing intractable mutual information with variational lower bounds
+- **Correspondence Between Mutual Information and Attention**: Softmax attention weights $\alpha_{ij}$ can be interpreted as information channel allocations from key $j$ to query $i$
+
+## AI Module Form
+
+**Core Idea**: Treat attention as an information bottleneck -- attention weights determine "how much information flows from values to output," while KL regularization constrains the information throughput.
+
+**Scheme A: KL-Regularized Attention (Simplest IB Attention)**:
+```python
+# KL divergence between attention distribution q(z|x) and uniform prior as IB regularizer
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+uniform_prior = ones_like(attn) / n
+kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+loss = task_loss + beta * kl_reg
+output = attn @ V
+```
+
+**Scheme B: Variational Information Bottleneck Attention (VIB-Attention)**:
+```python
+# Introduce stochastic bottleneck Z ~ q(Z|attention) to limit information throughput
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+mu_z, log_var_z = attn, linear(attn)  # mean and learnable variance
+z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # reparameterization + softmax
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs prior
+output = z @ V
+loss = task_loss + beta * kl
+```
+
+**Scheme C: Mutual Information Maximization Attention (DIM Style)**:
+```python
+# Directly maximize I(Z;Y) via InfoNCE, with KL constraining I(X;Z)
+Z = softmax((Q @ K.T) / sqrt(d)) @ V
+info_nce = infonce_loss(Z, target_embedding, negatives, tau=0.1)
+kl_bottleneck = estimate_kl(X, Z)  # MINE/NWJ estimator
+loss = -info_nce + beta * kl_bottleneck
+```
+
+## Implementable Architectures
+- **IB-Sparse Attention**: KL regularization serves as a soft constraint on attention sparsity -- larger KL relative to uniform prior leads to more concentrated attention, yielding implicit Top-K selection
+- **IB Interpretation of Dropout**: Dropout is a form of stochastic information bottleneck -- randomly blocking information channels forces the model to learn robust representations. The dropout rate corresponds to the $\beta$ parameter
+- **Multi-Head Information Allocation**: Different heads learn different information bottlenecks (different $\beta$); some heads transmit global information, others only local information
+
+## GPU Feasibility
+- **Dimension 1 Tensorization**: KL regularization involves element-wise operations; VIB reparameterization sampling is element-wise
+- **Dimension 2 GEMM-mappability**: The main body $QK^T$ and $attn \cdot V$ are standard GEMM; regularization introduces no new GEMM operations
+- **Dimension 3 Complexity**: KL regularization is $O(n)$ per token, adding no asymptotic complexity
+- **Dimension 4 Memory**: VIB requires additional $\mu_z$ and $\log\sigma_z$, approximately doubling attention weight memory
+- **Dimension 5 Low Precision**: log/exp in KL computations are stable under bf16 (standard log-softmax tricks)
+- **Dimension 6 Parallelism**: Regularization can be computed in parallel with forward propagation, introducing no serial dependencies
+- **Dimension 7 Sparsity**: KL regularization implicitly induces attention sparsity, amenable to block-sparse acceleration
+- **Dimension 8 Operator Fusion**: KL can be fused into the softmax kernel (FusedSoftmaxKL)
+
+## Paper Phrasing
+"We propose information bottleneck attention, which models attention as an information bottleneck variational problem. By maximizing the mutual information between output and target while minimizing redundant information transmission from the input, it achieves theoretically optimal information selection and demonstrates enhanced sparsity and interpretability in experiments."
+
+## Risks
+- **High Variance of Mutual Information Estimators**: The MINE/NWJ/InfoNCE estimators in Scheme C exhibit high variance in high-dimensional spaces, potentially causing training instability. It is recommended to first validate the basic effect of IB attention with Scheme A (KL regularization) before attempting the full IB objective.
+- **Difficulty of $\beta$ Tuning**: $\beta$ controls the compression-prediction trade-off, and the optimal $\beta$ varies significantly across tasks. Too small a $\beta$ degenerates to standard attention; too large a $\beta$ causes underfitting. Adaptive $\beta$ scheduling or information plane monitoring is recommended.

package/design-patterns/attention/information-bottleneck-attention.md

@@ -0,0 +1,72 @@
+# 信息瓶颈注意力 / Information Bottleneck Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当注意力机制需要**选择性传递有用信息、抑制冗余/噪声信息**时，用信息瓶颈理论指导注意力权重的学习——使注意力分布最大化关于目标 $Y$ 的互信息 $I(Z;Y)$，同时最小化关于输入 $X$ 的互信息 $I(X;Z)$。典型场景：长文档摘要（大量无关 token 需被过滤）、多模态对齐（跨模态噪声抑制）、可解释性（注意力权重作为信息流的可视化）。
+
+## 数学思想来源
+- 透镜：[categorical（范畴化透镜 — 信息论框架统一注意力设计）, variational（变分透镜 — 约束优化与拉格朗日对偶）]
+- 知识：[`../../knowledge-base/probability/information-bottleneck.md`（IB 目标函数与变分下界）, `../../knowledge-base/probability/kl-divergence.md`（KL 正则项的实现）, `../../knowledge-base/probability/entropy.md`（互信息估计）]
+
+## 需要的数学知识
+- **信息瓶颈 IB 目标**：$\min I(X;Z) - \beta I(Z;Y)$，压缩-预测权衡
+- **变分信息瓶颈 VIB**：用变分下界替代难以估计的互信息
+- **互信息与注意力的对应**：softmax 注意力权重 $\alpha_{ij}$ 可解释为从 key $j$ 到 query $i$ 的信息信道分配
+
+## AI 模块形式
+
+**核心思路**：将注意力视为信息瓶颈——注意力权重决定"让多少信息从 value 流向输出"，同时用 KL 正则约束信息量不过大。
+
+**方案 A：KL 正则化注意力（最简 IB 注意力）**：
+```python
+# 注意力分布 q(z|x) 与均匀先验的 KL 散度作为 IB 正则
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+uniform_prior = ones_like(attn) / n
+kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+loss = task_loss + beta * kl_reg
+output = attn @ V
+```
+
+**方案 B：变分信息瓶颈注意力（VIB-Attention）**：
+```python
+# 引入随机瓶颈 Z ~ q(Z|attention)，限制信息通过量
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+mu_z, log_var_z = attn, linear(attn)  # 均值与可学习方差
+z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # 重参数化 + softmax
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs 先验
+output = z @ V
+loss = task_loss + beta * kl
+```
+
+**方案 C：互信息最大化注意力（DIM 风格）**：
+```python
+# 用 InfoNCE 直接最大化 I(Z;Y)，配合 KL 约束 I(X;Z)
+Z = softmax((Q @ K.T) / sqrt(d)) @ V
+info_nce = infonce_loss(Z, target_embedding, negatives, tau=0.1)
+kl_bottleneck = estimate_kl(X, Z)  # MINE/NWJ 估计器
+loss = -info_nce + beta * kl_bottleneck
+```
+
+## 可实现结构
+- **IB-Sparse Attention**：将 KL 正则化为注意力稀疏性的软约束——KL 对均匀先越大 → 注意力越集中 → 隐式 Top-K 选择
+- **Dropout 的 IB 解释**：Dropout 是一种随机信息瓶颈——随机阻断部分信息通道，迫使模型学习鲁棒表示。Dropout rate 对应 $\beta$ 参数
+- **Multi-Head 信息分配**：不同 head 学习不同的信息瓶颈（不同 $\beta$），有的 head 传递全局信息，有的只传递局部信息
+
+## GPU 可行性
+- **维度 1 张量化 ✅**：KL 正则项为逐元素运算，VIB 重参数化采样为逐元素
+- **维度 2 GEMM 可映射 ✅**：主体 $QK^T$ 和 $attn \cdot V$ 为标准 GEMM，正则项不引入新 GEMM
+- **维度 3 复杂度 ✅**：KL 正则 $O(n)$ 逐 token，不增加渐近复杂度
+- **维度 4 显存 ⚠️**：VIB 需额外 $\mu_z$ 和 $\log\sigma_z$，约 2x 注意力权重显存
+- **维度 5 低精度 ✅**：KL 中 log/exp 在 bf16 下稳定（标准 log-softmax 技巧）
+- **维度 6 并行 ✅**：正则项可与前向传播并行，不引入串行依赖
+- **维度 7 稀疏 ✅**：KL 正则隐式诱导注意力稀疏性，可利用 block-sparse 加速
+- **维度 8 算子融合 ✅**：KL 可融入 softmax kernel（FusedSoftmaxKL）
+
+## 论文表述方式
+"我们提出信息瓶颈注意力机制，通过将注意力建模为信息瓶颈优化问题，在最大化输出与目标的互信息的同时最小化输入信息的冗余传递，实现理论上最优的信息选择，并在实验中展现出更强的稀疏性和可解释性。"
+
+## 风险
+- **互信息估计的方差问题**：方案 C 中的 MINE/NWJ/InfoNCE 估计器在高维空间中方差大，可能导致训练不稳定。建议先用方案 A（KL 正则）验证 IB 注意力的基本效果，再尝试完整 IB 目标。
+- **$\beta$ 调参困难**：$\beta$ 控制压缩-预测权衡，不同任务最优 $\beta$ 差异大。$\beta$ 过小退化为标准注意力，$\beta$ 过大导致欠拟合。建议自适应 $\beta$ 调度或信息平面监控。

package/design-patterns/attention/projection-attention.en.md

@@ -0,0 +1,68 @@
+# Projection Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the token/key space dimensionality is too high, the standard attention $Q K^T$ inner product tends to become uniform in high-dimensional spaces ("attention collapse"). In such cases, key/query vectors must first be projected onto a **subspace with superior geometric structure** before computing attention. Typical scenarios include: KV-Cache compression for long-context LLMs, attention sparsification in high-dimensional embedding spaces, and attention alignment of heterogeneous features in multimodal fusion.
+
+## Mathematical Inspiration
+- Lenses: [projection, spectral, probabilistic]
+- Knowledge: [`../../knowledge-base/probability/concentration-inequality.md` (high-dimensional concentration inequalities provide theoretical explanation for attention collapse), `../../knowledge-base/probability/entropy.md` (entropy of attention distributions as a quality metric)]
+
+## Required Mathematical Knowledge
+- **Johnson-Lindenstrauss Lemma**: High-dimensional point sets can be projected onto an $O(\log n / \epsilon^2)$-dimensional subspace while approximately preserving distances
+- **Random Projections and Subspace Embeddings**: Sparse projection matrices (e.g., CountSketch, SRHT) approximately preserve inner products
+- **SVD / PCA Truncation**: Optimal low-rank subspace projection that maximizes retained variance
+
+## AI Module Form
+
+**Core Idea**: Project $Q, K \in \mathbb{R}^{n \times d}$ onto an $r$-dimensional subspace ($r \ll d$) before computing attention:
+
+$$\text{ProjAttn}(Q, K, V) = \text{softmax}\left(\frac{Q P_Q (K P_K)^T}{\sqrt{r}}\right) V$$
+
+where $P_Q, P_K \in \mathbb{R}^{d \times r}$ are projection matrices.
+
+**Three Projection Strategies**:
+
+1. **Learnable Projection** ($P$ as trainable parameters):
+```
+P_Q = Linear(d, r, bias=False)  # r << d
+P_K = Linear(d, r, bias=False)
+scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
+attn = softmax(scores) @ V
+```
+
+2. **Random Fixed Projection** (guaranteed by the JL Lemma):
+```
+P = random_gaussian(d, r) / sqrt(r)  # fixed, not trained
+scores = (Q @ P) @ (K @ P).T / sqrt(r)
+```
+
+3. **Data-Adaptive Projection** (online PCA):
+```
+# Maintain running covariance of K, take top-r eigenvectors
+C = running_mean(K^T @ K)  # d x d
+eigenvecs = top_r_eigenvectors(C)  # d x r
+scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
+```
+
+## Implementable Architectures
+- **Multi-Head Projection**: Each head uses a different $P_h \in \mathbb{R}^{d_h \times r}$, with total computation $O(n \cdot d \cdot r + n^2 \cdot r)$. When $r \ll d$, this saves the $O(n^2 d)$ cost of $Q K^T$
+- **Key-cache Compression**: The projected $K' = K P_K \in \mathbb{R}^{n \times r}$ replaces the original $K$ in storage, reducing Key-cache memory by about $d/r$. Full KV-Cache compression requires a separate V compression/reconstruction mechanism
+- **Hierarchical Projection**: Shallow layers use small $r$ (coarse filtering), while deep layers use large $r$ (fine ranking)
+
+## GPU Feasibility
+- **Dimension 1 Tensorization**: Projection = matrix multiplication, attention = matrix multiplication chain; entirely tensor operations
+- **Dimension 2 GEMM-mappability**: $Q P_Q$ and $K P_K$ are both standard GEMM operations, fully utilizing Tensor Cores
+- **Dimension 3 Complexity**: Projection cost $O(ndr)$ is far below attention cost $O(n^2 d)$, and after projection $r \ll d$ reduces attention to $O(n^2 r)$
+- **Dimension 4 Memory**: Key-cache compressed by $d/r$; V-cache must be handled separately
+- **Dimension 5 Low Precision**: Projection matrices are orthogonal or near-orthogonal, yielding numerical stability; bf16 is acceptable
+- **Dimension 6 Parallelism**: Projection can be pipelined with attention; Multi-Head is naturally parallel across heads
+- **Dimension 7 Sparsity**: Projection matrices are inherently dense; sparse projections (CountSketch) may introduce gather/scatter operations
+- **Dimension 8 Operator Fusion ⚠️ Retrofittable, needs kernel-level validation**: Projection may be integrated into a FlashAttention-style kernel, but requires kernel-level verification
+
+## Paper Phrasing
+"We decompose attention computation into two steps -- low-dimensional subspace projection and projected-space attention -- achieving a $d/r$-fold Key-cache compression (full KV-Cache compression requires separate V compression/reconstruction) while preserving attention quality and guaranteeing Johnson-Lindenstrauss distance preservation."
+
+## Risks
+- **Projection Direction Degeneracy**: Learnable projections may collapse onto a few directions (deterioration of the condition number of $P^T P$), causing attention distributions to degenerate. Orthogonal regularization $\|P^T P - I\|_F^2$ is required.
+- **Irreversible Information Loss**: Projection discards $(d-r)$ dimensions of information; if the task relies on features in these dimensions, performance will degrade. It is recommended to combine with residual connections (weighted combination of original attention and projected attention).

package/design-patterns/attention/projection-attention.md

@@ -0,0 +1,68 @@
+# 投影注意力 / Projection Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入的 token/key 空间维度过高，标准注意力的 $Q K^T$ 内积在高维空间中趋于均匀（"注意力坍缩"）时，需要先将 key/query 投影到一个**几何结构更优的子空间**再做注意力计算。典型场景：长上下文 LLM 的 KV-Cache 压缩、高维嵌入空间的注意力稀疏化、多模态融合中异构特征的注意力对齐。
+
+## 数学思想来源
+- 透镜：[projection（投影与分解）, spectral（谱分解）, probabilistic（概率）]
+- 知识：[`../../knowledge-base/probability/concentration-inequality.md`（高维浓度不等式 → 注意力坍缩的理论解释）, `../../knowledge-base/probability/entropy.md`（注意力分布的熵作为质量指标）]
+
+## 需要的数学知识
+- **Johnson-Lindenstrauss 引理**：高维点集可投影到 $O(\log n / \epsilon^2)$ 维子空间而近似保持距离
+- **随机投影与子空间嵌入**：稀疏投影矩阵（如 CountSketch、SRHT）近似保持内积
+- **SVD / PCA 截断**：最优低秩子空间投影，最大化保留方差
+
+## AI 模块形式
+
+**核心思路**：将 $Q, K \in \mathbb{R}^{n \times d}$ 先投影到 $r$ 维子空间（$r \ll d$），再计算注意力：
+
+$$\text{ProjAttn}(Q, K, V) = \text{softmax}\left(\frac{Q P_Q (K P_K)^T}{\sqrt{r}}\right) V$$
+
+其中 $P_Q, P_K \in \mathbb{R}^{d \times r}$ 为投影矩阵。
+
+**三种投影策略**：
+
+1. **可学习投影**（$P$ 为训练参数）：
+```
+P_Q = Linear(d, r, bias=False)  # r << d
+P_K = Linear(d, r, bias=False)
+scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
+attn = softmax(scores) @ V
+```
+
+2. **随机固定投影**（JL 引理保证）：
+```
+P = random_gaussian(d, r) / sqrt(r)  # 固定不训练
+scores = (Q @ P) @ (K @ P).T / sqrt(r)
+```
+
+3. **数据自适应投影**（在线 PCA）：
+```
+# 维护 K 的 running covariance，取 top-r 特征向量
+C = running_mean(K^T @ K)  # d x d
+eigenvecs = top_r_eigenvectors(C)  # d x r
+scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
+```
+
+## 可实现结构
+- **Multi-Head 投影**：每个 head 使用不同的 $P_h \in \mathbb{R}^{d_h \times r}$，总计算量 $O(n \cdot d \cdot r + n^2 \cdot r)$，当 $r \ll d$ 时节省 $Q K^T$ 的 $O(n^2 d)$ 开销
+- **KV-Cache 压缩**：投影后的 $K' = K P_K \in \mathbb{R}^{n \times r}$ 替代原始 $K$ 存储，Key 部分显存节省约 $d/r$ 倍（完整 KV-Cache 压缩需额外设计 V 的压缩/重构机制）
+- **分层投影**：浅层用小 $r$（粗筛），深层用大 $r$（精排）
+
+## GPU 可行性
+- **维度 1 张量化 ✅**：投影 = 矩阵乘，注意力 = 矩阵乘链，全程张量运算
+- **维度 2 GEMM 可映射 ✅**：$Q P_Q$ 和 $K P_K$ 均为标准 GEMM，吃满 Tensor Core
+- **维度 3 复杂度 ✅**：投影 $O(ndr)$ 远低于注意力 $O(n^2 d)$，且投影后 $r \ll d$ 使注意力 $O(n^2 r)$
+- **维度 4 显存 ✅**：Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），直接降低显存峰值
+- **维度 5 低精度 ✅**：投影矩阵正交/近正交时数值稳定，bf16 可接受
+- **维度 6 并行 ✅**：投影可与注意力流水线并行，Multi-Head 天然跨头并行
+- **维度 7 稀疏 ⚠️**：投影矩阵本身稠密；若用稀疏投影（CountSketch），可能引入 gather/scatter
+- **维度 8 算子融合 ⚠️可改造需验证**：投影可融入 FlashAttention 的 online softmax 循环中
+
+## 论文表述方式
+"我们将注意力计算分解为低维子空间投影与投影空间注意力两步，在保持注意力质量的同时将 Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），并保证 Johnson-Lindenstrauss 距离保持性。"
+
+## 风险
+- **投影方向退化**：可学习投影可能坍缩到少数方向（$P^T P$ 条件数恶化），导致注意力分布退化。需加正交正则化 $\|P^T P - I\|_F^2$。
+- **信息丢失不可逆**：投影丢弃了 $(d-r)$ 维信息，若任务恰需这些维度的特征则性能下降。建议配合残差连接（原始注意力 + 投影注意力的加权组合）。