math-skill 2.1.0 → 3.0.1

package/design-patterns/attention/geometry-aware-attention.md

@@ -0,0 +1,72 @@
+# 几何感知注意力 / Geometry-Aware Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当 token/key 之间存在**已知的几何关系**（空间距离、流形测地距、层级结构、时序间隔）时，标准注意力的位置无关内积无法利用这些结构信息。几何感知注意力将**几何先验直接注入注意力权重计算**，使模型天然尊重底空间的度量结构。典型场景：3D 场景理解、分子构象、时序预测、层级文本结构（段落-句子-词）、知识图谱。
+
+## 数学思想来源
+- 透镜：[symmetry（对称透镜 — 度量不变性）, duality（对偶透镜 — 坐标系无关表达）]
+- 知识：[`../../knowledge-base/information-geometry/fisher-metric.md`（分布空间的几何度量）, `../../knowledge-base/probability/concentration-inequality.md`（几何约束下的浓度行为）]
+
+## 需要的数学知识
+- **度量空间与距离函数**：欧氏距离、测地距、树距离、Wasserstein 距离
+- **RBF / 核方法**：$k(x, y) = \exp(-d(x,y)^2 / 2\sigma^2)$，距离→相似度的转换
+- **位置编码的几何解释**：RoPE = 旋转群 $SO(2)$ 的作用（相对距离编码为旋转角度差）；ALiBi = 指数衰减的距离偏置
+
+## AI 模块形式
+
+**核心思路**：在注意力分数中显式引入几何距离项，使远距离 token 的注意力天然衰减：
+
+$$\text{GeoAttn}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}} + \text{GeoBias}(i, j)\right) V$$
+
+**方案 A：距离偏置注意力（通用度量空间）**：
+```python
+# d_ij 为 token i 和 j 之间的几何距离（预计算或在线计算）
+# 可学习距离偏置函数
+distance_bias = MLP(d_ij)  # 或简单的 -α * d_ij
+scores = (Q @ K.T) / sqrt(d) + distance_bias
+attn = softmax(scores) @ V
+```
+
+**方案 B：相对位置编码（RoPE / ALiBi 的几何推广）**：
+```python
+# 推广 RoPE：位置 → 群元素 g_i，相对位置 → g_i g_j^{-1}
+# 可学习任意度量空间上的相对位置偏置
+def geo_attention(Q, K, V, positions):
+    rel_pos = pairwise_difference(positions)       # (n, n, coord_dim)
+    geo_bias = geo_encoder(rel_pos)               # (n, n) 可学习映射
+    scores = (Q @ K.T) / sqrt(d) + geo_bias
+    return softmax(scores) @ V
+```
+
+**方案 C：流形感知注意力（非欧空间）**：
+```python
+# token 位于非欧流形（双曲空间、球面）时，用测地距替代欧氏距
+def manifold_attention(Q, K, V, manifold):
+    geodist = manifold.geodesic_distance_matrix(positions)  # (n, n)
+    geo_kernel = exp(-geodist^2 / (2 * sigma^2))  # RBF 核
+    scores = (Q @ K.T) / sqrt(d) + log(geo_kernel + eps)
+    return softmax(scores) @ V
+```
+
+## 可实现结构
+- **RoPE 扩展**：从 $SO(2)$ 旋转到更高维旋转群 $SO(2k)$，编码多维位置信息（2D 图像 patch、3D voxel）
+- **层级位置偏置**：树结构中用 LCA（最近公共祖先）深度作为距离，适合文档/代码的层级建模
+- **分子构象注意力**：3D 原子坐标 → 距离矩阵 → 几何偏置，用于分子 GNN 和蛋白质结构预测
+
+## GPU 可行性
+- **D1[v]**：距离矩阵和偏置矩阵均为稠密张量，逐元素运算
+- **D2[v]**：主体 $Q K^T$ 为标准 GEMM；几何偏置为加法，不阻断 GEMM
+- **D3[~]**：成对距离矩阵 $O(n^2)$ 构建和存储；但可用分块计算 + online softmax 避免物化完整矩阵
+- **D4[~]**：$n \times n$ 距离矩阵占用显存；可分块/流式计算（与 FlashAttention 兼容）
+- **D5[v]**：距离计算和偏置加法在 bf16 下稳定；RBF 的 exp 需注意上溢（clamp 距离）
+- **D6[v]**：距离计算和注意力可流水线并行，成对距离可分块并行
+- **D7[v]**：远距离偏置趋于 $-\infty$（softmax 后趋零），天然诱导结构化稀疏（局部注意力窗口）
+- **D8[v]**：几何偏置可融入 FlashAttention 的 online softmax 循环中（加在 $QK^T$ 之后、softmax 之前）
+
+## 论文表述方式
+"我们提出几何感知注意力，通过在注意力分数中显式注入可学习的几何距离偏置项，使模型天然尊重输入空间的度量结构，在不增加参数量的情况下将远距离 token 的注意力指数衰减，同时保持与 FlashAttention 的兼容性。"
+
+## 风险
+- **几何先验与数据冲突**：若几何距离与语义相关性不一致（如文本中远距离但语义相关的 token），强几何偏置会伤害模型表达。需让偏置可学习且可被内容注意力覆盖。
+- **距离计算的非可微性**：某些距离（图最短路径、树 LCA 深度）不可微或计算复杂。需使用可微松弛（softmin 替代 min）或预计算距离矩阵。

package/design-patterns/attention/information-bottleneck-attention.en.md

@@ -0,0 +1,72 @@
+# Information Bottleneck Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the attention mechanism needs to **selectively transmit useful information while suppressing redundant/noisy information**, information bottleneck theory can guide the learning of attention weights -- maximizing the mutual information $I(Z;Y)$ of the attention distribution with respect to the target $Y$, while minimizing the mutual information $I(X;Z)$ with respect to the input $X$. Typical scenarios include: long-document summarization (filtering large numbers of irrelevant tokens), multimodal alignment (cross-modal noise suppression), and interpretability (attention weights as visualization of information flow).
+
+## Mathematical Inspiration
+- Lenses: [categorical (information-theoretic framework unifying attention design), variational (constrained variational and Lagrangian duality)]
+- Knowledge: [`../../knowledge-base/probability/information-bottleneck.md` (IB objective and variational lower bound), `../../knowledge-base/probability/kl-divergence.md` (implementation of KL regularization), `../../knowledge-base/probability/entropy.md` (mutual information estimation)]
+
+## Required Mathematical Knowledge
+- **Information Bottleneck Objective**: $\min I(X;Z) - \beta I(Z;Y)$, balancing compression and prediction
+- **Variational Information Bottleneck (VIB)**: Replacing intractable mutual information with variational lower bounds
+- **Correspondence Between Mutual Information and Attention**: Softmax attention weights $\alpha_{ij}$ can be interpreted as information channel allocations from key $j$ to query $i$
+
+## AI Module Form
+
+**Core Idea**: Treat attention as an information bottleneck -- attention weights determine "how much information flows from values to output," while KL regularization constrains the information throughput.
+
+**Scheme A: KL-Regularized Attention (Simplest IB Attention)**:
+```python
+# KL divergence between attention distribution q(z|x) and uniform prior as IB regularizer
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+uniform_prior = ones_like(attn) / n
+kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+loss = task_loss + beta * kl_reg
+output = attn @ V
+```
+
+**Scheme B: Variational Information Bottleneck Attention (VIB-Attention)**:
+```python
+# Introduce stochastic bottleneck Z ~ q(Z|attention) to limit information throughput
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+mu_z, log_var_z = attn, linear(attn)  # mean and learnable variance
+z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # reparameterization + softmax
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs prior
+output = z @ V
+loss = task_loss + beta * kl
+```
+
+**Scheme C: Mutual Information Maximization Attention (DIM Style)**:
+```python
+# Directly maximize I(Z;Y) via InfoNCE, with KL constraining I(X;Z)
+Z = softmax((Q @ K.T) / sqrt(d)) @ V
+info_nce = infonce_loss(Z, target_embedding, negatives, tau=0.1)
+kl_bottleneck = estimate_kl(X, Z)  # MINE/NWJ estimator
+loss = -info_nce + beta * kl_bottleneck
+```
+
+## Implementable Architectures
+- **IB-Sparse Attention**: KL regularization serves as a soft constraint on attention sparsity -- larger KL relative to uniform prior leads to more concentrated attention, yielding implicit Top-K selection
+- **IB Interpretation of Dropout**: Dropout is a form of stochastic information bottleneck -- randomly blocking information channels forces the model to learn robust representations. The dropout rate corresponds to the $\beta$ parameter
+- **Multi-Head Information Allocation**: Different heads learn different information bottlenecks (different $\beta$); some heads transmit global information, others only local information
+
+## GPU Feasibility
+- **D1**: KL regularization involves element-wise operations; VIB reparameterization sampling is element-wise
+- **D2**: The main body $QK^T$ and $attn \cdot V$ are standard GEMM; regularization introduces no new GEMM operations
+- **D3**: KL regularization is $O(n)$ per token, adding no asymptotic complexity
+- **D4**: VIB requires additional $\mu_z$ and $\log\sigma_z$, approximately doubling attention weight memory
+- **D5**: log/exp in KL computations are stable under bf16 (standard log-softmax tricks)
+- **D6**: Regularization can be computed in parallel with forward propagation, introducing no serial dependencies
+- **D7**: KL regularization implicitly induces attention sparsity, amenable to block-sparse acceleration
+- **D8**: KL can be fused into the softmax kernel (FusedSoftmaxKL)
+
+## Paper Phrasing
+"We propose information bottleneck attention, which models attention as an information bottleneck variational problem. By maximizing the mutual information between output and target while minimizing redundant information transmission from the input, it achieves theoretically optimal information selection and demonstrates enhanced sparsity and interpretability in experiments."
+
+## Risks
+- **High Variance of Mutual Information Estimators**: The MINE/NWJ/InfoNCE estimators in Scheme C exhibit high variance in high-dimensional spaces, potentially causing training instability. It is recommended to first validate the basic effect of IB attention with Scheme A (KL regularization) before attempting the full IB objective.
+- **Difficulty of $\beta$ Tuning**: $\beta$ controls the compression-prediction trade-off, and the optimal $\beta$ varies significantly across tasks. Too small a $\beta$ degenerates to standard attention; too large a $\beta$ causes underfitting. Adaptive $\beta$ scheduling or information plane monitoring is recommended.

package/design-patterns/attention/information-bottleneck-attention.md

@@ -0,0 +1,72 @@
+# 信息瓶颈注意力 / Information Bottleneck Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当注意力机制需要**选择性传递有用信息、抑制冗余/噪声信息**时，用信息瓶颈理论指导注意力权重的学习——使注意力分布最大化关于目标 $Y$ 的互信息 $I(Z;Y)$，同时最小化关于输入 $X$ 的互信息 $I(X;Z)$。典型场景：长文档摘要（大量无关 token 需被过滤）、多模态对齐（跨模态噪声抑制）、可解释性（注意力权重作为信息流的可视化）。
+
+## 数学思想来源
+- 透镜：[categorical（范畴化透镜 — 信息论框架统一注意力设计）, variational（变分透镜 — 约束优化与拉格朗日对偶）]
+- 知识：[`../../knowledge-base/probability/information-bottleneck.md`（IB 目标函数与变分下界）, `../../knowledge-base/probability/kl-divergence.md`（KL 正则项的实现）, `../../knowledge-base/probability/entropy.md`（互信息估计）]
+
+## 需要的数学知识
+- **信息瓶颈 IB 目标**：$\min I(X;Z) - \beta I(Z;Y)$，压缩-预测权衡
+- **变分信息瓶颈 VIB**：用变分下界替代难以估计的互信息
+- **互信息与注意力的对应**：softmax 注意力权重 $\alpha_{ij}$ 可解释为从 key $j$ 到 query $i$ 的信息信道分配
+
+## AI 模块形式
+
+**核心思路**：将注意力视为信息瓶颈——注意力权重决定"让多少信息从 value 流向输出"，同时用 KL 正则约束信息量不过大。
+
+**方案 A：KL 正则化注意力（最简 IB 注意力）**：
+```python
+# 注意力分布 q(z|x) 与均匀先验的 KL 散度作为 IB 正则
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+uniform_prior = ones_like(attn) / n
+kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+loss = task_loss + beta * kl_reg
+output = attn @ V
+```
+
+**方案 B：变分信息瓶颈注意力（VIB-Attention）**：
+```python
+# 引入随机瓶颈 Z ~ q(Z|attention)，限制信息通过量
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+mu_z, log_var_z = attn, linear(attn)  # 均值与可学习方差
+z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # 重参数化 + softmax
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs 先验
+output = z @ V
+loss = task_loss + beta * kl
+```
+
+**方案 C：互信息最大化注意力（DIM 风格）**：
+```python
+# 用 InfoNCE 直接最大化 I(Z;Y)，配合 KL 约束 I(X;Z)
+Z = softmax((Q @ K.T) / sqrt(d)) @ V
+info_nce = infonce_loss(Z, target_embedding, negatives, tau=0.1)
+kl_bottleneck = estimate_kl(X, Z)  # MINE/NWJ 估计器
+loss = -info_nce + beta * kl_bottleneck
+```
+
+## 可实现结构
+- **IB-Sparse Attention**：将 KL 正则化为注意力稀疏性的软约束——KL 对均匀先越大 → 注意力越集中 → 隐式 Top-K 选择
+- **Dropout 的 IB 解释**：Dropout 是一种随机信息瓶颈——随机阻断部分信息通道，迫使模型学习鲁棒表示。Dropout rate 对应 $\beta$ 参数
+- **Multi-Head 信息分配**：不同 head 学习不同的信息瓶颈（不同 $\beta$），有的 head 传递全局信息，有的只传递局部信息
+
+## GPU 可行性
+- **D1[v]**：KL 正则项为逐元素运算，VIB 重参数化采样为逐元素
+- **D2[v]**：主体 $QK^T$ 和 $attn \cdot V$ 为标准 GEMM，正则项不引入新 GEMM
+- **D3[v]**：KL 正则 $O(n)$ 逐 token，不增加渐近复杂度
+- **D4[~]**：VIB 需额外 $\mu_z$ 和 $\log\sigma_z$，约 2x 注意力权重显存
+- **D5[v]**：KL 中 log/exp 在 bf16 下稳定（标准 log-softmax 技巧）
+- **D6[v]**：正则项可与前向传播并行，不引入串行依赖
+- **D7[v]**：KL 正则隐式诱导注意力稀疏性，可利用 block-sparse 加速
+- **D8[v]**：KL 可融入 softmax kernel（FusedSoftmaxKL）
+
+## 论文表述方式
+"我们提出信息瓶颈注意力机制，通过将注意力建模为信息瓶颈优化问题，在最大化输出与目标的互信息的同时最小化输入信息的冗余传递，实现理论上最优的信息选择，并在实验中展现出更强的稀疏性和可解释性。"
+
+## 风险
+- **互信息估计的方差问题**：方案 C 中的 MINE/NWJ/InfoNCE 估计器在高维空间中方差大，可能导致训练不稳定。建议先用方案 A（KL 正则）验证 IB 注意力的基本效果，再尝试完整 IB 目标。
+- **$\beta$ 调参困难**：$\beta$ 控制压缩-预测权衡，不同任务最优 $\beta$ 差异大。$\beta$ 过小退化为标准注意力，$\beta$ 过大导致欠拟合。建议自适应 $\beta$ 调度或信息平面监控。

package/design-patterns/attention/projection-attention.en.md

@@ -0,0 +1,68 @@
+# Projection Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the token/key space dimensionality is too high, the standard attention $Q K^T$ inner product tends to become uniform in high-dimensional spaces ("attention collapse"). In such cases, key/query vectors must first be projected onto a **subspace with superior geometric structure** before computing attention. Typical scenarios include: KV-Cache compression for long-context LLMs, attention sparsification in high-dimensional embedding spaces, and attention alignment of heterogeneous features in multimodal fusion.
+
+## Mathematical Inspiration
+- Lenses: [projection, spectral, probabilistic]
+- Knowledge: [`../../knowledge-base/probability/concentration-inequality.md` (high-dimensional concentration inequalities provide theoretical explanation for attention collapse), `../../knowledge-base/probability/entropy.md` (entropy of attention distributions as a quality metric)]
+
+## Required Mathematical Knowledge
+- **Johnson-Lindenstrauss Lemma**: High-dimensional point sets can be projected onto an $O(\log n / \epsilon^2)$-dimensional subspace while approximately preserving distances
+- **Random Projections and Subspace Embeddings**: Sparse projection matrices (e.g., CountSketch, SRHT) approximately preserve inner products
+- **SVD / PCA Truncation**: Optimal low-rank subspace projection that maximizes retained variance
+
+## AI Module Form
+
+**Core Idea**: Project $Q, K \in \mathbb{R}^{n \times d}$ onto an $r$-dimensional subspace ($r \ll d$) before computing attention:
+
+$$\text{ProjAttn}(Q, K, V) = \text{softmax}\left(\frac{Q P_Q (K P_K)^T}{\sqrt{r}}\right) V$$
+
+where $P_Q, P_K \in \mathbb{R}^{d \times r}$ are projection matrices.
+
+**Three Projection Strategies**:
+
+1. **Learnable Projection** ($P$ as trainable parameters):
+```
+P_Q = Linear(d, r, bias=False)  # r << d
+P_K = Linear(d, r, bias=False)
+scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
+attn = softmax(scores) @ V
+```
+
+2. **Random Fixed Projection** (guaranteed by the JL Lemma):
+```
+P = random_gaussian(d, r) / sqrt(r)  # fixed, not trained
+scores = (Q @ P) @ (K @ P).T / sqrt(r)
+```
+
+3. **Data-Adaptive Projection** (online PCA):
+```
+# Maintain running covariance of K, take top-r eigenvectors
+C = running_mean(K^T @ K)  # d x d
+eigenvecs = top_r_eigenvectors(C)  # d x r
+scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
+```
+
+## Implementable Architectures
+- **Multi-Head Projection**: Each head uses a different $P_h \in \mathbb{R}^{d_h \times r}$, with total computation $O(n \cdot d \cdot r + n^2 \cdot r)$. When $r \ll d$, this saves the $O(n^2 d)$ cost of $Q K^T$
+- **Key-cache Compression**: The projected $K' = K P_K \in \mathbb{R}^{n \times r}$ replaces the original $K$ in storage, reducing Key-cache memory by about $d/r$. Full KV-Cache compression requires a separate V compression/reconstruction mechanism
+- **Hierarchical Projection**: Shallow layers use small $r$ (coarse filtering), while deep layers use large $r$ (fine ranking)
+
+## GPU Feasibility
+- **D1**: Projection = matrix multiplication, attention = matrix multiplication chain; entirely tensor operations
+- **D2**: $Q P_Q$ and $K P_K$ are both standard GEMM operations, fully utilizing Tensor Cores
+- **D3**: Projection cost $O(ndr)$ is far below attention cost $O(n^2 d)$, and after projection $r \ll d$ reduces attention to $O(n^2 r)$
+- **D4**: Key-cache compressed by $d/r$; V-cache must be handled separately
+- **D5**: Projection matrices are orthogonal or near-orthogonal, yielding numerical stability; bf16 is acceptable
+- **D6**: Projection can be pipelined with attention; Multi-Head is naturally parallel across heads
+- **D7**: Projection matrices are inherently dense; sparse projections (CountSketch) may introduce gather/scatter operations
+- **D8[~] Retrofittable, needs kernel-level validation**: Projection may be integrated into a FlashAttention-style kernel, but requires kernel-level verification
+
+## Paper Phrasing
+"We decompose attention computation into two steps -- low-dimensional subspace projection and projected-space attention -- achieving a $d/r$-fold Key-cache compression (full KV-Cache compression requires separate V compression/reconstruction) while preserving attention quality and guaranteeing Johnson-Lindenstrauss distance preservation."
+
+## Risks
+- **Projection Direction Degeneracy**: Learnable projections may collapse onto a few directions (deterioration of the condition number of $P^T P$), causing attention distributions to degenerate. Orthogonal regularization $\|P^T P - I\|_F^2$ is required.
+- **Irreversible Information Loss**: Projection discards $(d-r)$ dimensions of information; if the task relies on features in these dimensions, performance will degrade. It is recommended to combine with residual connections (weighted combination of original attention and projected attention).

package/design-patterns/attention/projection-attention.md

@@ -0,0 +1,68 @@
+# 投影注意力 / Projection Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入的 token/key 空间维度过高，标准注意力的 $Q K^T$ 内积在高维空间中趋于均匀（"注意力坍缩"）时，需要先将 key/query 投影到一个**几何结构更优的子空间**再做注意力计算。典型场景：长上下文 LLM 的 KV-Cache 压缩、高维嵌入空间的注意力稀疏化、多模态融合中异构特征的注意力对齐。
+
+## 数学思想来源
+- 透镜：[projection（投影与分解）, spectral（谱分解）, probabilistic（概率）]
+- 知识：[`../../knowledge-base/probability/concentration-inequality.md`（高维浓度不等式 → 注意力坍缩的理论解释）, `../../knowledge-base/probability/entropy.md`（注意力分布的熵作为质量指标）]
+
+## 需要的数学知识
+- **Johnson-Lindenstrauss 引理**：高维点集可投影到 $O(\log n / \epsilon^2)$ 维子空间而近似保持距离
+- **随机投影与子空间嵌入**：稀疏投影矩阵（如 CountSketch、SRHT）近似保持内积
+- **SVD / PCA 截断**：最优低秩子空间投影，最大化保留方差
+
+## AI 模块形式
+
+**核心思路**：将 $Q, K \in \mathbb{R}^{n \times d}$ 先投影到 $r$ 维子空间（$r \ll d$），再计算注意力：
+
+$$\text{ProjAttn}(Q, K, V) = \text{softmax}\left(\frac{Q P_Q (K P_K)^T}{\sqrt{r}}\right) V$$
+
+其中 $P_Q, P_K \in \mathbb{R}^{d \times r}$ 为投影矩阵。
+
+**三种投影策略**：
+
+1. **可学习投影**（$P$ 为训练参数）：
+```
+P_Q = Linear(d, r, bias=False)  # r << d
+P_K = Linear(d, r, bias=False)
+scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
+attn = softmax(scores) @ V
+```
+
+2. **随机固定投影**（JL 引理保证）：
+```
+P = random_gaussian(d, r) / sqrt(r)  # 固定不训练
+scores = (Q @ P) @ (K @ P).T / sqrt(r)
+```
+
+3. **数据自适应投影**（在线 PCA）：
+```
+# 维护 K 的 running covariance，取 top-r 特征向量
+C = running_mean(K^T @ K)  # d x d
+eigenvecs = top_r_eigenvectors(C)  # d x r
+scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
+```
+
+## 可实现结构
+- **Multi-Head 投影**：每个 head 使用不同的 $P_h \in \mathbb{R}^{d_h \times r}$，总计算量 $O(n \cdot d \cdot r + n^2 \cdot r)$，当 $r \ll d$ 时节省 $Q K^T$ 的 $O(n^2 d)$ 开销
+- **KV-Cache 压缩**：投影后的 $K' = K P_K \in \mathbb{R}^{n \times r}$ 替代原始 $K$ 存储，Key 部分显存节省约 $d/r$ 倍（完整 KV-Cache 压缩需额外设计 V 的压缩/重构机制）
+- **分层投影**：浅层用小 $r$（粗筛），深层用大 $r$（精排）
+
+## GPU 可行性
+- **D1[v]**：投影 = 矩阵乘，注意力 = 矩阵乘链，全程张量运算
+- **D2[v]**：$Q P_Q$ 和 $K P_K$ 均为标准 GEMM，吃满 Tensor Core
+- **D3[v]**：投影 $O(ndr)$ 远低于注意力 $O(n^2 d)$，且投影后 $r \ll d$ 使注意力 $O(n^2 r)$
+- **D4[v]**：Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），直接降低显存峰值
+- **D5[v]**：投影矩阵正交/近正交时数值稳定，bf16 可接受
+- **D6[v]**：投影可与注意力流水线并行，Multi-Head 天然跨头并行
+- **D7[~]**：投影矩阵本身稠密；若用稀疏投影（CountSketch），可能引入 gather/scatter
+- **D8[~]可改造需验证**：投影可融入 FlashAttention 的 online softmax 循环中
+
+## 论文表述方式
+"我们将注意力计算分解为低维子空间投影与投影空间注意力两步，在保持注意力质量的同时将 Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），并保证 Johnson-Lindenstrauss 距离保持性。"
+
+## 风险
+- **投影方向退化**：可学习投影可能坍缩到少数方向（$P^T P$ 条件数恶化），导致注意力分布退化。需加正交正则化 $\|P^T P - I\|_F^2$。
+- **信息丢失不可逆**：投影丢弃了 $(d-r)$ 维信息，若任务恰需这些维度的特征则性能下降。建议配合残差连接（原始注意力 + 投影注意力的加权组合）。

package/design-patterns/attention/spectral-attention.en.md

@@ -0,0 +1,71 @@
+# Spectral Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the input signal exhibits **frequency-domain/spectral structure** (periodicity, cyclic symmetry, graph structure), computing attention in the spectral domain rather than the spatial domain can dramatically reduce complexity while exploiting the signal's intrinsic structure. Typical scenarios include: time series forecasting (periodic signals), graph neural networks (graph Laplacian spectral decomposition), positional encoding (frequency-domain interpretation of RoPE/ALiBi), and $O(n \log n)$ acceleration of long-sequence attention.
+
+## Mathematical Inspiration
+- Lenses: [duality (frequency-domain transform), symmetry (cyclic/translation invariance)]
+- Knowledge: [`../../knowledge-base/probability/entropy.md` (spectral entropy for measuring signal complexity), `../../knowledge-base/probability/concentration-inequality.md` (frequency-domain concentration inequalities)]
+
+## Required Mathematical Knowledge
+- **Discrete Fourier Transform (DFT/FFT)**: $O(n \log n)$ frequency-domain transform and the cyclic convolution theorem
+- **Graph Laplacian Spectral Decomposition**: $L = U \Lambda U^T$, where $U$ is the graph Fourier basis
+- **Irreducible Representations of the Cyclic Group $\mathbb{Z}_n$**: The DFT matrix is precisely the representation matrix of the cyclic group (see `references/books/abstract-algebra.md` Ch.4, Ch.11)
+
+## AI Module Form
+
+**Core Idea**: Transform attention from the spatial-domain $Q K^T$ into diagonal/sparse operations in the spectral domain:
+
+**Scheme A: FFT-Accelerated Attention (Time Series)**:
+```python
+# Express cyclic convolution as element-wise multiplication in the frequency domain
+Q_hat = fft(Q, dim=seq)        # (n, d) -> (n, d) frequency domain
+K_hat = fft(K, dim=seq)
+# Attention ~ frequency-domain filtering: independent weighting per frequency component
+attn_hat = Q_hat * conj(K_hat)  # element-wise multiply = cyclic convolution
+attn = ifft(attn_hat, dim=seq)
+# Complexity: O(n log n * d) vs. standard O(n^2 * d)
+```
+
+**Scheme B: Graph Spectral Attention (GNN)**:
+```python
+# Precompute graph Laplacian spectral decomposition L = U Lambda U^T (offline)
+U = eigenvectors(L)  # (n, k), take top-k low-frequency eigenvectors
+# Spectral-domain attention: compute in the low-frequency subspace
+Q_spec = U^T @ Q   # (k, d) project to spectral domain
+K_spec = U^T @ K
+scores = (Q_spec @ W_q) @ (K_spec @ W_k).T / sqrt(d)
+attn_spec = softmax(scores) @ (U^T @ V)
+output = U @ attn_spec  # back-project to spatial domain
+```
+
+**Scheme C: Frequency-Adaptive Attention Weights**:
+```python
+freq_weights = learnable_parameter(num_freq_bands)  # learnable spectral weights
+Q_hat, K_hat = fft(Q), fft(K)
+scores_freq = freq_weights.unsqueeze(-1) * (Q_hat * conj(K_hat))
+attn = ifft(scores_freq)
+```
+
+## Implementable Architectures
+- **Spectral Transformer**: Replace $O(n^2)$ attention with FFT, suitable for periodic sequence data (meteorological, financial, audio)
+- **Graph Spectral Attention**: Leverage the first $k$ eigenvectors of the graph Laplacian for low-dimensional attention, suitable for large-scale graphs ($n > 10^5$)
+- **Frequency-Aware Positional Encoding**: The essence of RoPE is the unitary representation of the cyclic group $\mathbb{Z}$ (see Abstract Algebra Ch.4), generalizable to other groups
+
+## GPU Feasibility
+- **D1**: FFT and matrix multiplication are both standard tensor operations
+- **D2**: Spectral projection $U^T Q$ is a standard GEMM; although FFT is not GEMM, highly optimized cuFFT implementations are available
+- **D3**: FFT attention $O(n \log n \cdot d)$, far superior to $O(n^2 d)$
+- **D4**: Frequency-domain representation introduces no extra dimensions; spectral projection can reduce to $k \ll n$ dimensions
+- **D5**: Complex-valued FFT suffers precision loss under fp16; fp32 or real-valued FFT (RFFT) is required
+- **D6**: FFT can be parallelized across batch/head; cuFFT supports multi-stream execution
+- **D7**: High-frequency components can be truncated in the spectral domain (structured sparsity), retaining only top-k frequencies
+- **D8**: Fusing FFT with attention requires custom kernels; no ready-made fusion exists in standard libraries
+
+## Paper Phrasing
+"We propose a spectral-domain attention mechanism that transforms attention computation into the Fourier/Laplacian spectral domain, leveraging the cyclic convolution theorem to reduce sequence attention complexity from $O(n^2)$ to $O(n \log n)$ while preserving the ability to model dependencies at multiple scales through frequency-adaptive weights."
+
+## Risks
+- **Violation of Periodicity Assumption**: FFT implicitly assumes periodic boundary conditions, causing spectral leakage for non-periodic signals such as natural language. Windowing functions or zero-padding are needed.
+- **Graph Laplacian Precomputation Cost**: Eigendecomposition $O(n^3)$ is infeasible for large-scale graphs, requiring approximations (Nystrom/Lanczos), and dynamic graphs necessitate recomputation.

package/design-patterns/attention/spectral-attention.md

@@ -0,0 +1,71 @@
+# 谱注意力 / Spectral Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入信号具有**频域/谱结构**（周期性、循环对称性、图结构）时，在谱域而非空域计算注意力可大幅降低复杂度并利用信号的内在结构。典型场景：时间序列预测（周期性信号）、图神经网络（图 Laplacian 谱分解）、位置编码（RoPE/ALiBi 的频率解释）、长序列注意力的 $O(n \log n)$ 加速。
+
+## 数学思想来源
+- 透镜：[duality（对偶透镜 — 频域变换）, symmetry（对称透镜 — 循环/平移不变性）]
+- 知识：[`../../knowledge-base/probability/entropy.md`（谱熵度量信号复杂度）, `../../knowledge-base/probability/concentration-inequality.md`（频域浓度不等式）]
+
+## 需要的数学知识
+- **离散傅里叶变换 DFT / FFT**：$O(n \log n)$ 频域变换，循环卷积定理
+- **图 Laplacian 谱分解**：$L = U \Lambda U^T$，$U$ 为图 Fourier 基
+- **循环群 $\mathbb{Z}_n$ 的不可约表示**：DFT 矩阵即循环群的表示矩阵（参见 `references/books/abstract-algebra.md` Ch.4, Ch.11）
+
+## AI 模块形式
+
+**核心思路**：将注意力从空域的 $Q K^T$ 转换到谱域的对角/稀疏运算：
+
+**方案 A：FFT 加速注意力（时间序列）**：
+```python
+# 将循环卷积写成谱域逐元素乘
+Q_hat = fft(Q, dim=seq)        # (n, d) -> (n, d) 频域
+K_hat = fft(K, dim=seq)
+# 注意力 ≈ 频域滤波：每个频率分量独立加权
+attn_hat = Q_hat * conj(K_hat)  # 逐元素乘 = 循环卷积
+attn = ifft(attn_hat, dim=seq)
+# 复杂度：O(n log n * d) vs 标准 O(n^2 * d)
+```
+
+**方案 B：图谱注意力（GNN）**：
+```python
+# 预计算图 Laplacian 谱分解 L = U Λ U^T（离线）
+U = eigenvectors(L)  # (n, k), 取 top-k 低频特征向量
+# 谱域注意力：在低频子空间中计算
+Q_spec = U^T @ Q   # (k, d) 投影到谱域
+K_spec = U^T @ K
+scores = (Q_spec @ W_q) @ (K_spec @ W_k).T / sqrt(d)
+attn_spec = softmax(scores) @ (U^T @ V)
+output = U @ attn_spec  # 反投影回空域
+```
+
+**方案 C：频率自适应注意力权重**：
+```python
+freq_weights = learnable_parameter(num_freq_bands)  # 可学习频谱权重
+Q_hat, K_hat = fft(Q), fft(K)
+scores_freq = freq_weights.unsqueeze(-1) * (Q_hat * conj(K_hat))
+attn = ifft(scores_freq)
+```
+
+## 可实现结构
+- **Spectral Transformer**：用 FFT 替代 $O(n^2)$ 注意力，适合周期性序列数据（气象、金融、音频）
+- **Graph Spectral Attention**：利用图 Laplacian 的前 $k$ 个特征向量做低维注意力，适合大规模图（$n > 10^5$）
+- **频率感知位置编码**：RoPE 的本质即循环群 $\mathbb{Z}$ 的酉表示（参见抽象代数 Ch.4），可推广到其他群
+
+## GPU 可行性
+- **D1[v]**：FFT 和矩阵乘均为标准张量运算
+- **D2[v]**：谱投影 $U^T Q$ 为标准 GEMM；FFT 虽非 GEMM 但有高度优化的 cuFFT 实现
+- **D3[v]**：FFT 注意力 $O(n \log n \cdot d)$，远优于 $O(n^2 d)$
+- **D4[v]**：频域表示不增加额外维度，谱投影可降低到 $k \ll n$ 维
+- **D5[~]**：复数 FFT 在 fp16 下精度损失，需 fp32 或使用实数 FFT (RFFT)
+- **D6[v]**：FFT 可跨 batch/head 并行，cuFFT 支持多流
+- **D7[v]**：谱域中高频分量可截断（结构化稀疏），只保留 top-k 频率
+- **D8[~]**：FFT 与 attention 融合需自定义 kernel，标准库无现成融合
+
+## 论文表述方式
+"我们提出谱域注意力机制，通过将注意力计算转换到 Fourier/Laplacian 谱域，利用循环卷积定理将序列注意力复杂度从 $O(n^2)$ 降至 $O(n \log n)$，同时通过频率自适应权重保留对不同尺度依赖关系的建模能力。"
+
+## 风险
+- **非周期性假设违反**：FFT 隐式假设周期性边界条件，对自然语言等非周期信号产生谱泄漏，需加窗函数或零填充。
+- **图 Laplacian 预计算成本**：大规模图的特征分解 $O(n^3)$ 不可行，需近似（Nyström/Lanczos），且动态图需重算。

package/design-patterns/compression/leverage-score-selection.en.md

@@ -0,0 +1,71 @@
+# Leverage Score Selection
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Target Problem
+Use when selecting the most representative rows / columns / tokens from a large-scale matrix while guaranteeing the precision of downstream linear algebra operations: KV-Cache token selection, data coreset construction, Nystrom landmark sampling, distributed gradient compression. Core objective: **sample based on statistical leverage scores derived from subspace projections, with probabilistic guarantees of approximating full-computation accuracy**.
+
+## Mathematical Foundations
+- Lenses: lenses/spectral.md (leverage scores = projection energy of row vectors onto the principal subspace), lenses/probabilistic.md (probabilistic sampling and concentration inequality guarantees), lenses/algorithmic.md (complexity--accuracy trade-offs of randomized algorithms)
+- Knowledge: knowledge-base/matrix-analysis/low-rank-approximation.md (randomized SVD, nuclear norm), knowledge-base/matrix-analysis/projection.md (diagonal entries of the projection matrix = leverage scores), knowledge-base/probability/concentration-inequality.md (Bernstein matrix concentration bound)
+
+## Required Mathematical Background
+- **Statistical Leverage Scores**: $\ell_i = \|(V_k V_k^T)_i\|^2 = (V_k V_k^T)_{ii}$, the projection energy of the $i$-th row onto the rank-$k$ subspace; $\sum_i \ell_i = k$
+- **Leverage Score Sampling Guarantee**: sampling $s = O(k \log k / \epsilon^2)$ rows with probabilities $p_i = \ell_i / k$ yields a $(1+\epsilon)$ approximation to full least squares (Drineas--Mahoney)
+- **Bernstein Matrix Bound**: after sampling, $\|\hat{A}^T \hat{A} - A^T A\|_2 \leq \epsilon \|A\|_F^2$ with probability $\geq 1-\delta$
+- **Fast Approximation**: $\tilde{\ell}_i = \|(A\Omega)_i\|^2$ ($\Omega$ random Gaussian), avoids full SVD, $O(Ndk)$
+- **DPP Extension**: Determinantal Point Process $P(S) \propto \det(L_S)$ adds a diversity guarantee on top of leverage scores
+
+## AI Module Specification
+```
+Module: LeverageScoreSelector
+Input: A ∈ R^{N×d}    Parameters: sample size s << N, rank parameter k
+
+Method 1 - Random projection leverage scores (online / large-scale):
+  Omega = randn(d, k+p)                       // random matrix, p=5 oversampling
+  Q = qr(A @ Omega)[0]                        // N×(k+p), GEMM + QR
+  leverage = sum(Q ** 2, dim=1)               // N-dim, row-wise sum of squares
+  indices = topk(leverage, s)                 // deterministic top-s selection
+  A_selected = A[indices]
+
+Method 2 - Exact leverage scores (offline / small matrices):
+  U_k = svd(A)[:k][0]                         // truncated SVD left singular vectors
+  leverage = sum(U_k ** 2, dim=1)             // exact rank-k leverage scores
+  probs = leverage / leverage.sum()
+  indices = multinomial_sample(N, s, probs)    // probabilistic sampling + reweighting
+  weights = 1 / sqrt(s * probs[indices])
+
+Method 3 - KV-Cache sliding-window eviction:
+  Every M steps update Q = qr(K_cache @ Omega)[0]
+  leverage = sum(Q ** 2, dim=1)
+  Evict tokens with lowest leverage (least contribution to the subspace)
+
+Method 4 - DPP greedy diverse selection:
+  scores = leverage.clone(); L = A @ A^T       // PSD kernel matrix
+  for _ in range(s):
+    idx = argmax(scores); selected.append(idx)
+    scores -= α * |L[:, idx]|                   // penalize neighbors of already selected tokens
+```
+
+## Implementable Architectures
+- **Random projection leverage layer**: 1 GEMM + 1 QR yields approximate leverage scores in $O(Ndk)$
+- **KV-Cache eviction policy**: evict by leverage score ranking, with stronger theoretical guarantees than attention-score-based eviction
+- **Coreset constructor**: leverage score sampling + importance reweighting, ensuring empirical risk approximates the full-data risk
+- **DPP greedy extension**: leverage scores + exclusion penalty, balancing importance and diversity
+
+## GPU Feasibility
+- Tensorization / GEMM: $A\Omega$ is a GEMM; QR via cuSOLVER; leverage scores = elementwise row-wise sum of squares
+- Complexity: $O(Ndk)$ is far superior to $O(Nd^2)$ full SVD; top-s selection in $O(N \log s)$
+- Memory: $\Omega$ is only $d \times k$ (KB-scale); $Q$ is the same size as $A$ but can be computed in batches
+- Low precision: QR recommended in fp32 (small matrix with $k+p$ columns, negligible overhead); leverage scores can be cast back to bf16
+- Parallelism: the $A\Omega$ GEMM is highly parallelizable; top-s can use parallel radix sort
+- Operator fusion: $A\Omega$ + QR + row-norm² can be fused to avoid materializing intermediate matrices
+
+## Paper-Worthy Formulation
+"We adopt statistical leverage scores as the importance metric for token selection: approximating rank-$k$ subspace leverage scores via random projection in $O(Ndk)$, with the Drineas--Mahoney theory guaranteeing that $O(k \log k / \epsilon^2)$ samples suffice for a $(1+\epsilon)$ approximation of the full subspace."
+
+## Risks
+- **Rank parameter $k$ selection**: leverage scores depend on the rank-$k$ subspace; an incorrect $k$ leads to biased sampling -- the effective rank must be diagnosed first
+- **Sampling variance**: probabilistic sampling introduces variance; low-probability rows occasionally selected produce large weight noise -- deterministic top-s can be used as an alternative
+- **Leverage scores vs. semantic importance mismatch**: the metric measures subspace contribution, which does not necessarily reflect semantics; it can be combined with attention scores via weighted fusion
+- **DPP greedy suboptimality**: exact DPP sampling is $O(N^3)$; the greedy approximation may miss the globally optimal subset
+- **Stale scores under dynamic data**: in streaming scenarios leverage scores drift as data evolves, requiring periodic recomputation or incremental updates