math-skill 2.0.1 → 3.0.0

package/design-patterns/attention/information-bottleneck-attention.en.md

@@ -0,0 +1,72 @@
+# Information Bottleneck Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the attention mechanism needs to **selectively transmit useful information while suppressing redundant/noisy information**, information bottleneck theory can guide the learning of attention weights -- maximizing the mutual information $I(Z;Y)$ of the attention distribution with respect to the target $Y$, while minimizing the mutual information $I(X;Z)$ with respect to the input $X$. Typical scenarios include: long-document summarization (filtering large numbers of irrelevant tokens), multimodal alignment (cross-modal noise suppression), and interpretability (attention weights as visualization of information flow).
+
+## Mathematical Inspiration
+- Lenses: [categorical (information-theoretic framework unifying attention design), variational (constrained variational and Lagrangian duality)]
+- Knowledge: [`../../knowledge-base/probability/information-bottleneck.md` (IB objective and variational lower bound), `../../knowledge-base/probability/kl-divergence.md` (implementation of KL regularization), `../../knowledge-base/probability/entropy.md` (mutual information estimation)]
+
+## Required Mathematical Knowledge
+- **Information Bottleneck Objective**: $\min I(X;Z) - \beta I(Z;Y)$, balancing compression and prediction
+- **Variational Information Bottleneck (VIB)**: Replacing intractable mutual information with variational lower bounds
+- **Correspondence Between Mutual Information and Attention**: Softmax attention weights $\alpha_{ij}$ can be interpreted as information channel allocations from key $j$ to query $i$
+
+## AI Module Form
+
+**Core Idea**: Treat attention as an information bottleneck -- attention weights determine "how much information flows from values to output," while KL regularization constrains the information throughput.
+
+**Scheme A: KL-Regularized Attention (Simplest IB Attention)**:
+```python
+# KL divergence between attention distribution q(z|x) and uniform prior as IB regularizer
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+uniform_prior = ones_like(attn) / n
+kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+loss = task_loss + beta * kl_reg
+output = attn @ V
+```
+
+**Scheme B: Variational Information Bottleneck Attention (VIB-Attention)**:
+```python
+# Introduce stochastic bottleneck Z ~ q(Z|attention) to limit information throughput
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+mu_z, log_var_z = attn, linear(attn)  # mean and learnable variance
+z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # reparameterization + softmax
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs prior
+output = z @ V
+loss = task_loss + beta * kl
+```
+
+**Scheme C: Mutual Information Maximization Attention (DIM Style)**:
+```python
+# Directly maximize I(Z;Y) via InfoNCE, with KL constraining I(X;Z)
+Z = softmax((Q @ K.T) / sqrt(d)) @ V
+info_nce = infonce_loss(Z, target_embedding, negatives, tau=0.1)
+kl_bottleneck = estimate_kl(X, Z)  # MINE/NWJ estimator
+loss = -info_nce + beta * kl_bottleneck
+```
+
+## Implementable Architectures
+- **IB-Sparse Attention**: KL regularization serves as a soft constraint on attention sparsity -- larger KL relative to uniform prior leads to more concentrated attention, yielding implicit Top-K selection
+- **IB Interpretation of Dropout**: Dropout is a form of stochastic information bottleneck -- randomly blocking information channels forces the model to learn robust representations. The dropout rate corresponds to the $\beta$ parameter
+- **Multi-Head Information Allocation**: Different heads learn different information bottlenecks (different $\beta$); some heads transmit global information, others only local information
+
+## GPU Feasibility
+- **Dimension 1 Tensorization**: KL regularization involves element-wise operations; VIB reparameterization sampling is element-wise
+- **Dimension 2 GEMM-mappability**: The main body $QK^T$ and $attn \cdot V$ are standard GEMM; regularization introduces no new GEMM operations
+- **Dimension 3 Complexity**: KL regularization is $O(n)$ per token, adding no asymptotic complexity
+- **Dimension 4 Memory**: VIB requires additional $\mu_z$ and $\log\sigma_z$, approximately doubling attention weight memory
+- **Dimension 5 Low Precision**: log/exp in KL computations are stable under bf16 (standard log-softmax tricks)
+- **Dimension 6 Parallelism**: Regularization can be computed in parallel with forward propagation, introducing no serial dependencies
+- **Dimension 7 Sparsity**: KL regularization implicitly induces attention sparsity, amenable to block-sparse acceleration
+- **Dimension 8 Operator Fusion**: KL can be fused into the softmax kernel (FusedSoftmaxKL)
+
+## Paper Phrasing
+"We propose information bottleneck attention, which models attention as an information bottleneck variational problem. By maximizing the mutual information between output and target while minimizing redundant information transmission from the input, it achieves theoretically optimal information selection and demonstrates enhanced sparsity and interpretability in experiments."
+
+## Risks
+- **High Variance of Mutual Information Estimators**: The MINE/NWJ/InfoNCE estimators in Scheme C exhibit high variance in high-dimensional spaces, potentially causing training instability. It is recommended to first validate the basic effect of IB attention with Scheme A (KL regularization) before attempting the full IB objective.
+- **Difficulty of $\beta$ Tuning**: $\beta$ controls the compression-prediction trade-off, and the optimal $\beta$ varies significantly across tasks. Too small a $\beta$ degenerates to standard attention; too large a $\beta$ causes underfitting. Adaptive $\beta$ scheduling or information plane monitoring is recommended.

package/design-patterns/attention/information-bottleneck-attention.md

@@ -0,0 +1,72 @@
+# 信息瓶颈注意力 / Information Bottleneck Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当注意力机制需要**选择性传递有用信息、抑制冗余/噪声信息**时，用信息瓶颈理论指导注意力权重的学习——使注意力分布最大化关于目标 $Y$ 的互信息 $I(Z;Y)$，同时最小化关于输入 $X$ 的互信息 $I(X;Z)$。典型场景：长文档摘要（大量无关 token 需被过滤）、多模态对齐（跨模态噪声抑制）、可解释性（注意力权重作为信息流的可视化）。
+
+## 数学思想来源
+- 透镜：[categorical（范畴化透镜 — 信息论框架统一注意力设计）, variational（变分透镜 — 约束优化与拉格朗日对偶）]
+- 知识：[`../../knowledge-base/probability/information-bottleneck.md`（IB 目标函数与变分下界）, `../../knowledge-base/probability/kl-divergence.md`（KL 正则项的实现）, `../../knowledge-base/probability/entropy.md`（互信息估计）]
+
+## 需要的数学知识
+- **信息瓶颈 IB 目标**：$\min I(X;Z) - \beta I(Z;Y)$，压缩-预测权衡
+- **变分信息瓶颈 VIB**：用变分下界替代难以估计的互信息
+- **互信息与注意力的对应**：softmax 注意力权重 $\alpha_{ij}$ 可解释为从 key $j$ 到 query $i$ 的信息信道分配
+
+## AI 模块形式
+
+**核心思路**：将注意力视为信息瓶颈——注意力权重决定"让多少信息从 value 流向输出"，同时用 KL 正则约束信息量不过大。
+
+**方案 A：KL 正则化注意力（最简 IB 注意力）**：
+```python
+# 注意力分布 q(z|x) 与均匀先验的 KL 散度作为 IB 正则
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+uniform_prior = ones_like(attn) / n
+kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+loss = task_loss + beta * kl_reg
+output = attn @ V
+```
+
+**方案 B：变分信息瓶颈注意力（VIB-Attention）**：
+```python
+# 引入随机瓶颈 Z ~ q(Z|attention)，限制信息通过量
+scores = (Q @ K.T) / sqrt(d)
+attn = softmax(scores)
+mu_z, log_var_z = attn, linear(attn)  # 均值与可学习方差
+z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # 重参数化 + softmax
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs 先验
+output = z @ V
+loss = task_loss + beta * kl
+```
+
+**方案 C：互信息最大化注意力（DIM 风格）**：
+```python
+# 用 InfoNCE 直接最大化 I(Z;Y)，配合 KL 约束 I(X;Z)
+Z = softmax((Q @ K.T) / sqrt(d)) @ V
+info_nce = infonce_loss(Z, target_embedding, negatives, tau=0.1)
+kl_bottleneck = estimate_kl(X, Z)  # MINE/NWJ 估计器
+loss = -info_nce + beta * kl_bottleneck
+```
+
+## 可实现结构
+- **IB-Sparse Attention**：将 KL 正则化为注意力稀疏性的软约束——KL 对均匀先越大 → 注意力越集中 → 隐式 Top-K 选择
+- **Dropout 的 IB 解释**：Dropout 是一种随机信息瓶颈——随机阻断部分信息通道，迫使模型学习鲁棒表示。Dropout rate 对应 $\beta$ 参数
+- **Multi-Head 信息分配**：不同 head 学习不同的信息瓶颈（不同 $\beta$），有的 head 传递全局信息，有的只传递局部信息
+
+## GPU 可行性
+- **维度 1 张量化 ✅**：KL 正则项为逐元素运算，VIB 重参数化采样为逐元素
+- **维度 2 GEMM 可映射 ✅**：主体 $QK^T$ 和 $attn \cdot V$ 为标准 GEMM，正则项不引入新 GEMM
+- **维度 3 复杂度 ✅**：KL 正则 $O(n)$ 逐 token，不增加渐近复杂度
+- **维度 4 显存 ⚠️**：VIB 需额外 $\mu_z$ 和 $\log\sigma_z$，约 2x 注意力权重显存
+- **维度 5 低精度 ✅**：KL 中 log/exp 在 bf16 下稳定（标准 log-softmax 技巧）
+- **维度 6 并行 ✅**：正则项可与前向传播并行，不引入串行依赖
+- **维度 7 稀疏 ✅**：KL 正则隐式诱导注意力稀疏性，可利用 block-sparse 加速
+- **维度 8 算子融合 ✅**：KL 可融入 softmax kernel（FusedSoftmaxKL）
+
+## 论文表述方式
+"我们提出信息瓶颈注意力机制，通过将注意力建模为信息瓶颈优化问题，在最大化输出与目标的互信息的同时最小化输入信息的冗余传递，实现理论上最优的信息选择，并在实验中展现出更强的稀疏性和可解释性。"
+
+## 风险
+- **互信息估计的方差问题**：方案 C 中的 MINE/NWJ/InfoNCE 估计器在高维空间中方差大，可能导致训练不稳定。建议先用方案 A（KL 正则）验证 IB 注意力的基本效果，再尝试完整 IB 目标。
+- **$\beta$ 调参困难**：$\beta$ 控制压缩-预测权衡，不同任务最优 $\beta$ 差异大。$\beta$ 过小退化为标准注意力，$\beta$ 过大导致欠拟合。建议自适应 $\beta$ 调度或信息平面监控。

package/design-patterns/attention/projection-attention.en.md

@@ -0,0 +1,68 @@
+# Projection Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the token/key space dimensionality is too high, the standard attention $Q K^T$ inner product tends to become uniform in high-dimensional spaces ("attention collapse"). In such cases, key/query vectors must first be projected onto a **subspace with superior geometric structure** before computing attention. Typical scenarios include: KV-Cache compression for long-context LLMs, attention sparsification in high-dimensional embedding spaces, and attention alignment of heterogeneous features in multimodal fusion.
+
+## Mathematical Inspiration
+- Lenses: [projection, spectral, probabilistic]
+- Knowledge: [`../../knowledge-base/probability/concentration-inequality.md` (high-dimensional concentration inequalities provide theoretical explanation for attention collapse), `../../knowledge-base/probability/entropy.md` (entropy of attention distributions as a quality metric)]
+
+## Required Mathematical Knowledge
+- **Johnson-Lindenstrauss Lemma**: High-dimensional point sets can be projected onto an $O(\log n / \epsilon^2)$-dimensional subspace while approximately preserving distances
+- **Random Projections and Subspace Embeddings**: Sparse projection matrices (e.g., CountSketch, SRHT) approximately preserve inner products
+- **SVD / PCA Truncation**: Optimal low-rank subspace projection that maximizes retained variance
+
+## AI Module Form
+
+**Core Idea**: Project $Q, K \in \mathbb{R}^{n \times d}$ onto an $r$-dimensional subspace ($r \ll d$) before computing attention:
+
+$$\text{ProjAttn}(Q, K, V) = \text{softmax}\left(\frac{Q P_Q (K P_K)^T}{\sqrt{r}}\right) V$$
+
+where $P_Q, P_K \in \mathbb{R}^{d \times r}$ are projection matrices.
+
+**Three Projection Strategies**:
+
+1. **Learnable Projection** ($P$ as trainable parameters):
+```
+P_Q = Linear(d, r, bias=False)  # r << d
+P_K = Linear(d, r, bias=False)
+scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
+attn = softmax(scores) @ V
+```
+
+2. **Random Fixed Projection** (guaranteed by the JL Lemma):
+```
+P = random_gaussian(d, r) / sqrt(r)  # fixed, not trained
+scores = (Q @ P) @ (K @ P).T / sqrt(r)
+```
+
+3. **Data-Adaptive Projection** (online PCA):
+```
+# Maintain running covariance of K, take top-r eigenvectors
+C = running_mean(K^T @ K)  # d x d
+eigenvecs = top_r_eigenvectors(C)  # d x r
+scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
+```
+
+## Implementable Architectures
+- **Multi-Head Projection**: Each head uses a different $P_h \in \mathbb{R}^{d_h \times r}$, with total computation $O(n \cdot d \cdot r + n^2 \cdot r)$. When $r \ll d$, this saves the $O(n^2 d)$ cost of $Q K^T$
+- **Key-cache Compression**: The projected $K' = K P_K \in \mathbb{R}^{n \times r}$ replaces the original $K$ in storage, reducing Key-cache memory by about $d/r$. Full KV-Cache compression requires a separate V compression/reconstruction mechanism
+- **Hierarchical Projection**: Shallow layers use small $r$ (coarse filtering), while deep layers use large $r$ (fine ranking)
+
+## GPU Feasibility
+- **Dimension 1 Tensorization**: Projection = matrix multiplication, attention = matrix multiplication chain; entirely tensor operations
+- **Dimension 2 GEMM-mappability**: $Q P_Q$ and $K P_K$ are both standard GEMM operations, fully utilizing Tensor Cores
+- **Dimension 3 Complexity**: Projection cost $O(ndr)$ is far below attention cost $O(n^2 d)$, and after projection $r \ll d$ reduces attention to $O(n^2 r)$
+- **Dimension 4 Memory**: Key-cache compressed by $d/r$; V-cache must be handled separately
+- **Dimension 5 Low Precision**: Projection matrices are orthogonal or near-orthogonal, yielding numerical stability; bf16 is acceptable
+- **Dimension 6 Parallelism**: Projection can be pipelined with attention; Multi-Head is naturally parallel across heads
+- **Dimension 7 Sparsity**: Projection matrices are inherently dense; sparse projections (CountSketch) may introduce gather/scatter operations
+- **Dimension 8 Operator Fusion ⚠️ Retrofittable, needs kernel-level validation**: Projection may be integrated into a FlashAttention-style kernel, but requires kernel-level verification
+
+## Paper Phrasing
+"We decompose attention computation into two steps -- low-dimensional subspace projection and projected-space attention -- achieving a $d/r$-fold Key-cache compression (full KV-Cache compression requires separate V compression/reconstruction) while preserving attention quality and guaranteeing Johnson-Lindenstrauss distance preservation."
+
+## Risks
+- **Projection Direction Degeneracy**: Learnable projections may collapse onto a few directions (deterioration of the condition number of $P^T P$), causing attention distributions to degenerate. Orthogonal regularization $\|P^T P - I\|_F^2$ is required.
+- **Irreversible Information Loss**: Projection discards $(d-r)$ dimensions of information; if the task relies on features in these dimensions, performance will degrade. It is recommended to combine with residual connections (weighted combination of original attention and projected attention).

package/design-patterns/attention/projection-attention.md

@@ -0,0 +1,68 @@
+# 投影注意力 / Projection Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入的 token/key 空间维度过高，标准注意力的 $Q K^T$ 内积在高维空间中趋于均匀（"注意力坍缩"）时，需要先将 key/query 投影到一个**几何结构更优的子空间**再做注意力计算。典型场景：长上下文 LLM 的 KV-Cache 压缩、高维嵌入空间的注意力稀疏化、多模态融合中异构特征的注意力对齐。
+
+## 数学思想来源
+- 透镜：[projection（投影与分解）, spectral（谱分解）, probabilistic（概率）]
+- 知识：[`../../knowledge-base/probability/concentration-inequality.md`（高维浓度不等式 → 注意力坍缩的理论解释）, `../../knowledge-base/probability/entropy.md`（注意力分布的熵作为质量指标）]
+
+## 需要的数学知识
+- **Johnson-Lindenstrauss 引理**：高维点集可投影到 $O(\log n / \epsilon^2)$ 维子空间而近似保持距离
+- **随机投影与子空间嵌入**：稀疏投影矩阵（如 CountSketch、SRHT）近似保持内积
+- **SVD / PCA 截断**：最优低秩子空间投影，最大化保留方差
+
+## AI 模块形式
+
+**核心思路**：将 $Q, K \in \mathbb{R}^{n \times d}$ 先投影到 $r$ 维子空间（$r \ll d$），再计算注意力：
+
+$$\text{ProjAttn}(Q, K, V) = \text{softmax}\left(\frac{Q P_Q (K P_K)^T}{\sqrt{r}}\right) V$$
+
+其中 $P_Q, P_K \in \mathbb{R}^{d \times r}$ 为投影矩阵。
+
+**三种投影策略**：
+
+1. **可学习投影**（$P$ 为训练参数）：
+```
+P_Q = Linear(d, r, bias=False)  # r << d
+P_K = Linear(d, r, bias=False)
+scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
+attn = softmax(scores) @ V
+```
+
+2. **随机固定投影**（JL 引理保证）：
+```
+P = random_gaussian(d, r) / sqrt(r)  # 固定不训练
+scores = (Q @ P) @ (K @ P).T / sqrt(r)
+```
+
+3. **数据自适应投影**（在线 PCA）：
+```
+# 维护 K 的 running covariance，取 top-r 特征向量
+C = running_mean(K^T @ K)  # d x d
+eigenvecs = top_r_eigenvectors(C)  # d x r
+scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
+```
+
+## 可实现结构
+- **Multi-Head 投影**：每个 head 使用不同的 $P_h \in \mathbb{R}^{d_h \times r}$，总计算量 $O(n \cdot d \cdot r + n^2 \cdot r)$，当 $r \ll d$ 时节省 $Q K^T$ 的 $O(n^2 d)$ 开销
+- **KV-Cache 压缩**：投影后的 $K' = K P_K \in \mathbb{R}^{n \times r}$ 替代原始 $K$ 存储，Key 部分显存节省约 $d/r$ 倍（完整 KV-Cache 压缩需额外设计 V 的压缩/重构机制）
+- **分层投影**：浅层用小 $r$（粗筛），深层用大 $r$（精排）
+
+## GPU 可行性
+- **维度 1 张量化 ✅**：投影 = 矩阵乘，注意力 = 矩阵乘链，全程张量运算
+- **维度 2 GEMM 可映射 ✅**：$Q P_Q$ 和 $K P_K$ 均为标准 GEMM，吃满 Tensor Core
+- **维度 3 复杂度 ✅**：投影 $O(ndr)$ 远低于注意力 $O(n^2 d)$，且投影后 $r \ll d$ 使注意力 $O(n^2 r)$
+- **维度 4 显存 ✅**：Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），直接降低显存峰值
+- **维度 5 低精度 ✅**：投影矩阵正交/近正交时数值稳定，bf16 可接受
+- **维度 6 并行 ✅**：投影可与注意力流水线并行，Multi-Head 天然跨头并行
+- **维度 7 稀疏 ⚠️**：投影矩阵本身稠密；若用稀疏投影（CountSketch），可能引入 gather/scatter
+- **维度 8 算子融合 ⚠️可改造需验证**：投影可融入 FlashAttention 的 online softmax 循环中
+
+## 论文表述方式
+"我们将注意力计算分解为低维子空间投影与投影空间注意力两步，在保持注意力质量的同时将 Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），并保证 Johnson-Lindenstrauss 距离保持性。"
+
+## 风险
+- **投影方向退化**：可学习投影可能坍缩到少数方向（$P^T P$ 条件数恶化），导致注意力分布退化。需加正交正则化 $\|P^T P - I\|_F^2$。
+- **信息丢失不可逆**：投影丢弃了 $(d-r)$ 维信息，若任务恰需这些维度的特征则性能下降。建议配合残差连接（原始注意力 + 投影注意力的加权组合）。

package/design-patterns/attention/spectral-attention.en.md

@@ -0,0 +1,71 @@
+# Spectral Attention
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Applicable Problems
+When the input signal exhibits **frequency-domain/spectral structure** (periodicity, cyclic symmetry, graph structure), computing attention in the spectral domain rather than the spatial domain can dramatically reduce complexity while exploiting the signal's intrinsic structure. Typical scenarios include: time series forecasting (periodic signals), graph neural networks (graph Laplacian spectral decomposition), positional encoding (frequency-domain interpretation of RoPE/ALiBi), and $O(n \log n)$ acceleration of long-sequence attention.
+
+## Mathematical Inspiration
+- Lenses: [duality (frequency-domain transform), symmetry (cyclic/translation invariance)]
+- Knowledge: [`../../knowledge-base/probability/entropy.md` (spectral entropy for measuring signal complexity), `../../knowledge-base/probability/concentration-inequality.md` (frequency-domain concentration inequalities)]
+
+## Required Mathematical Knowledge
+- **Discrete Fourier Transform (DFT/FFT)**: $O(n \log n)$ frequency-domain transform and the cyclic convolution theorem
+- **Graph Laplacian Spectral Decomposition**: $L = U \Lambda U^T$, where $U$ is the graph Fourier basis
+- **Irreducible Representations of the Cyclic Group $\mathbb{Z}_n$**: The DFT matrix is precisely the representation matrix of the cyclic group (see `references/books/abstract-algebra.md` Ch.4, Ch.11)
+
+## AI Module Form
+
+**Core Idea**: Transform attention from the spatial-domain $Q K^T$ into diagonal/sparse operations in the spectral domain:
+
+**Scheme A: FFT-Accelerated Attention (Time Series)**:
+```python
+# Express cyclic convolution as element-wise multiplication in the frequency domain
+Q_hat = fft(Q, dim=seq)        # (n, d) -> (n, d) frequency domain
+K_hat = fft(K, dim=seq)
+# Attention ~ frequency-domain filtering: independent weighting per frequency component
+attn_hat = Q_hat * conj(K_hat)  # element-wise multiply = cyclic convolution
+attn = ifft(attn_hat, dim=seq)
+# Complexity: O(n log n * d) vs. standard O(n^2 * d)
+```
+
+**Scheme B: Graph Spectral Attention (GNN)**:
+```python
+# Precompute graph Laplacian spectral decomposition L = U Lambda U^T (offline)
+U = eigenvectors(L)  # (n, k), take top-k low-frequency eigenvectors
+# Spectral-domain attention: compute in the low-frequency subspace
+Q_spec = U^T @ Q   # (k, d) project to spectral domain
+K_spec = U^T @ K
+scores = (Q_spec @ W_q) @ (K_spec @ W_k).T / sqrt(d)
+attn_spec = softmax(scores) @ (U^T @ V)
+output = U @ attn_spec  # back-project to spatial domain
+```
+
+**Scheme C: Frequency-Adaptive Attention Weights**:
+```python
+freq_weights = learnable_parameter(num_freq_bands)  # learnable spectral weights
+Q_hat, K_hat = fft(Q), fft(K)
+scores_freq = freq_weights.unsqueeze(-1) * (Q_hat * conj(K_hat))
+attn = ifft(scores_freq)
+```
+
+## Implementable Architectures
+- **Spectral Transformer**: Replace $O(n^2)$ attention with FFT, suitable for periodic sequence data (meteorological, financial, audio)
+- **Graph Spectral Attention**: Leverage the first $k$ eigenvectors of the graph Laplacian for low-dimensional attention, suitable for large-scale graphs ($n > 10^5$)
+- **Frequency-Aware Positional Encoding**: The essence of RoPE is the unitary representation of the cyclic group $\mathbb{Z}$ (see Abstract Algebra Ch.4), generalizable to other groups
+
+## GPU Feasibility
+- **Dimension 1 Tensorization**: FFT and matrix multiplication are both standard tensor operations
+- **Dimension 2 GEMM-mappability**: Spectral projection $U^T Q$ is a standard GEMM; although FFT is not GEMM, highly optimized cuFFT implementations are available
+- **Dimension 3 Complexity**: FFT attention $O(n \log n \cdot d)$, far superior to $O(n^2 d)$
+- **Dimension 4 Memory**: Frequency-domain representation introduces no extra dimensions; spectral projection can reduce to $k \ll n$ dimensions
+- **Dimension 5 Low Precision**: Complex-valued FFT suffers precision loss under fp16; fp32 or real-valued FFT (RFFT) is required
+- **Dimension 6 Parallelism**: FFT can be parallelized across batch/head; cuFFT supports multi-stream execution
+- **Dimension 7 Sparsity**: High-frequency components can be truncated in the spectral domain (structured sparsity), retaining only top-k frequencies
+- **Dimension 8 Operator Fusion**: Fusing FFT with attention requires custom kernels; no ready-made fusion exists in standard libraries
+
+## Paper Phrasing
+"We propose a spectral-domain attention mechanism that transforms attention computation into the Fourier/Laplacian spectral domain, leveraging the cyclic convolution theorem to reduce sequence attention complexity from $O(n^2)$ to $O(n \log n)$ while preserving the ability to model dependencies at multiple scales through frequency-adaptive weights."
+
+## Risks
+- **Violation of Periodicity Assumption**: FFT implicitly assumes periodic boundary conditions, causing spectral leakage for non-periodic signals such as natural language. Windowing functions or zero-padding are needed.
+- **Graph Laplacian Precomputation Cost**: Eigendecomposition $O(n^3)$ is infeasible for large-scale graphs, requiring approximations (Nystrom/Lanczos), and dynamic graphs necessitate recomputation.

package/design-patterns/attention/spectral-attention.md

@@ -0,0 +1,71 @@
+# 谱注意力 / Spectral Attention
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当输入信号具有**频域/谱结构**（周期性、循环对称性、图结构）时，在谱域而非空域计算注意力可大幅降低复杂度并利用信号的内在结构。典型场景：时间序列预测（周期性信号）、图神经网络（图 Laplacian 谱分解）、位置编码（RoPE/ALiBi 的频率解释）、长序列注意力的 $O(n \log n)$ 加速。
+
+## 数学思想来源
+- 透镜：[duality（对偶透镜 — 频域变换）, symmetry（对称透镜 — 循环/平移不变性）]
+- 知识：[`../../knowledge-base/probability/entropy.md`（谱熵度量信号复杂度）, `../../knowledge-base/probability/concentration-inequality.md`（频域浓度不等式）]
+
+## 需要的数学知识
+- **离散傅里叶变换 DFT / FFT**：$O(n \log n)$ 频域变换，循环卷积定理
+- **图 Laplacian 谱分解**：$L = U \Lambda U^T$，$U$ 为图 Fourier 基
+- **循环群 $\mathbb{Z}_n$ 的不可约表示**：DFT 矩阵即循环群的表示矩阵（参见 `references/books/abstract-algebra.md` Ch.4, Ch.11）
+
+## AI 模块形式
+
+**核心思路**：将注意力从空域的 $Q K^T$ 转换到谱域的对角/稀疏运算：
+
+**方案 A：FFT 加速注意力（时间序列）**：
+```python
+# 将循环卷积写成谱域逐元素乘
+Q_hat = fft(Q, dim=seq)        # (n, d) -> (n, d) 频域
+K_hat = fft(K, dim=seq)
+# 注意力 ≈ 频域滤波：每个频率分量独立加权
+attn_hat = Q_hat * conj(K_hat)  # 逐元素乘 = 循环卷积
+attn = ifft(attn_hat, dim=seq)
+# 复杂度：O(n log n * d) vs 标准 O(n^2 * d)
+```
+
+**方案 B：图谱注意力（GNN）**：
+```python
+# 预计算图 Laplacian 谱分解 L = U Λ U^T（离线）
+U = eigenvectors(L)  # (n, k), 取 top-k 低频特征向量
+# 谱域注意力：在低频子空间中计算
+Q_spec = U^T @ Q   # (k, d) 投影到谱域
+K_spec = U^T @ K
+scores = (Q_spec @ W_q) @ (K_spec @ W_k).T / sqrt(d)
+attn_spec = softmax(scores) @ (U^T @ V)
+output = U @ attn_spec  # 反投影回空域
+```
+
+**方案 C：频率自适应注意力权重**：
+```python
+freq_weights = learnable_parameter(num_freq_bands)  # 可学习频谱权重
+Q_hat, K_hat = fft(Q), fft(K)
+scores_freq = freq_weights.unsqueeze(-1) * (Q_hat * conj(K_hat))
+attn = ifft(scores_freq)
+```
+
+## 可实现结构
+- **Spectral Transformer**：用 FFT 替代 $O(n^2)$ 注意力，适合周期性序列数据（气象、金融、音频）
+- **Graph Spectral Attention**：利用图 Laplacian 的前 $k$ 个特征向量做低维注意力，适合大规模图（$n > 10^5$）
+- **频率感知位置编码**：RoPE 的本质即循环群 $\mathbb{Z}$ 的酉表示（参见抽象代数 Ch.4），可推广到其他群
+
+## GPU 可行性
+- **维度 1 张量化 ✅**：FFT 和矩阵乘均为标准张量运算
+- **维度 2 GEMM 可映射 ✅**：谱投影 $U^T Q$ 为标准 GEMM；FFT 虽非 GEMM 但有高度优化的 cuFFT 实现
+- **维度 3 复杂度 ✅**：FFT 注意力 $O(n \log n \cdot d)$，远优于 $O(n^2 d)$
+- **维度 4 显存 ✅**：频域表示不增加额外维度，谱投影可降低到 $k \ll n$ 维
+- **维度 5 低精度 ⚠️**：复数 FFT 在 fp16 下精度损失，需 fp32 或使用实数 FFT (RFFT)
+- **维度 6 并行 ✅**：FFT 可跨 batch/head 并行，cuFFT 支持多流
+- **维度 7 稀疏 ✅**：谱域中高频分量可截断（结构化稀疏），只保留 top-k 频率
+- **维度 8 算子融合 ⚠️**：FFT 与 attention 融合需自定义 kernel，标准库无现成融合
+
+## 论文表述方式
+"我们提出谱域注意力机制，通过将注意力计算转换到 Fourier/Laplacian 谱域，利用循环卷积定理将序列注意力复杂度从 $O(n^2)$ 降至 $O(n \log n)$，同时通过频率自适应权重保留对不同尺度依赖关系的建模能力。"
+
+## 风险
+- **非周期性假设违反**：FFT 隐式假设周期性边界条件，对自然语言等非周期信号产生谱泄漏，需加窗函数或零填充。
+- **图 Laplacian 预计算成本**：大规模图的特征分解 $O(n^3)$ 不可行，需近似（Nyström/Lanczos），且动态图需重算。

package/design-patterns/compression/leverage-score-selection.en.md

@@ -0,0 +1,71 @@
+# Leverage Score Selection
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+
+## Target Problem
+Use when selecting the most representative rows / columns / tokens from a large-scale matrix while guaranteeing the precision of downstream linear algebra operations: KV-Cache token selection, data coreset construction, Nystrom landmark sampling, distributed gradient compression. Core objective: **sample based on statistical leverage scores derived from subspace projections, with probabilistic guarantees of approximating full-computation accuracy**.
+
+## Mathematical Foundations
+- Lenses: lenses/spectral.md (leverage scores = projection energy of row vectors onto the principal subspace), lenses/probabilistic.md (probabilistic sampling and concentration inequality guarantees), lenses/algorithmic.md (complexity--accuracy trade-offs of randomized algorithms)
+- Knowledge: knowledge-base/matrix-analysis/low-rank-approximation.md (randomized SVD, nuclear norm), knowledge-base/matrix-analysis/projection.md (diagonal entries of the projection matrix = leverage scores), knowledge-base/probability/concentration-inequality.md (Bernstein matrix concentration bound)
+
+## Required Mathematical Background
+- **Statistical Leverage Scores**: $\ell_i = \|(V_k V_k^T)_i\|^2 = (V_k V_k^T)_{ii}$, the projection energy of the $i$-th row onto the rank-$k$ subspace; $\sum_i \ell_i = k$
+- **Leverage Score Sampling Guarantee**: sampling $s = O(k \log k / \epsilon^2)$ rows with probabilities $p_i = \ell_i / k$ yields a $(1+\epsilon)$ approximation to full least squares (Drineas--Mahoney)
+- **Bernstein Matrix Bound**: after sampling, $\|\hat{A}^T \hat{A} - A^T A\|_2 \leq \epsilon \|A\|_F^2$ with probability $\geq 1-\delta$
+- **Fast Approximation**: $\tilde{\ell}_i = \|(A\Omega)_i\|^2$ ($\Omega$ random Gaussian), avoids full SVD, $O(Ndk)$
+- **DPP Extension**: Determinantal Point Process $P(S) \propto \det(L_S)$ adds a diversity guarantee on top of leverage scores
+
+## AI Module Specification
+```
+Module: LeverageScoreSelector
+Input: A ∈ R^{N×d}    Parameters: sample size s << N, rank parameter k
+
+Method 1 - Random projection leverage scores (online / large-scale):
+  Omega = randn(d, k+p)                       // random matrix, p=5 oversampling
+  Q = qr(A @ Omega)[0]                        // N×(k+p), GEMM + QR
+  leverage = sum(Q ** 2, dim=1)               // N-dim, row-wise sum of squares
+  indices = topk(leverage, s)                 // deterministic top-s selection
+  A_selected = A[indices]
+
+Method 2 - Exact leverage scores (offline / small matrices):
+  U_k = svd(A)[:k][0]                         // truncated SVD left singular vectors
+  leverage = sum(U_k ** 2, dim=1)             // exact rank-k leverage scores
+  probs = leverage / leverage.sum()
+  indices = multinomial_sample(N, s, probs)    // probabilistic sampling + reweighting
+  weights = 1 / sqrt(s * probs[indices])
+
+Method 3 - KV-Cache sliding-window eviction:
+  Every M steps update Q = qr(K_cache @ Omega)[0]
+  leverage = sum(Q ** 2, dim=1)
+  Evict tokens with lowest leverage (least contribution to the subspace)
+
+Method 4 - DPP greedy diverse selection:
+  scores = leverage.clone(); L = A @ A^T       // PSD kernel matrix
+  for _ in range(s):
+    idx = argmax(scores); selected.append(idx)
+    scores -= α * |L[:, idx]|                   // penalize neighbors of already selected tokens
+```
+
+## Implementable Architectures
+- **Random projection leverage layer**: 1 GEMM + 1 QR yields approximate leverage scores in $O(Ndk)$
+- **KV-Cache eviction policy**: evict by leverage score ranking, with stronger theoretical guarantees than attention-score-based eviction
+- **Coreset constructor**: leverage score sampling + importance reweighting, ensuring empirical risk approximates the full-data risk
+- **DPP greedy extension**: leverage scores + exclusion penalty, balancing importance and diversity
+
+## GPU Feasibility
+- Tensorization / GEMM: $A\Omega$ is a GEMM; QR via cuSOLVER; leverage scores = elementwise row-wise sum of squares
+- Complexity: $O(Ndk)$ is far superior to $O(Nd^2)$ full SVD; top-s selection in $O(N \log s)$
+- Memory: $\Omega$ is only $d \times k$ (KB-scale); $Q$ is the same size as $A$ but can be computed in batches
+- Low precision: QR recommended in fp32 (small matrix with $k+p$ columns, negligible overhead); leverage scores can be cast back to bf16
+- Parallelism: the $A\Omega$ GEMM is highly parallelizable; top-s can use parallel radix sort
+- Operator fusion: $A\Omega$ + QR + row-norm² can be fused to avoid materializing intermediate matrices
+
+## Paper-Worthy Formulation
+"We adopt statistical leverage scores as the importance metric for token selection: approximating rank-$k$ subspace leverage scores via random projection in $O(Ndk)$, with the Drineas--Mahoney theory guaranteeing that $O(k \log k / \epsilon^2)$ samples suffice for a $(1+\epsilon)$ approximation of the full subspace."
+
+## Risks
+- **Rank parameter $k$ selection**: leverage scores depend on the rank-$k$ subspace; an incorrect $k$ leads to biased sampling -- the effective rank must be diagnosed first
+- **Sampling variance**: probabilistic sampling introduces variance; low-probability rows occasionally selected produce large weight noise -- deterministic top-s can be used as an alternative
+- **Leverage scores vs. semantic importance mismatch**: the metric measures subspace contribution, which does not necessarily reflect semantics; it can be combined with attention scores via weighted fusion
+- **DPP greedy suboptimality**: exact DPP sampling is $O(N^3)$; the greedy approximation may miss the globally optimal subset
+- **Stale scores under dynamic data**: in streaming scenarios leverage scores drift as data evolves, requiring periodic recomputation or incremental updates

package/design-patterns/compression/leverage-score-selection.md

@@ -0,0 +1,71 @@
+# Leverage Score Selection（杠杆分数选择）
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+
+## 适用问题
+当需要从大规模矩阵中选取最有代表性的行/列/token，且需保证下游线性代数运算精度时使用：KV-Cache token 选择、数据 coreset 构建、Nyström landmark 采样、分布式梯度压缩。核心诉求：**基于子空间投影的统计杠杆分数做采样，以概率保证逼近全量计算精度**。
+
+## 数学思想来源
+- 透镜：lenses/spectral.md（杠杆分数 = 行向量在主子空间上的投影能量）、lenses/probabilistic.md（概率采样与浓度不等式保证）、lenses/algorithmic.md（随机化算法的复杂度与精度权衡）
+- 知识：knowledge-base/matrix-analysis/low-rank-approximation.md（随机化 SVD、核范数）、knowledge-base/matrix-analysis/projection.md（投影矩阵对角元 = 杠杆分数）、knowledge-base/probability/concentration-inequality.md（Bernstein 矩阵浓度界）
+
+## 需要的数学知识
+- **统计杠杆分数**：$\ell_i = \|(V_k V_k^T)_i\|^2 = (V_k V_k^T)_{ii}$，第 $i$ 行在 rank-$k$ 子空间的投影能量；$\sum_i \ell_i = k$
+- **杠杆分数采样保证**：以 $p_i = \ell_i / k$ 采样 $s = O(k \log k / \epsilon^2)$ 行，$(1+\epsilon)$ 近似全量最小二乘（Drineas-Mahoney）
+- **Bernstein 矩阵界**：采样后 $\|\hat{A}^T \hat{A} - A^T A\|_2 \leq \epsilon \|A\|_F^2$，概率 $\geq 1-\delta$
+- **快速近似**：$\tilde{\ell}_i = \|(A\Omega)_i\|^2$（$\Omega$ 随机高斯），避免完整 SVD，$O(Ndk)$
+- **DPP 扩展**：行列式点过程 $P(S) \propto \det(L_S)$ 在杠杆分数基础上增加多样性保证
+
+## AI 模块形式
+```
+模块：LeverageScoreSelector
+输入：A ∈ R^{N×d}    参数：采样数 s << N，秩参数 k
+
+方法1 - 随机投影杠杆分数（在线/大规模）：
+  Omega = randn(d, k+p)                       // 随机矩阵，p=5 过采样
+  Q = qr(A @ Omega)[0]                        // N×(k+p)，GEMM + QR
+  leverage = sum(Q ** 2, dim=1)               // N 维，逐行平方和
+  indices = topk(leverage, s)                 // 确定性选 top-s
+  A_selected = A[indices]
+
+方法2 - 精确杠杆分数（离线/小矩阵）：
+  U_k = svd(A)[:k][0]                         // 截断 SVD 左奇异向量
+  leverage = sum(U_k ** 2, dim=1)             // 精确 rank-k 杠杆分数
+  probs = leverage / leverage.sum()
+  indices = multinomial_sample(N, s, probs)    // 概率采样 + 重加权
+  weights = 1 / sqrt(s * probs[indices])
+
+方法3 - KV-Cache 滑动窗口驱逐：
+  每 M 步更新 Q = qr(K_cache @ Omega)[0]
+  leverage = sum(Q ** 2, dim=1)
+  驱逐 leverage 最低的 token（对子空间贡献最小）
+
+方法4 - DPP 贪心多样化选择：
+  scores = leverage.clone(); L = A @ A^T       // PSD 核矩阵
+  for _ in range(s):
+    idx = argmax(scores); selected.append(idx)
+    scores -= α * |L[:, idx]|                   // 惩罚已选 token 的邻居
+```
+
+## 可实现结构
+- **随机投影杠杆层**：1 次 GEMM + 1 次 QR 即得近似杠杆分数，$O(Ndk)$
+- **KV-Cache 驱逐策略**：按杠杆分数排序驱逐，比 attention score 驱逐更有理论保证
+- **Coreset 构建器**：杠杆分数采样 + 重要性重加权，保证经验风险逼近全量
+- **DPP 贪心扩展**：杠杆分数 + 互斥惩罚，兼顾重要性与多样性
+
+## GPU 可行性
+- 张量化/GEMM：$A\Omega$ 为 GEMM；QR 有 cuSOLVER；杠杆分数 = elementwise 逐行平方和
+- 复杂度：$O(Ndk)$ 远优于 $O(Nd^2)$ 完整 SVD；top-s 选择 $O(N \log s)$
+- 显存：$\Omega$ 仅 $d \times k$（KB 级）；$Q$ 与 $A$ 同尺寸但可分批计算
+- 低精度：QR 建议 fp32（小矩阵 $k+p$ 列，开销可忽略）；杠杆分数可回 bf16
+- 并行：$A\Omega$ 的 GEMM 高度并行；top-s 可用 radix sort 并行
+- 算子融合：$A\Omega$ + QR + row-norm² 可融合避免物化中间矩阵
+
+## 论文表述方式
+"采用统计杠杆分数作为 token 选择的重要性度量：通过随机投影在 $O(Ndk)$ 内近似 rank-$k$ 子空间杠杆分数，Drineas-Mahoney 理论保证 $O(k \log k / \epsilon^2)$ 次采样即可 $(1+\epsilon)$ 近似全量子空间。"
+
+## 风险
+- **秩参数 $k$ 选择**：杠杆分数依赖 rank-$k$ 子空间，$k$ 选错导致采样偏差；需先诊断有效秩
+- **采样方差**：概率采样引入方差，低概率行偶尔被选中产生大权重噪声；可改用确定性 top-s
+- **杠杆分数 vs. 语义重要性不一致**：度量的是子空间贡献，不一定反映语义；可与 attention score 加权融合
+- **DPP 贪心次优**：精确 DPP 采样 $O(N^3)$，贪心近似可能漏掉全局最优子集
+- **动态数据过时**：streaming 场景下杠杆分数随数据漂移，需定期重算或增量更新