math-skill 3.0.0 → 3.1.0

package/design-patterns/attention/equivariant-attention.en.md

@@ -1,17 +1,17 @@
 # Equivariant Attention
-> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Applicable Problems
 When the input possesses an **explicit symmetry group $G$ action** (rotation, translation, permutation, reflection, etc.), and the desired model output should be **equivariant** (covariant) rather than invariant under the same transformations, equivariant constraints must be directly encoded into the attention mechanism. Typical scenarios include: 3D point clouds / molecules ($E(3)$ rigid-body group), image classification ($D_n$ rotation/reflection group), set data ($S_n$ permutation group), and multi-view / multi-sensor fusion.
 
 ## Mathematical Inspiration
 - Lenses: [symmetry, categorical (unified framework for group actions)]
-- Knowledge: [`../../knowledge-base/probability/concentration-inequality.md` (sample efficiency gains under equivariant constraints -- data equivalence along orbits), `../../knowledge-base/probability/entropy.md` (equivariant constraints reduce output distribution entropy, yielding stronger inductive bias)]
+- Knowledge: [`../../knowledge-base/probability/concentration-inequality.en.md` (sample efficiency gains under equivariant constraints -- data equivalence along orbits), `../../knowledge-base/probability/entropy.en.md` (equivariant constraints reduce output distribution entropy, yielding stronger inductive bias)]
 
 ## Required Mathematical Knowledge
 - **Group Representation Theory Basics**: Linear representation of a group $G$, $\rho: G \to GL(V)$, and irreducible representation decomposition
 - **Equivariant Map Definition**: $f(g \cdot x) = \rho_{\text{out}}(g) \cdot f(x)$ for all $g \in G$
-- **Orbit-Stabilizer Theorem** (see `references/books/abstract-algebra.md` Ch.5): $|orbit| = |G|/|stab|$, giving the parameter sharing multiplier
+- **Orbit-Stabilizer Theorem** (see `../../references/books/abstract-algebra.en.md` Ch.5): $|orbit| = |G|/|stab|$, giving the parameter sharing multiplier
 - **Schur's Lemma**: An equivariant linear map between irreducible representations is either zero or a scalar multiple
 
 ## AI Module Form
@@ -55,14 +55,20 @@ output = mean(softmax((rho(g)@X@W_q) @ (rho(g)@X@W_k).T/sqrt(d)) @ (rho(g)@X@W_v
 - **G-CNN Attention**: $D_n$ rotation/reflection equivariance for remote sensing imagery and medical imaging
 
 ## GPU Feasibility
-- **Dimension 1 Tensorization**: Group actions implemented as $\rho(g)$ matrix multiplications; equivariant features stored as batched tensors
-- **Dimension 2 GEMM-mappability**: $\rho(g) X$ and $Q K^T$ are both GEMM operations; $|G|$ group elements map to batched GEMM
-- **Dimension 3 Complexity**: $|G|$-fold computation overhead; acceptable for small groups ($|D_4|=8$), infeasible for large groups ($|S_n|=n!$). Remedy: use generators + Cayley graph propagation instead of full group enumeration
-- **Dimension 4 Memory**: Requires storing $|G|$ copies of intermediate features; mitigated by chunking + gradient checkpointing
-- **Dimension 5 Low Precision**: Orthogonal representation matrices are numerically stable under bf16
-- **Dimension 6 Parallelism**: $|G|$ group elements are naturally parallelizable (along the batch dimension)
-- **Dimension 7 Sparsity**: Permutation $\rho(g)$ is extremely sparse and can be encoded as gather indices
-- **Dimension 8 Operator Fusion**: Group action + linear duality can be fused into a single batched GEMM
+- **D1**: Group actions implemented as $\rho(g)$ matrix multiplications; equivariant features stored as batched tensors
+- **D2**: $\rho(g) X$ and $Q K^T$ are both GEMM operations; $|G|$ group elements map to batched GEMM
+- **D3**: $|G|$-fold computation overhead; acceptable for small groups ($|D_4|=8$), infeasible for large groups ($|S_n|=n!$). Remedy: use generators + Cayley graph propagation instead of full group enumeration
+- **D4**: Requires storing $|G|$ copies of intermediate features; mitigated by chunking + gradient checkpointing
+- **D5**: Orthogonal representation matrices are numerically stable under bf16
+- **D6**: $|G|$ group elements are naturally parallelizable (along the batch dimension)
+- **D7**: Permutation $\rho(g)$ is extremely sparse and can be encoded as gather indices
+- **D8**: Group action + linear duality can be fused into a single batched GEMM
+
+**Quantitative assessment example** (E(3) equivariant, |G|=24 rotation group, d=64, n=512):
+- D3: |G|x standard attention FLOPs = 24 · 2·512²·64 ≈ 805M (vs standard 33.6M)
+- D4: Requires storing |G|=24 copies of intermediate features, 24 · 512 · 64 · 2B ≈ 1.5MB extra memory
+- D6: 24 group elements parallelizable along the batch dimension, ideal speedup 24x
+- D8: ρ(g)·X·W_q three-step fusion into a single batched GEMM
 
 ## Paper Phrasing
 "We propose an equivariant attention mechanism that constrains attention weights to be group invariants and attention outputs to be group equivariants, directly encoding the inductive bias of symmetry group $G$ into the model architecture without additional data augmentation, achieving a $|G|/|stab|$-fold improvement in parameter efficiency."

package/design-patterns/attention/equivariant-attention.md

@@ -1,5 +1,5 @@
 # 等变注意力 / Equivariant Attention
-> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
 当输入具有**明确的对称群 $G$ 作用**（旋转、平移、置换、反射等），且期望模型输出在相同变换下**协变**（equivariant）而非不变时，需要将等变约束直接编入注意力机制。典型场景：3D 点云/分子（$E(3)$ 刚体群）、图像分类（$D_n$ 旋转/镜像群）、集合数据（$S_n$ 置换群）、多视角/多传感器融合。
@@ -11,7 +11,7 @@
 ## 需要的数学知识
 - **群表示论基础**：群 $G$ 的线性表示 $\rho: G \to GL(V)$，不可约表示分解
 - **等变映射定义**：$f(g \cdot x) = \rho_{\text{out}}(g) \cdot f(x)$，对所有 $g \in G$
-- **轨道-稳定子定理**（参见 `references/books/abstract-algebra.md` Ch.5）：$|orbit| = |G|/|stab|$，参数共享倍率
+- **轨道-稳定子定理**（参见 `../../references/books/abstract-algebra.md` Ch.5）：$|orbit| = |G|/|stab|$，参数共享倍率
 - **Schur 引理**：不可约表示间的等变线性映射要么为零要么为标量乘
 
 ## AI 模块形式
@@ -55,14 +55,20 @@ output = mean(softmax((rho(g)@X@W_q) @ (rho(g)@X@W_k).T/sqrt(d)) @ (rho(g)@X@W_v
 - **G-CNN 注意力**：$D_n$ 旋转/镜像等变，用于遥感图像和医学影像
 
 ## GPU 可行性
-- **维度 1 张量化 ✅**：群作用实现为 $\rho(g)$ 矩阵乘，等变特征为 batched 张量
-- **维度 2 GEMM 可映射 ✅**：$\rho(g) X$ 和 $Q K^T$ 均为 GEMM；$|G|$ 个群元素 → batched GEMM
-- **维度 3 复杂度 ⚠️**：$|G|$ 倍计算量，小群（$|D_4|=8$）可接受，大群（$|S_n|=n!$）不可。改造：用生成元 + Cayley 图传播替代整群枚举
-- **维度 4 显存 ⚠️**：需存储 $|G|$ 份中间特征，可分块 + gradient checkpointing
-- **维度 5 低精度 ✅**：正交表示矩阵在 bf16 下数值稳定
-- **维度 6 并行 ✅**：$|G|$ 个群元素天然并行（batch 维）
-- **维度 7 稀疏 ⚠️**：置换 $\rho(g)$ 极度稀疏，可编码为 gather 索引
-- **维度 8 算子融合 ✅**：群作用 + 线性变换可融合为单次 batched GEMM
+- **D1[v]**：群作用实现为 $\rho(g)$ 矩阵乘，等变特征为 batched 张量
+- **D2[v]**：$\rho(g) X$ 和 $Q K^T$ 均为 GEMM；$|G|$ 个群元素 → batched GEMM
+- **D3[~]**：$|G|$ 倍计算量，小群（$|D_4|=8$）可接受，大群（$|S_n|=n!$）不可。改造：用生成元 + Cayley 图传播替代整群枚举
+- **D4[~]**：需存储 $|G|$ 份中间特征，可分块 + gradient checkpointing
+- **D5[v]**：正交表示矩阵在 bf16 下数值稳定
+- **D6[v]**：$|G|$ 个群元素天然并行（batch 维）
+- **D7[~]**：置换 $\rho(g)$ 极度稀疏，可编码为 gather 索引
+- **D8[v]**：群作用 + 线性变换可融合为单次 batched GEMM
+
+**量化评估示例**（E(3) 等变, |G|=24 旋转群, d=64, n=512）：
+- D3: |G|× 标准 attention FLOPs = 24 · 2·512²·64 ≈ 805M（vs 标准 33.6M）
+- D4: 需存储 |G|=24 份中间特征，24 · 512 · 64 · 2B ≈ 1.5MB 额外显存
+- D6: 24 个群元素沿 batch 维并行，理想加速比 24x
+- D8: ρ(g)·X·W_q 三步融合为单次 batched GEMM
 
 ## 论文表述方式
 "我们提出等变注意力机制，通过将注意力权重约束为群不变量、注意力输出约束为群等变量，在不增加数据增强的情况下将对称群 $G$ 的归纳偏置直接编码进模型结构，参数效率提升 $|G|/|stab|$ 倍。"

package/design-patterns/attention/geometry-aware-attention.en.md

@@ -1,12 +1,12 @@
 # Geometry-Aware Attention
-> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Applicable Problems
 When **known geometric relationships** exist between tokens/keys (spatial distance, manifold geodesic distance, hierarchical structure, temporal interval), the position-agnostic inner product of standard attention cannot exploit this structural information. Geometry-aware attention **directly injects geometric priors into attention weight computation**, enabling the model to naturally respect the metric structure of the underlying space. Typical scenarios include: 3D scene understanding, molecular conformation, temporal forecasting, hierarchical text structure (paragraph-sentence-word), and knowledge graphs.
 
 ## Mathematical Inspiration
 - Lenses: [symmetry (metric invariance), duality (coordinate-system-independent representation)]
-- Knowledge: [`../../knowledge-base/information-geometry/fisher-metric.md` (geometric metrics on distribution spaces), `../../knowledge-base/probability/concentration-inequality.md` (concentration behavior under geometric constraints)]
+- Knowledge: [`../../knowledge-base/information-geometry/fisher-metric.en.md` (geometric metrics on distribution spaces), `../../knowledge-base/probability/concentration-inequality.en.md` (concentration behavior under geometric constraints)]
 
 ## Required Mathematical Knowledge
 - **Metric Spaces and Distance Functions**: Euclidean distance, geodesic distance, tree distance, Wasserstein distance
@@ -56,17 +56,17 @@ def manifold_attention(Q, K, V, manifold):
 - **Molecular Conformation Attention**: 3D atomic coordinates -> distance matrix -> geometric bias, used in molecular GNNs and protein structure prediction
 
 ## GPU Feasibility
-- **Dimension 1 Tensorization**: Distance matrices and bias matrices are dense tensors with element-wise operations
-- **Dimension 2 GEMM-mappability**: The main body $Q K^T$ is standard GEMM; geometric bias is additive and does not interrupt GEMM
-- **Dimension 3 Complexity**: Pairwise distance matrix construction and storage are $O(n^2)$; however, block-wise computation + online softmax can avoid materializing the full matrix
-- **Dimension 4 Memory**: The $n \times n$ distance matrix occupies GPU memory; mitigated by block/streaming computation (compatible with FlashAttention)
-- **Dimension 5 Low Precision**: Distance computation and bias addition are stable under bf16; the exp in RBF requires overflow caution (clamp distances)
-- **Dimension 6 Parallelism**: Distance computation and attention can be pipelined; pairwise distances can be computed in parallel blocks
-- **Dimension 7 Sparsity**: Distant biases tend toward $-\infty$ (approaching zero after softmax), naturally inducing structured sparsity (local attention windows)
-- **Dimension 8 Operator Fusion**: Geometric bias can be fused into FlashAttention's online softmax loop (added after $QK^T$, before softmax)
+- **D1**: Distance matrices and bias matrices are dense tensors with element-wise operations
+- **D2**: The main body $Q K^T$ is standard GEMM; geometric bias is additive and does not interrupt GEMM
+- **D3**: Pairwise distance matrix construction and storage are $O(n^2)$; however, block-wise computation + online softmax can avoid materializing the full matrix
+- **D4**: The $n \times n$ distance matrix occupies GPU memory; mitigated by block/streaming computation (compatible with FlashAttention)
+- **D5**: Distance computation and bias addition are stable under bf16; the exp in RBF requires overflow caution (clamp distances)
+- **D6**: Distance computation and attention can be pipelined; pairwise distances can be computed in parallel blocks
+- **D7**: Distant biases tend toward $-\infty$ (approaching zero after softmax), naturally inducing structured sparsity (local attention windows)
+- **D8**: Geometric bias can be fused into FlashAttention's online softmax loop (added after $QK^T$, before softmax)
 
 ## Paper Phrasing
-"We propose geometry-aware attention, which explicitly injects learnable geometric distance bias terms into attention scores, enabling the model to naturally respect the metric structure of the input space. This achieves exponential decay of attention to distant tokens without increasing parameter count, while maintaining compatibility with FlashAttention."
+"We propose geometry-aware attention, which explicitly injects geometric distance bias terms into attention scores, enabling the model to naturally respect the metric structure of the input space. This achieves exponential decay of attention to distant tokens with only a small number of additional geometric bias parameters (zero for fixed bias variants such as ALiBi), while maintaining compatibility with FlashAttention."
 
 ## Risks
 - **Geometric Prior Conflicting with Data**: If geometric distance is inconsistent with semantic relevance (e.g., distant but semantically related tokens in text), strong geometric bias will impair model expressiveness. The bias should be learnable and overridable by content-based attention.

package/design-patterns/attention/geometry-aware-attention.md

@@ -1,5 +1,5 @@
 # 几何感知注意力 / Geometry-Aware Attention
-> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
 当 token/key 之间存在**已知的几何关系**（空间距离、流形测地距、层级结构、时序间隔）时，标准注意力的位置无关内积无法利用这些结构信息。几何感知注意力将**几何先验直接注入注意力权重计算**，使模型天然尊重底空间的度量结构。典型场景：3D 场景理解、分子构象、时序预测、层级文本结构（段落-句子-词）、知识图谱。
@@ -55,17 +55,17 @@ def manifold_attention(Q, K, V, manifold):
 - **分子构象注意力**：3D 原子坐标 → 距离矩阵 → 几何偏置，用于分子 GNN 和蛋白质结构预测
 
 ## GPU 可行性
-- **维度 1 张量化 ✅**：距离矩阵和偏置矩阵均为稠密张量，逐元素运算
-- **维度 2 GEMM 可映射 ✅**：主体 $Q K^T$ 为标准 GEMM；几何偏置为加法，不阻断 GEMM
-- **维度 3 复杂度 ⚠️**：成对距离矩阵 $O(n^2)$ 构建和存储；但可用分块计算 + online softmax 避免物化完整矩阵
-- **维度 4 显存 ⚠️**：$n \times n$ 距离矩阵占用显存；可分块/流式计算（与 FlashAttention 兼容）
-- **维度 5 低精度 ✅**：距离计算和偏置加法在 bf16 下稳定；RBF 的 exp 需注意上溢（clamp 距离）
-- **维度 6 并行 ✅**：距离计算和注意力可流水线并行，成对距离可分块并行
-- **维度 7 稀疏 ✅**：远距离偏置趋于 $-\infty$（softmax 后趋零），天然诱导结构化稀疏（局部注意力窗口）
-- **维度 8 算子融合 ✅**：几何偏置可融入 FlashAttention 的 online softmax 循环中（加在 $QK^T$ 之后、softmax 之前）
+- **D1[v]**：距离矩阵和偏置矩阵均为稠密张量，逐元素运算
+- **D2[v]**：主体 $Q K^T$ 为标准 GEMM；几何偏置为加法，不阻断 GEMM
+- **D3[~]**：成对距离矩阵 $O(n^2)$ 构建和存储；但可用分块计算 + online softmax 避免物化完整矩阵
+- **D4[~]**：$n \times n$ 距离矩阵占用显存；可分块/流式计算（与 FlashAttention 兼容）
+- **D5[v]**：距离计算和偏置加法在 bf16 下稳定；RBF 的 exp 需注意上溢（clamp 距离）
+- **D6[v]**：距离计算和注意力可流水线并行，成对距离可分块并行
+- **D7[v]**：远距离偏置趋于 $-\infty$（softmax 后趋零），天然诱导结构化稀疏（局部注意力窗口）
+- **D8[v]**：几何偏置可融入 FlashAttention 的 online softmax 循环中（加在 $QK^T$ 之后、softmax 之前）
 
 ## 论文表述方式
-"我们提出几何感知注意力，通过在注意力分数中显式注入可学习的几何距离偏置项，使模型天然尊重输入空间的度量结构，在不增加参数量的情况下将远距离 token 的注意力指数衰减，同时保持与 FlashAttention 的兼容性。"
+"我们提出几何感知注意力，通过在注意力分数中显式注入几何距离偏置项，使模型天然尊重输入空间的度量结构，在仅增加少量几何偏置参数（固定偏置如 ALiBi 则零额外参数）的情况下将远距离 token 的注意力指数衰减，同时保持与 FlashAttention 的兼容性。"
 
 ## 风险
 - **几何先验与数据冲突**：若几何距离与语义相关性不一致（如文本中远距离但语义相关的 token），强几何偏置会伤害模型表达。需让偏置可学习且可被内容注意力覆盖。

package/design-patterns/attention/information-bottleneck-attention.en.md

@@ -1,12 +1,12 @@
 # Information Bottleneck Attention
-> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Applicable Problems
 When the attention mechanism needs to **selectively transmit useful information while suppressing redundant/noisy information**, information bottleneck theory can guide the learning of attention weights -- maximizing the mutual information $I(Z;Y)$ of the attention distribution with respect to the target $Y$, while minimizing the mutual information $I(X;Z)$ with respect to the input $X$. Typical scenarios include: long-document summarization (filtering large numbers of irrelevant tokens), multimodal alignment (cross-modal noise suppression), and interpretability (attention weights as visualization of information flow).
 
 ## Mathematical Inspiration
 - Lenses: [categorical (information-theoretic framework unifying attention design), variational (constrained variational and Lagrangian duality)]
-- Knowledge: [`../../knowledge-base/probability/information-bottleneck.md` (IB objective and variational lower bound), `../../knowledge-base/probability/kl-divergence.md` (implementation of KL regularization), `../../knowledge-base/probability/entropy.md` (mutual information estimation)]
+- Knowledge: [`../../knowledge-base/probability/information-bottleneck.en.md` (IB objective and variational lower bound), `../../knowledge-base/probability/kl-divergence.en.md` (implementation of KL regularization), `../../knowledge-base/probability/entropy.en.md` (mutual information estimation)]
 
 ## Required Mathematical Knowledge
 - **Information Bottleneck Objective**: $\min I(X;Z) - \beta I(Z;Y)$, balancing compression and prediction
@@ -19,27 +19,47 @@ When the attention mechanism needs to **selectively transmit useful information
 
 **Scheme A: KL-Regularized Attention (Simplest IB Attention)**:
 ```python
-# KL divergence between attention distribution q(z|x) and uniform prior as IB regularizer
+# ⚠ Note the semantic direction of KL:
+# KL(attn || uniform) = log(n) - H(attn)
+# Minimizing +beta * KL(attn || uniform) = maximizing H(attn) → pushes toward uniform (max entropy / max compression)
+# This does NOT induce sparsity by itself! Sparsity emerges from the tension with task_loss.
+# For explicit sparsity induction, use an entropy penalty +beta * H(attn) (minimize entropy).
+
 scores = (Q @ K.T) / sqrt(d)
 attn = softmax(scores)
 uniform_prior = ones_like(attn) / n
-kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
-loss = task_loss + beta * kl_reg
+
+# Correct IB compression term: KL(q(z|x) || p(z)), where p(z) is a fixed prior
+# Minimizing this term → compresses information (pushes toward prior); tension with task_loss yields useful attention
+kl_compression = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+# = log(n) - H(attn)
+
+# IB objective: min task_loss + beta * I(X;Z), where I(X;Z) ≈ KL(attn || prior)
+loss_ib = task_loss + beta * kl_compression
+# Note: this term alone pushes toward uniform; sparsity arises from the counter-pull of task_loss
+
+# Alternative: explicit sparsity induction (not IB compression, but entropy penalty)
+entropy = -(attn * log(attn + eps)).sum(dim=-1).mean()
+loss_sparse = task_loss + beta * entropy  # minimize entropy → concentrated attention → implicit Top-K
+
 output = attn @ V
 ```
 
 **Scheme B: Variational Information Bottleneck Attention (VIB-Attention)**:
 ```python
-# Introduce stochastic bottleneck Z ~ q(Z|attention) to limit information throughput
+# Introduce a continuous stochastic bottleneck Z ~ q(Z|context) after value aggregation
 scores = (Q @ K.T) / sqrt(d)
 attn = softmax(scores)
-mu_z, log_var_z = attn, linear(attn)  # mean and learnable variance
-z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # reparameterization + softmax
-kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs prior
-output = z @ V
+context = attn @ V
+mu_z, log_var_z = linear_mu(context), linear_logvar(context)
+z = mu_z + exp(0.5 * log_var_z) * randn_like(mu_z)  # Gaussian reparameterization
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL[q(z|context)||N(0,I)]
+output = linear_out(z)
 loss = task_loss + beta * kl
 ```
 
+If the bottleneck is applied directly on the attention simplex (e.g., noisy logits followed by softmax), the random variable is logistic-normal / Concrete-like. The closed-form Gaussian KL above no longer applies; use Monte Carlo KL, a Concrete KL approximation, or place the Gaussian KL on pre-softmax logits.
+
 **Scheme C: Mutual Information Maximization Attention (DIM Style)**:
 ```python
 # Directly maximize I(Z;Y) via InfoNCE, with KL constraining I(X;Z)
@@ -50,22 +70,22 @@ loss = -info_nce + beta * kl_bottleneck
 ```
 
 ## Implementable Architectures
-- **IB-Sparse Attention**: KL regularization serves as a soft constraint on attention sparsity -- larger KL relative to uniform prior leads to more concentrated attention, yielding implicit Top-K selection
+- **IB-Sparse Attention**: The IB compression term KL(attn || uniform) by itself pushes toward uniform (maximum entropy), but its tension with task_loss forces the attention to balance between "compressing all information" and "selectively transmitting useful information," implicitly producing non-uniform attention. For **explicit** sparsity induction (Top-K selection), use an entropy penalty `+beta * H(attn)` (minimize attention entropy → concentration), rather than relying on the IB compression term alone
 - **IB Interpretation of Dropout**: Dropout is a form of stochastic information bottleneck -- randomly blocking information channels forces the model to learn robust representations. The dropout rate corresponds to the $\beta$ parameter
 - **Multi-Head Information Allocation**: Different heads learn different information bottlenecks (different $\beta$); some heads transmit global information, others only local information
 
 ## GPU Feasibility
-- **Dimension 1 Tensorization**: KL regularization involves element-wise operations; VIB reparameterization sampling is element-wise
-- **Dimension 2 GEMM-mappability**: The main body $QK^T$ and $attn \cdot V$ are standard GEMM; regularization introduces no new GEMM operations
-- **Dimension 3 Complexity**: KL regularization is $O(n)$ per token, adding no asymptotic complexity
-- **Dimension 4 Memory**: VIB requires additional $\mu_z$ and $\log\sigma_z$, approximately doubling attention weight memory
-- **Dimension 5 Low Precision**: log/exp in KL computations are stable under bf16 (standard log-softmax tricks)
-- **Dimension 6 Parallelism**: Regularization can be computed in parallel with forward propagation, introducing no serial dependencies
-- **Dimension 7 Sparsity**: KL regularization implicitly induces attention sparsity, amenable to block-sparse acceleration
-- **Dimension 8 Operator Fusion**: KL can be fused into the softmax kernel (FusedSoftmaxKL)
+- **D1**: KL regularization involves element-wise operations; VIB reparameterization sampling is element-wise
+- **D2**: The main body $QK^T$ and $attn \cdot V$ are standard GEMM; regularization introduces no new GEMM operations
+- **D3**: KL regularization is $O(n)$ per token, adding no asymptotic complexity
+- **D4**: VIB requires additional $\mu_z$ and $\log\sigma_z$, approximately doubling attention weight memory
+- **D5**: log/exp in KL computations are stable under bf16 (standard log-softmax tricks)
+- **D6**: Regularization can be computed in parallel with forward propagation, introducing no serial dependencies
+- **D7[~]**: KL regularization by itself does not induce sparsity (pushes toward uniform distribution); for sparsity, use an entropy penalty +beta * H(attn), which then enables block-sparse acceleration
+- **D8**: KL can be fused into the softmax kernel (FusedSoftmaxKL)
 
 ## Paper Phrasing
-"We propose information bottleneck attention, which models attention as an information bottleneck variational problem. By maximizing the mutual information between output and target while minimizing redundant information transmission from the input, it achieves theoretically optimal information selection and demonstrates enhanced sparsity and interpretability in experiments."
+"We propose information bottleneck attention, which models attention as an information bottleneck optimization problem. By maximizing the mutual information between output and target while constraining redundant information transmission from the input, the IB compression term by itself pushes toward a uniform distribution (maximum entropy), and its tension with the task loss produces non-uniform attention; explicit sparsity induction requires an additional entropy penalty term."
 
 ## Risks
 - **High Variance of Mutual Information Estimators**: The MINE/NWJ/InfoNCE estimators in Scheme C exhibit high variance in high-dimensional spaces, potentially causing training instability. It is recommended to first validate the basic effect of IB attention with Scheme A (KL regularization) before attempting the full IB objective.

package/design-patterns/attention/information-bottleneck-attention.md

@@ -1,5 +1,5 @@
 # 信息瓶颈注意力 / Information Bottleneck Attention
-> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
 当注意力机制需要**选择性传递有用信息、抑制冗余/噪声信息**时，用信息瓶颈理论指导注意力权重的学习——使注意力分布最大化关于目标 $Y$ 的互信息 $I(Z;Y)$，同时最小化关于输入 $X$ 的互信息 $I(X;Z)$。典型场景：长文档摘要（大量无关 token 需被过滤）、多模态对齐（跨模态噪声抑制）、可解释性（注意力权重作为信息流的可视化）。
@@ -19,27 +19,47 @@
 
 **方案 A：KL 正则化注意力（最简 IB 注意力）**：
 ```python
-# 注意力分布 q(z|x) 与均匀先验的 KL 散度作为 IB 正则
+# ⚠ 注意 KL 方向的语义：
+# KL(attn || uniform) = log(n) - H(attn)
+# 最小化 +beta * KL(attn || uniform) = 最大化 H(attn) → 推向均匀分布（最大熵/最大压缩）
+# 这本身不诱导稀疏性！稀疏性来自压缩项与 task_loss 的张力。
+# 若需要显式诱导稀疏注意力，应使用熵惩罚 +beta * H(attn)（最小化熵）。
+
 scores = (Q @ K.T) / sqrt(d)
 attn = softmax(scores)
 uniform_prior = ones_like(attn) / n
-kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
-loss = task_loss + beta * kl_reg
+
+# 正确的 IB 压缩项：KL(q(z|x) || p(z))，p(z) 为固定先验
+# 最小化此项 → 压缩信息（推向先验），与 task_loss 的张力产生有用注意力
+kl_compression = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+# = log(n) - H(attn)
+
+# IB 目标：min task_loss + beta * I(X;Z)，其中 I(X;Z) ≈ KL(attn || prior)
+loss_ib = task_loss + beta * kl_compression
+# 注意：此项单独作用会推向均匀；稀疏性来自 task_loss 的反向拉力
+
+# 替代方案：显式稀疏性诱导（非 IB 压缩，而是熵惩罚）
+entropy = -(attn * log(attn + eps)).sum(dim=-1).mean()
+loss_sparse = task_loss + beta * entropy  # 最小化熵 → 注意力集中 → 隐式 Top-K
+
 output = attn @ V
 ```
 
 **方案 B：变分信息瓶颈注意力（VIB-Attention）**：
 ```python
-# 引入随机瓶颈 Z ~ q(Z|attention)，限制信息通过量
+# 引入随机连续瓶颈 Z ~ q(Z|context)，限制 value 聚合后的信息通过量
 scores = (Q @ K.T) / sqrt(d)
 attn = softmax(scores)
-mu_z, log_var_z = attn, linear(attn)  # 均值与可学习方差
-z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # 重参数化 + softmax
-kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs 先验
-output = z @ V
+context = attn @ V
+mu_z, log_var_z = linear_mu(context), linear_logvar(context)
+z = mu_z + exp(0.5 * log_var_z) * randn_like(mu_z)  # 高斯重参数化
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL[q(z|context)||N(0,I)]
+output = linear_out(z)
 loss = task_loss + beta * kl
 ```
 
+若瓶颈直接作用在 attention simplex 上（如对 logits 加噪后 softmax），变量分布是 logistic-normal / Concrete 类，不能再使用普通高斯到标准正态的闭式 KL；应使用 Monte Carlo KL、Concrete KL 近似，或把 KL 放在 softmax 前的 logits 高斯变量上。
+
 **方案 C：互信息最大化注意力（DIM 风格）**：
 ```python
 # 用 InfoNCE 直接最大化 I(Z;Y)，配合 KL 约束 I(X;Z)
@@ -50,23 +70,23 @@ loss = -info_nce + beta * kl_bottleneck
 ```
 
 ## 可实现结构
-- **IB-Sparse Attention**：将 KL 正则化为注意力稀疏性的软约束——KL 对均匀先越大 → 注意力越集中 → 隐式 Top-K 选择
+- **IB-Sparse Attention**：IB 压缩项 KL(attn || uniform) 本身推向均匀分布（最大熵），但与 task_loss 的张力迫使注意力在"压缩所有信息"与"选择性传递有用信息"之间取舍，从而隐式产生非均匀注意力。若要**显式**诱导稀疏（Top-K 选择），应使用熵惩罚 `+beta * H(attn)`（最小化注意力熵 → 集中化），而非依赖 IB 压缩项
 - **Dropout 的 IB 解释**：Dropout 是一种随机信息瓶颈——随机阻断部分信息通道，迫使模型学习鲁棒表示。Dropout rate 对应 $\beta$ 参数
 - **Multi-Head 信息分配**：不同 head 学习不同的信息瓶颈（不同 $\beta$），有的 head 传递全局信息，有的只传递局部信息
 
 ## GPU 可行性
-- **维度 1 张量化 ✅**：KL 正则项为逐元素运算，VIB 重参数化采样为逐元素
-- **维度 2 GEMM 可映射 ✅**：主体 $QK^T$ 和 $attn \cdot V$ 为标准 GEMM，正则项不引入新 GEMM
-- **维度 3 复杂度 ✅**：KL 正则 $O(n)$ 逐 token，不增加渐近复杂度
-- **维度 4 显存 ⚠️**：VIB 需额外 $\mu_z$ 和 $\log\sigma_z$，约 2x 注意力权重显存
-- **维度 5 低精度 ✅**：KL 中 log/exp 在 bf16 下稳定（标准 log-softmax 技巧）
-- **维度 6 并行 ✅**：正则项可与前向传播并行，不引入串行依赖
-- **维度 7 稀疏 ✅**：KL 正则隐式诱导注意力稀疏性，可利用 block-sparse 加速
-- **维度 8 算子融合 ✅**：KL 可融入 softmax kernel（FusedSoftmaxKL）
+- **D1[v]**：KL 正则项为逐元素运算，VIB 重参数化采样为逐元素
+- **D2[v]**：主体 $QK^T$ 和 $attn \cdot V$ 为标准 GEMM，正则项不引入新 GEMM
+- **D3[v]**：KL 正则 $O(n)$ 逐 token，不增加渐近复杂度
+- **D4[~]**：VIB 需额外 $\mu_z$ 和 $\log\sigma_z$，约 2x 注意力权重显存
+- **D5[v]**：KL 中 log/exp 在 bf16 下稳定（标准 log-softmax 技巧）
+- **D6[v]**：正则项可与前向传播并行，不引入串行依赖
+- **D7[~]**：KL 正则本身不诱导稀疏性（推向均匀分布）；若需稀疏，应改用熵惩罚 +beta * H(attn)，此时可 block-sparse 加速
+- **D8[v]**：KL 可融入 softmax kernel（FusedSoftmaxKL）
 
 ## 论文表述方式
-"我们提出信息瓶颈注意力机制，通过将注意力建模为信息瓶颈优化问题，在最大化输出与目标的互信息的同时最小化输入信息的冗余传递，实现理论上最优的信息选择，并在实验中展现出更强的稀疏性和可解释性。"
+"我们提出信息瓶颈注意力机制，通过将注意力建模为信息瓶颈优化问题，在最大化输出与目标的互信息的同时约束输入信息的冗余传递。IB 压缩项本身推向均匀分布（最大熵），与任务损失的张力产生非均匀注意力；若要显式诱导稀疏性，需额外使用熵惩罚项。"
 
 ## 风险
 - **互信息估计的方差问题**：方案 C 中的 MINE/NWJ/InfoNCE 估计器在高维空间中方差大，可能导致训练不稳定。建议先用方案 A（KL 正则）验证 IB 注意力的基本效果，再尝试完整 IB 目标。
-- **$\beta$ 调参困难**：$\beta$ 控制压缩-预测权衡，不同任务最优 $\beta$ 差异大。$\beta$ 过小退化为标准注意力，$\beta$ 过大导致欠拟合。建议自适应 $\beta$ 调度或信息平面监控。
+- **$\beta$ 调参困难**：$\beta$ 控制压缩-预测权衡，不同任务最优 $\beta$ 差异大。$\beta$ 过小退化为标准注意力，$\beta$ 过大导致欠拟合。建议自适应 $\beta$ 调度或信息平面监控。

package/design-patterns/attention/projection-attention.en.md

@@ -1,12 +1,12 @@
 # Projection Attention
-> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Applicable Problems
 When the token/key space dimensionality is too high, the standard attention $Q K^T$ inner product tends to become uniform in high-dimensional spaces ("attention collapse"). In such cases, key/query vectors must first be projected onto a **subspace with superior geometric structure** before computing attention. Typical scenarios include: KV-Cache compression for long-context LLMs, attention sparsification in high-dimensional embedding spaces, and attention alignment of heterogeneous features in multimodal fusion.
 
 ## Mathematical Inspiration
 - Lenses: [projection, spectral, probabilistic]
-- Knowledge: [`../../knowledge-base/probability/concentration-inequality.md` (high-dimensional concentration inequalities provide theoretical explanation for attention collapse), `../../knowledge-base/probability/entropy.md` (entropy of attention distributions as a quality metric)]
+- Knowledge: [`../../knowledge-base/probability/concentration-inequality.en.md` (high-dimensional concentration inequalities provide theoretical explanation for attention collapse), `../../knowledge-base/probability/entropy.en.md` (entropy of attention distributions as a quality metric)]
 
 ## Required Mathematical Knowledge
 - **Johnson-Lindenstrauss Lemma**: High-dimensional point sets can be projected onto an $O(\log n / \epsilon^2)$-dimensional subspace while approximately preserving distances
@@ -31,7 +31,7 @@ scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
 attn = softmax(scores) @ V
 ```
 
-2. **Random Fixed Projection** (guaranteed by the JL Lemma):
+2. **Random Fixed Projection** (distance-preservation bound when JL conditions hold):
 ```
 P = random_gaussian(d, r) / sqrt(r)  # fixed, not trained
 scores = (Q @ P) @ (K @ P).T / sqrt(r)
@@ -51,17 +51,17 @@ scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
 - **Hierarchical Projection**: Shallow layers use small $r$ (coarse filtering), while deep layers use large $r$ (fine ranking)
 
 ## GPU Feasibility
-- **Dimension 1 Tensorization**: Projection = matrix multiplication, attention = matrix multiplication chain; entirely tensor operations
-- **Dimension 2 GEMM-mappability**: $Q P_Q$ and $K P_K$ are both standard GEMM operations, fully utilizing Tensor Cores
-- **Dimension 3 Complexity**: Projection cost $O(ndr)$ is far below attention cost $O(n^2 d)$, and after projection $r \ll d$ reduces attention to $O(n^2 r)$
-- **Dimension 4 Memory**: Key-cache compressed by $d/r$; V-cache must be handled separately
-- **Dimension 5 Low Precision**: Projection matrices are orthogonal or near-orthogonal, yielding numerical stability; bf16 is acceptable
-- **Dimension 6 Parallelism**: Projection can be pipelined with attention; Multi-Head is naturally parallel across heads
-- **Dimension 7 Sparsity**: Projection matrices are inherently dense; sparse projections (CountSketch) may introduce gather/scatter operations
-- **Dimension 8 Operator Fusion ⚠️ Retrofittable, needs kernel-level validation**: Projection may be integrated into a FlashAttention-style kernel, but requires kernel-level verification
+- **D1**: Projection = matrix multiplication, attention = matrix multiplication chain; entirely tensor operations
+- **D2**: $Q P_Q$ and $K P_K$ are both standard GEMM operations, fully utilizing Tensor Cores
+- **D3**: Projection cost $O(ndr)$ is far below attention cost $O(n^2 d)$, and after projection $r \ll d$ reduces attention to $O(n^2 r)$
+- **D4**: Key-cache compressed by $d/r$; V-cache must be handled separately
+- **D5**: Projection matrices are orthogonal or near-orthogonal, yielding numerical stability; bf16 is acceptable
+- **D6**: Projection can be pipelined with attention; Multi-Head is naturally parallel across heads
+- **D7**: Projection matrices are inherently dense; sparse projections (CountSketch) may introduce gather/scatter operations
+- **D8[~] Retrofittable, needs kernel-level validation**: Projection may be integrated into a FlashAttention-style kernel, but requires kernel-level verification
 
 ## Paper Phrasing
-"We decompose attention computation into two steps -- low-dimensional subspace projection and projected-space attention -- achieving a $d/r$-fold Key-cache compression (full KV-Cache compression requires separate V compression/reconstruction) while preserving attention quality and guaranteeing Johnson-Lindenstrauss distance preservation."
+"We decompose attention computation into low-dimensional subspace projection and projected-space attention to reduce Key dimensionality; V-cache must be handled separately. With an independent random projection and the sample-size / target-dimension conditions of the Johnson-Lindenstrauss lemma, Euclidean distances of the projected objects can be approximately preserved. This does not automatically guarantee softmax-attention quality, so attention/output error and task metrics must be reported."
 
 ## Risks
 - **Projection Direction Degeneracy**: Learnable projections may collapse onto a few directions (deterioration of the condition number of $P^T P$), causing attention distributions to degenerate. Orthogonal regularization $\|P^T P - I\|_F^2$ is required.

package/design-patterns/attention/projection-attention.md

@@ -1,5 +1,5 @@
 # 投影注意力 / Projection Attention
-> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
 当输入的 token/key 空间维度过高，标准注意力的 $Q K^T$ 内积在高维空间中趋于均匀（"注意力坍缩"）时，需要先将 key/query 投影到一个**几何结构更优的子空间**再做注意力计算。典型场景：长上下文 LLM 的 KV-Cache 压缩、高维嵌入空间的注意力稀疏化、多模态融合中异构特征的注意力对齐。
@@ -31,7 +31,7 @@ scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
 attn = softmax(scores) @ V
 ```
 
-2. **随机固定投影**（JL 引理保证）：
+2. **随机固定投影**（满足 JL 条件时可给距离保持界）：
 ```
 P = random_gaussian(d, r) / sqrt(r)  # 固定不训练
 scores = (Q @ P) @ (K @ P).T / sqrt(r)
@@ -51,17 +51,17 @@ scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
 - **分层投影**：浅层用小 $r$（粗筛），深层用大 $r$（精排）
 
 ## GPU 可行性
-- **维度 1 张量化 ✅**：投影 = 矩阵乘，注意力 = 矩阵乘链，全程张量运算
-- **维度 2 GEMM 可映射 ✅**：$Q P_Q$ 和 $K P_K$ 均为标准 GEMM，吃满 Tensor Core
-- **维度 3 复杂度 ✅**：投影 $O(ndr)$ 远低于注意力 $O(n^2 d)$，且投影后 $r \ll d$ 使注意力 $O(n^2 r)$
-- **维度 4 显存 ✅**：Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），直接降低显存峰值
-- **维度 5 低精度 ✅**：投影矩阵正交/近正交时数值稳定，bf16 可接受
-- **维度 6 并行 ✅**：投影可与注意力流水线并行，Multi-Head 天然跨头并行
-- **维度 7 稀疏 ⚠️**：投影矩阵本身稠密；若用稀疏投影（CountSketch），可能引入 gather/scatter
-- **维度 8 算子融合 ⚠️可改造需验证**：投影可融入 FlashAttention 的 online softmax 循环中
+- **D1[v]**：投影 = 矩阵乘，注意力 = 矩阵乘链，全程张量运算
+- **D2[v]**：$Q P_Q$ 和 $K P_K$ 均为标准 GEMM，吃满 Tensor Core
+- **D3[v]**：投影 $O(ndr)$ 远低于注意力 $O(n^2 d)$，且投影后 $r \ll d$ 使注意力 $O(n^2 r)$
+- **D4[v]**：Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），直接降低显存峰值
+- **D5[v]**：投影矩阵正交/近正交时数值稳定，bf16 可接受
+- **D6[v]**：投影可与注意力流水线并行，Multi-Head 天然跨头并行
+- **D7[~]**：投影矩阵本身稠密；若用稀疏投影（CountSketch），可能引入 gather/scatter
+- **D8[~]可改造需验证**：投影可融入 FlashAttention 的 online softmax 循环中
 
 ## 论文表述方式
-"我们将注意力计算分解为低维子空间投影与投影空间注意力两步，在保持注意力质量的同时将 Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），并保证 Johnson-Lindenstrauss 距离保持性。"
+"我们将注意力计算分解为低维子空间投影与投影空间注意力两步，以降低 Key 表示维度；V-cache 需独立处理。若使用独立随机投影并满足 Johnson-Lindenstrauss 的样本规模与维度条件，可近似保持投影对象的欧氏距离；这不自动保证 softmax attention 质量，需报告 attention/output 误差与任务指标。"
 
 ## 风险
 - **投影方向退化**：可学习投影可能坍缩到少数方向（$P^T P$ 条件数恶化），导致注意力分布退化。需加正交正则化 $\|P^T P - I\|_F^2$。

package/design-patterns/attention/spectral-attention.en.md

@@ -1,22 +1,24 @@
 # Spectral Attention
-> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Applicable Problems
-When the input signal exhibits **frequency-domain/spectral structure** (periodicity, cyclic symmetry, graph structure), computing attention in the spectral domain rather than the spatial domain can dramatically reduce complexity while exploiting the signal's intrinsic structure. Typical scenarios include: time series forecasting (periodic signals), graph neural networks (graph Laplacian spectral decomposition), positional encoding (frequency-domain interpretation of RoPE/ALiBi), and $O(n \log n)$ acceleration of long-sequence attention.
+When the input signal exhibits **frequency-domain/spectral structure** (periodicity, cyclic symmetry, graph structure), computing attention in the spectral domain rather than the spatial domain can dramatically reduce complexity while exploiting the signal's intrinsic structure. Typical scenarios include: time series forecasting (periodic signals), graph neural networks (graph Laplacian spectral decomposition), positional encoding (frequency-domain interpretation of RoPE/ALiBi), and $O(n \log n)$ acceleration of long-sequence **position-dependent** attention (requires circulant/Toeplitz structure assumption; does not apply to general content-dependent softmax attention).
 
 ## Mathematical Inspiration
 - Lenses: [duality (frequency-domain transform), symmetry (cyclic/translation invariance)]
-- Knowledge: [`../../knowledge-base/probability/entropy.md` (spectral entropy for measuring signal complexity), `../../knowledge-base/probability/concentration-inequality.md` (frequency-domain concentration inequalities)]
+- Knowledge: [`../../knowledge-base/probability/entropy.en.md` (spectral entropy for measuring signal complexity), `../../knowledge-base/probability/concentration-inequality.en.md` (frequency-domain concentration inequalities)]
 
 ## Required Mathematical Knowledge
 - **Discrete Fourier Transform (DFT/FFT)**: $O(n \log n)$ frequency-domain transform and the cyclic convolution theorem
 - **Graph Laplacian Spectral Decomposition**: $L = U \Lambda U^T$, where $U$ is the graph Fourier basis
-- **Irreducible Representations of the Cyclic Group $\mathbb{Z}_n$**: The DFT matrix is precisely the representation matrix of the cyclic group (see `references/books/abstract-algebra.md` Ch.4, Ch.11)
+- **Irreducible Representations of the Cyclic Group $\mathbb{Z}_n$**: The DFT matrix is precisely the representation matrix of the cyclic group (see `../../references/books/abstract-algebra.en.md` Ch.4, Ch.11)
 
 ## AI Module Form
 
 **Core Idea**: Transform attention from the spatial-domain $Q K^T$ into diagonal/sparse operations in the spectral domain:
 
+> **Critical Prerequisite**: The equivalence "spectral-domain diagonalization = attention" **holds only when the attention matrix is circulant or Toeplitz**, i.e., attention weights depend solely on relative position $a_{ij} = f(i-j)$ and are independent of token content. In this case, the cyclic convolution theorem guarantees that FFT diagonalizes the convolution kernel. For standard content-dependent softmax attention $\text{softmax}(QK^T/\sqrt{d})$, the attention matrix is determined by query/key content and is generally **not** circulant or Toeplitz; therefore FFT element-wise multiplication **cannot** equivalently replace the $QK^T$ computation. Schemes A/C are essentially position-dependent convolutional attention approximations, not exact substitutes for general softmax attention.
+
 **Scheme A: FFT-Accelerated Attention (Time Series)**:
 ```python
 # Express cyclic convolution as element-wise multiplication in the frequency domain
@@ -54,18 +56,30 @@ attn = ifft(scores_freq)
 - **Frequency-Aware Positional Encoding**: The essence of RoPE is the unitary representation of the cyclic group $\mathbb{Z}$ (see Abstract Algebra Ch.4), generalizable to other groups
 
 ## GPU Feasibility
-- **Dimension 1 Tensorization**: FFT and matrix multiplication are both standard tensor operations
-- **Dimension 2 GEMM-mappability**: Spectral projection $U^T Q$ is a standard GEMM; although FFT is not GEMM, highly optimized cuFFT implementations are available
-- **Dimension 3 Complexity**: FFT attention $O(n \log n \cdot d)$, far superior to $O(n^2 d)$
-- **Dimension 4 Memory**: Frequency-domain representation introduces no extra dimensions; spectral projection can reduce to $k \ll n$ dimensions
-- **Dimension 5 Low Precision**: Complex-valued FFT suffers precision loss under fp16; fp32 or real-valued FFT (RFFT) is required
-- **Dimension 6 Parallelism**: FFT can be parallelized across batch/head; cuFFT supports multi-stream execution
-- **Dimension 7 Sparsity**: High-frequency components can be truncated in the spectral domain (structured sparsity), retaining only top-k frequencies
-- **Dimension 8 Operator Fusion**: Fusing FFT with attention requires custom kernels; no ready-made fusion exists in standard libraries
+- **D1**: FFT and matrix multiplication are both standard tensor operations
+- **D2**: Spectral projection $U^T Q$ is a standard GEMM; although FFT is not GEMM, highly optimized cuFFT implementations are available
+- **D3[~]**: FFT attention $O(n \log n \cdot d)$, far superior to $O(n^2 d)$ -- **but only when attention has translation-invariant / position-dependent structure** (e.g., convolutional attention). General content-dependent softmax attention remains $O(n^2 d)$; linear attention approximations can achieve $O(n d^2)$, but via a different mechanism unrelated to FFT spectral methods.
+- **D4**: Frequency-domain representation introduces no extra dimensions; spectral projection can reduce to $k \ll n$ dimensions
+- **D5**: Complex-valued FFT suffers precision loss under fp16; fp32 or real-valued FFT (RFFT) is required
+- **D6**: FFT can be parallelized across batch/head; cuFFT supports multi-stream execution
+- **D7**: High-frequency components can be truncated in the spectral domain (structured sparsity), retaining only top-k frequencies
+- **D8**: Fusing FFT with attention requires custom kernels; no ready-made fusion exists in standard libraries
+
+**Quantitative assessment example** (standard transformer, d=128, n=2048, h=16):
+- D3: FFT path FLOPs ≈ 2·n·log₂(n)·d ≈ 2·2048·11·128 ≈ 5.8M vs standard attention 2·n²·d ≈ 1.1G (only when attention is convolutional)
+- D4: No n×n attention matrix materialization; FFT intermediates O(n·d) ≈ 1MB
+- D5: FFT under bf16 has twiddle factor error ~10⁻³, acceptable
+- D8: FFT + pointwise can be fused into a single CUDA kernel
 
 ## Paper Phrasing
-"We propose a spectral-domain attention mechanism that transforms attention computation into the Fourier/Laplacian spectral domain, leveraging the cyclic convolution theorem to reduce sequence attention complexity from $O(n^2)$ to $O(n \log n)$ while preserving the ability to model dependencies at multiple scales through frequency-adaptive weights."
+"We propose a spectral-domain attention mechanism that transforms attention computation into the Fourier/Laplacian spectral domain, leveraging the cyclic convolution theorem to reduce **translation-invariant** sequence attention complexity from $O(n^2)$ to $O(n \log n)$ while preserving the ability to model dependencies at multiple scales through frequency-adaptive weights. Note: this acceleration **requires position-dependent (not content-dependent) attention structure**; for general content-dependent softmax attention, the spectral-domain equivalence does not hold."
+
+## Applicability Conditions
+- **Core restriction of FFT spectral attention**: The FFT acceleration in Schemes A/C **requires the attention matrix to be circulant or Toeplitz**, i.e., attention weights depend only on relative position $a_{ij} = f(i-j)$ and are independent of the specific token content (query/key vectors).
+- **When the condition is NOT satisfied**: Standard softmax attention $\text{softmax}(QK^T/\sqrt{d})$ in most NLP tasks is content-dependent -- attention weights are jointly determined by query and key content, and the resulting attention matrix generally lacks circulant/Toeplitz structure. In this case, FFT element-wise multiplication **cannot** replace the $QK^T$ operation, and the $O(n \log n)$ complexity advantage does not hold.
+- **When the condition IS satisfied**: Positional encoding (e.g., frequency components of RoPE/ALiBi), periodic convolutional kernels for time series, fixed-pattern attention with translation-invariant priors. Scheme B (graph spectral attention) is not subject to this restriction, as it projects onto graph Laplacian eigenbases rather than exploiting the cyclic convolution theorem.
 
 ## Risks
+- **[x] Content-dependent attention incompatibility**: The core assumption of FFT spectral methods (circulant/Toeplitz structure) is **fundamentally incompatible** with the mainstream content-dependent softmax attention used in NLP. Applying FFT methods indiscriminately to general attention mechanisms constitutes a mathematical error -- they compute different quantities. Any usage must explicitly declare the position-dependence assumption being made.
 - **Violation of Periodicity Assumption**: FFT implicitly assumes periodic boundary conditions, causing spectral leakage for non-periodic signals such as natural language. Windowing functions or zero-padding are needed.
 - **Graph Laplacian Precomputation Cost**: Eigendecomposition $O(n^3)$ is infeasible for large-scale graphs, requiring approximations (Nystrom/Lanczos), and dynamic graphs necessitate recomputation.